A Dynamic Scheduling Method Combining Iterative Optimization and Deep Reinforcement Learning to Solve Sudden Disturbance Events in Flexible Manufacturing Process

1. Introduction

The flexible manufacturing process involves many factors, such as component installation, positioning, handling, inspection, manual operations and order addition, which make it difficult to predict the actual process. Unpredictable sudden disturbance events such as equipment failure (Lu et al., 2021a; Lu et al., 2021b), order changes (He et al., 2020), and processing time lag (Yu and Pan, 2012) will increase the deviation between actual production and the planned schedule. The randomness and frequency of sudden events will lead to uncontrollability in production management and seriously affect production efficiency. Dynamic scheduling is a method that dynamically adjusts the original plan to cope with the impact of unexpected events according to the real-time production status, which is more in line with the needs of actual complex manufacturing environments (Ding et al., 2022; Kamali et al., 2023; Lu et al., 2017). Static scheduling is generally used in the initial stage before production execution. There is usually enough computation time at this stage. We can explore high quality solutions to complex problems through iterative optimization methods. However, it is not applicable to sudden disturbance events during execution. Therefore, there is an urgent need for a dynamic scheduling method with high responsiveness, fast feedback and high-quality capability to solve this problem.

Rescheduling is the main implementation method of dynamic scheduling (Nouiri et al., 2018). According to the triggering mechanism and type of sudden disturbance events, there are two key issues in rescheduling, namely “when” and “how” to reschedule (Adibi et al., 2010; Khodke and Bhongade, 2013).

There are three typical methods for “when” rescheduling, namely event-driven rescheduling, period-driven rescheduling, and event-driven and period-driven hybrid rescheduling (Ghaleb and Taghipour, 2023; Stevenson et al., 2020). (Hamzadayi and Yildiz, 2016) proposed a fully rescheduling method for multiple identical parallel machines based on event-driven strategy; (Ghaleb et al., 2021) researched task rescheduling methods for workshop production driven by equipment failure; (Shi et al., 2020b) and others researched dynamic scheduling methods based on rolling temporal optimization strategy driven by period; (Ning et al., 2016), (Baykasoğlu et al., 2020) and others studied workshop dynamic scheduling problems using a mixed method driven by a combination of period and time. Among the three triggering mechanisms, event-driven is suitable for situations where sudden events affect normal production. Period-driven is suitable for handling disturbance events that do not require immediate resolution. Hybrid driven combines the advantages of the previous two methods and is more suitable for dynamic scheduling in complex practical workshops.

With regard to “how to reschedule”, most of the studies focus on the computational efficiency of heuristic methods to improve the corresponding dynamic scheduling. (Adibi and Shahrabi, 2014) proposed an improved variable neighborhood search algorithm based on clustering for workshop dynamic scheduling problems; (Kundakcı and Kulak, 2016) studied a hybrid genetic algorithm to reduce the cycle of dynamic scheduling; (Park et al., 2018) researched a combination plan based on genetic planning integration to solve dynamic job shop scheduling problems; (Wang et al., 2022) studied the flexible workshop dynamic scheduling problem based on industrial big data. However, rescheduling based on heuristic algorithms is difficult to overcome the problems of long computation time and slow feedback for large order problems. It cannot meet the timeliness requirements of dynamic scheduling. To improve the response time, (Kouider and Bouzouia, 2012) has proposed algorithms based on multi-agent intelligence to solve dynamic scheduling problems. Although the decision speed of this method is fast, the solution quality is generally poor.

In recent years, the development of technologies such as the Industrial Internet (Laili et al., 2022) and Digital Twins (Yan et al., 2021) provides an immediate interconnection technology foundation for real-time rescheduling response. In terms of response speed, deep reinforcement learning (DRL) has provided a new direction for solving dynamic scheduling decision problems. Reinforcement learning is an important branch of machine learning and is the third machine learning method, alongside supervised learning and unsupervised learning. With this method, high-quality, dynamic and fast decisions can be made on complex problems. Deep Q-network (DQN) is one of the most effective DRL methods and is widely used in practice. DQN was first proposed by DeepMind, a subsidiary of Google, and applied to AlphaGo, which beat human champions at the game of (Mnih et al., 2015). This event has become a milestone event in the field of artificial intelligence in recent years.

Due to the successful application of DRL in the gaming field, its application in intelligent manufacturing has also begun to be explored and studied. (Shiue et al., 2018) applied DRL to real-time scheduling in intelligent factories. (Shi et al., 2020a) proposed an automated production line intelligent scheduling algorithm based on DRL. (Wang et al., 2021) proposed a dynamic scheduling method based on DRL. (Li et al., 2022) began exploring dynamic scheduling methods based on DQN, but did not fully consider the uncertainty of the production environment. (Hu et al., 2020) proposed a hybrid rule strategy for real-time scheduling optimization of automated guided vehicles based on DQN. (Luo, 2020) solved the dynamic scheduling problem of FJSP using a newly proposed job-insertion approach based on DQN. (Han and Yang, 2020) studied a new method for solving the job shop scheduling problem based on double DQN. These studies have begun to explore the application of reinforcement learning methods in production scheduling. However, the complex rules of the production environment limit the learning of the intelligent agent in its interactions. Therefore, modeling of feasible and reasonable production environment states, actions and rewards is particularly critical in reinforcement learning. It determines the learning direction of the intelligent agent. In summary, research into reinforcement learning algorithms such as DQN in dynamic production scheduling is still very limited.

This paper proposed a dynamic scheduling method combining iterative optimization and DRL to solve sudden disturbance events in flexible manufacturing process. A DRL environment model is established, which is closely related to dynamic flexible job shop scheduling. A training algorithm combining a genetic algorithm (GA) and double deep Q-network (DDQN) is proposed for scheduling decisions. Based on the above algorithm, the hybrid rescheduling method has been improved to achieve dynamic scheduling. Combined with the trained DRL model, the proposed method achieves efficient and high-quality dynamic autonomous decision making for scheduling tasks under uncertain events. Finally, the effectiveness of the proposed dynamic scheduling method is verified through data experiments.

The structure of this paper is as follows. Section 1 introduces the research significance and current research status. Section 2 starts with the problem analysis and the construction of the DRL environment model. The training strategy of the DRL decision method is proposed in section 3. Section 4 proposes a dynamic scheduling implementation method based on DRL. In section 5, data experiments and application validation are conducted. The conclusion of this paper is presented in section 6.

2. Problem Formulation and Workshop Scheduling DRL Model

With reinforcement learning, the intelligent agent continuously learns through interactions with the environment and achieves autonomous decision making. The implementation process of production scheduling calculation based on DRL is shown in Figure 1. Through the interaction and iteration between the agent and the environment, each process is sequentially sorted. Finally, the scheduling Gantt chart is calculated.

The essential components of the DRL model consist of environment state, action, policy, and reward. This section primarily focuses on the study subject and optimization objectives of dynamic scheduling problems. The environmental status, actions, policies, and reward models of DRL have been established in detail.

2.1. Problem Description and Scheduling Objective Model

The flexible job shop scheduling problem (FJSP) is a typical flexible manufacturing process scheduling problem, which is defined as follows. There are n jobs that need to be processed on m machines. Each job consists of ${\mathrm{n}}_{\mathrm{i}}$（${\mathrm{n}}_{\mathrm{i}}\ge 1$）operations. Each operation can be processed on different machines. The processing time for each operation may vary depending on the machine being used.

The time composition of an operation is defined as shown in Figure 2. An operation time includes waiting time (${\mathrm{T}}_{\mathrm{ik}}^{\mathrm{w}}$), transportation time (${\mathrm{T}}_{\mathrm{ik}}^{\mathrm{t}}$), and processing time (${\mathrm{T}}_{\mathrm{ik}}^{\mathrm{p}}$). Where i is the job number, and k is the operation number. For example, ${\mathrm{T}}_{\mathrm{ik}}^{\mathrm{w}}$ represents waiting time of the k-th operation of the i-th job. Moreover, ${\mathrm{st}}_{\mathrm{ik}}$,${\mathrm{tt}}_{\mathrm{ik}}$,${\mathrm{pt}}_{\mathrm{ik}}$,${\mathrm{et}}_{\mathrm{ik}}$ represent the total start moment, transportation start moment, processing start moment, and total end moment of the k-th operation of the i-th job, respectively. Therefore, the makespan of a job is defined as follows:

$$T_i={\sum }_{k=1}^{n_i}{T}_{ik}^{\mathrm{w}}+T_{ik}^{\mathrm{t}}+T_{ik}^{\mathrm{p}}$$

(1)

The maximum makespan refers to the maximum value of the makespan of all jobs, which is used to measure production efficiency. The optimization objective of dynamic scheduling is to minimize the maximum makespan, which can be represented as follows:

$$\min C_{\max }=\underset{i\in \left\{1，2，3，\cdots ,n\right\}}{\max }\left[\sum _{k=1}^{n_i}{T}_{ik}^{\mathrm{w}}+T_{ik}^{\mathrm{t}}+T_{ik}^{\mathrm{p}}\right]$$

(2)

Where n is the total number of jobs, and n_i is the total number of operations for the i-th job.

2.2. State Model for Scheduling Information

The state model for scheduling information needs to reflect real-time information on key production elements in the manufacturing workshop. The environmental state model is constructed from the perspectives of jobs and equipment. The state for the job includes the number of operations completed and the current time spent, reflecting the current overall manufacturing progress. The state for equipment mainly includes the idle status of all processing and transportation equipment. Therefore, the environmental state model includes the current number of operations completed ($s^{\mathrm{N}}$), the current time progress ($s^{\mathrm{P}}$), the idle status of processing equipment ($s^{\mathrm{M}}$), and the idle status of transportation equipment ($s^{\mathrm{T}}$), which can be represented as follows:

$$s=\left[s^{\mathrm{N}},s^{\mathrm{P}},s^{\mathrm{M}},s^{\mathrm{T}}\right]$$

(3)

(1) Current number of completed operations (s^N) represents the current progress of the task, which can be defined as follows:

$$s^{\mathrm{N}}={\displaystyle \frac{t^{\mathrm{N}}}{{\sum }_{i=1}^n{n}_i}}$$

(4)

Where $t^{\mathrm{N}}$ is the number of operations completed at the current moment.

(2) Current time progress (s^P) represents the progress status based on the benchmark time at the current moment, which is used to measure the quality of the completed task plan at the current time. The definition is as follows:

$$s^{\mathrm{P}}={\displaystyle \frac{C^{\mathrm{t}}-C_{\max }^{\mathrm{GA}}}{C_{\max }^{\mathrm{GA}}}}$$

(5)

Where $C_{\mathrm{m}\mathrm{a}\mathrm{x}}^{\mathrm{G}\mathrm{A}}$ is the benchmark time. The benchmark time is the minimum value of the maximum makespan of all jobs calculated by genetic algorithm. $C^{\mathrm{t}}$ is the time to complete the current task at the current moment, which is the time to complete s^N operations.

(3) Idle status of processing equipment (s^M) represents the degree of idle status of all processing equipment at the current moment. Idle status can be measured by idle time. The idle time of each processing equipment is different. When the processing equipment is not assigned a task at the current moment, idle time of the processing equipment is less than the current moment. Otherwise, idle time is the completion time of the current task. The idle time of each processing equipment is set as follows:

$$T^{\mathrm{end}}=\left(T_1^{\mathrm{E}},T_2^{\mathrm{E}},\dots ,T_j^{\mathrm{E}},\dots ,T_m^{\mathrm{E}}\right)$$

(6)

Where ${\mathrm{T}}_{\mathrm{j}}^{\mathrm{E}}$ represents the completion time of the current task for the j-th processing equipment. In order to standardise the measurement of idle status, the idle time of all equipment is sorted according to Algorithm 1.

Algorithm 1: The process of calculating the idle state of machining equipment

imput:Tend output: sM 1: for i = 1 to m do 2:  sM[i] ← 0 3: end for 4: for i = 1 to m do 5:  T min ← min(T end) 6:  p ← index(T min in T end) 7:  sM[p] ← i 8:  T end[p] ← inf 9: end for 10: Return sM

By inputting the T ^end into Algorithm 1, the idle level ranking of the processing equipment can be calculated, represented as s^M. It can be expressed as:

$$s^{\mathrm{M}}=\left(s_1^{\mathrm{M}},s_2^{\mathrm{M}},\dots ,s_j^{\mathrm{M}},\dots ,s_m^{\mathrm{M}}\right)$$

(7)

Where $s^{\mathrm{M}}$ represents the idle state for the j-th processing equipment at the current moment, m is the total number of processing equipment.

(4) Idle state of transportation equipment (s^T) represents the degree of idle status of all transportation equipment at the current moment. By inputting s^T into algorithm 1, the idle level ranking of transportation equipment can be calculated, represented as s^M. It can be expressed as:

$$s^{\mathrm{T}}=\left(s_1^{\mathrm{T}},s_2^{\mathrm{T}},\dots ,s_{mt}^{\mathrm{T}}\right)$$

(8)

Where mt is the total number of transportation equipment.

By combining the above four state information, the state model of the reinforcement learning environment can be expressed as:

$$s=\left(s^{\mathrm{N}},s^{\mathrm{P}},s_1^{\mathrm{M}},s_2^{\mathrm{M}},\dots ,s_j^{\mathrm{M}},\dots ,s_m^{\mathrm{M}},s_1^{\mathrm{T}},s_2^{\mathrm{T}},\dots ,s_{mt}^{\mathrm{T}}\right)$$

(9)

The total number of state information nodes is calculated as follows:

$$n^{\mathrm{input}}=2+m+mt$$

(10)

The n^input is the number of input parameters of the neural network in deep reinforcement learning, which will be used in the subsequent algorithm.

2.3. Action Model

The agent will make a decision about the next action based on the current state of the environment. For the scheduling problem, the action of each step needs to include decisions on three key issues.

(1) Which process should be scheduled next?

(2) Which transportation equipment should be used for the transportation of the process?

(3) Which processing equipment should be used for the processing of the process?

In FJSP, each process can be processed by different equipment. When using different processing equipment in the same process, the processing time may also be different. A nested sequence approach is designed to express processing time, which is defined as a Matrix. In the Matrix, each row represents the information of a job, and each array in the row represents the processing time information of a process. The amount of data in the array represents the number of available processing equipment, and the value at the corresponding position represents the processing time of the process by the corresponding equipment.

Figure 2. An example of the structure of Matrix.

Figure 2. An example of the structure of Matrix.

Figure 2 shows a Matrix containing two job processing times. According to the structure of the Matrix, it can be seen that job 1 requires two processes and job 2 requires 3 processes. And there are 5 available processing equipment. The processing time for each process on the 5 equipment is different. Therefore, to eliminate the randomness of processing time on a single machine, the pruning average method is used to measure the processing time required for each process. The metric for processing time of a process is defined as follows:

$$t_{ik}=\left\{\begin{array}{c}{\displaystyle \frac{{\sum }_{j\in M_{ik}}\left(t_{ikj}\right)-\left(t_{ik}^{\max }+t_{ik}^{\min }\right)}{m-2}}，\mathrm{if}{m}_{ik}\ge 3\\ {\displaystyle \frac{{\sum }_{j\in M_{ik}}\left(t_{ikj}\right)}{m_{ik}}}，\mathrm{else}\end{array}\right.$$

(11)

In Equation (11), t_ijk represents the processing time of process k of job i using equipment j, M_ik represents the set of optional processing equipment for process k of job i, m_ik represents the amount of optional processing equipment for process k of job i, ${\mathrm{t}}_{\mathrm{ik}}^{\max }$ is the maximum processing time for process k of job i, and ${\mathrm{t}}_{\mathrm{ik}}^{\min }$ is the minimum processing time for process k of job i. Other metrics associated with processes can be calculated based on the processing time metric t_ik of the process, which includes the processing time of the next process (t_i(h+1)), the total processing time of the remaining processes (${\mathrm{t}}_{\mathrm{t}}^{\mathrm{r}1}$), and the total processing time of the remaining processes except for the next process (${\mathrm{t}}_{\mathrm{t}}^{\mathrm{r}2}$). These metrics are represented as follows:

$$t_t^{r1}=\sum _{k=h}^{n_i}{t}_{ik}$$

(12)

$$t_t^{r2}=\left\{\begin{array}{c}\sum _{k=h+1}^{n_i}{t}_{ik}，\mathrm{if}h<k\\ t_{ik}，\mathrm{else}\end{array}\right.$$

(13)

Process selection: the optimization goal is to minimize the maximum makespan. Therefore, the processing time of the process is the most important factor. The heuristic methods listed in Table 2-1 are all closely related to the processing time of the process. The selection result of the process for each decision can be calculated according to different rules based on the above metrics.

Equipment selection: during the equipment selection process, processing/transportation equipment is assigned to selected processes based on the the results of the process selection and equipment load balancing principle. The specific equipment selection process is shown in Algorithm 2. The input information of Algorithm 2 includes the process selection results (O^list) and the nested matrix of processing times for each process (Matrix), while the output information includes the equipment selection results (M^list).

Algorithm 2: Equipment selection process based on load balancing

imput: Olist, Matrix output: Mlist 1: for i = 1 to m do 2:  T m[i] ← 0 3: end for 4: for i = 1 to length (O list) do 5:  H[i] ← Matrix[O list[i]] 6:  for j = 1 to length(H) do 7:   D[j] ← H[j] 8:   for k = 1 to length(D) do 9:     S [k] ← (T m[k]+D[k]) 10:   end for 11:   t min ← min(S) 12:   p ← index(t min in S) 13:   T m[p] ← t min 14:   M list ← p 15:  end for 16: end for 17: Return M list

2.4. Policy Model

To improve exploration efficiency, the epsilon-greedy decreasing strategy is adopted. In the early iteration stage of this method, there is a higher probability of random selection to increase the diversity of the algorithm. As the number of iterations increases, the probability of random selection decreases while the probability of decision-making selection increases. There will be better convergence in the later stages of the iteration. The strategy can be expressed as:

$$a_t=\left\{\begin{array}{l}random,\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y} \mathrm{i}\mathrm{s}\epsilon -{\displaystyle \frac{\epsilon }{N^{\mathrm{iter}}}}\cdot n\_step\\ \underset{a}{\mathrm{argmax}}Q\left(s_t,a_t;\theta \right)，\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y} \mathrm{i}\mathrm{s}1-\left(\epsilon -{\displaystyle \frac{\epsilon }{N^{\mathrm{iter}}}}\cdot n_{step}\right)\\ {\pi }^{\mathrm{best}}\left(a\right)，\mathrm{I}\mathrm{f} \mathrm{t}\mathrm{h}\mathrm{i}\mathrm{s} \mathrm{i}\mathrm{s} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{l}\mathrm{a}\mathrm{s}\mathrm{t} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\end{array}\right.$$

(14)

In the Equation (14), ε is an initial parameter ranging from 0 to 1, n_step is a variable indicating the number of training times, and π^best(a) represents the optimal action selection result during the training period.

When the last training iteration is reached, the best action decision is output according to the policy π^best(a). At the beginning of the training, n_step is a very small number, and the probability of random action selection is high. As the training time increases, the value of n_step also increases. The probability of random action selection decreases, while the probability of decision action selection by the reinforcement learning model increases. With this strategy, more diverse solutions are explored in the early stage, and better and more stable solutions are obtained in the later stage.

2.5. Reward Fuction

The reward function is used to assess the level of impact of the current action on the state of the environment. In the problem proposed in this paper, when all process and equipment selections have been completed, the makespan can be used to evaluate the efficiency of the scheduling scheme. Therefore, a sparse reward approach is adopted where rewards for intermediate processes are set to zero. The intelligent agent is rewarded based on the quality of the scheduling result after all process and equipment selections are completed.

In most cases, the optimal value of the makespan for actual scheduling problems is unknown. Therefore, the time benchmark A in the current time progress of the state model in Section 2.2 is adopted. A normalization method for measuring rewards is designed, which improves the discrimination between results with different makespan. The time benchmark ${\mathrm{C}}_{\max }^{\mathrm{GA}}$ is a feasible solution obtained through genetic algorithms. If the makespan C obtained by the intelligent agent is less than ${\mathrm{C}}_{\max }^{\mathrm{GA}}$, a negative reward is given. It indicates the punishment for intelligent agents. Conversely, if C is greater than or equal to ${\mathrm{C}}_{\max }^{\mathrm{GA}}$, the intelligent agent receives a positive reward. The reward function is expressed as follows:

$$r_t=\left\{\begin{array}{l}{\displaystyle \frac{C_{\max }^{\mathrm{GA}}-C}{C_{\max }^{\mathrm{GA}}}},\mathrm{I}\mathrm{f} \mathrm{t}\mathrm{h}\mathrm{i}\mathrm{s} \mathrm{i}\mathrm{s} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{l}\mathrm{a}\mathrm{s}\mathrm{t} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\\ 0,\mathrm{else}\end{array}\right.$$

(15)

3. The Dynamic Scheduling Decision-Making Method Based on DRL

3.1. Algorithm Training Framework

In the DDQN algorithm, there are two deep neural networks with the same structure but different parameters, which are referred to as the main Q-network (parameters θ) and the target Q-network (parameters θ’). The main Q-network is used for agent decision-making, taking environmental states as inputs and outputting decision actions. The target Q-network is used to evaluate the quality of the main Q-network. The parameters θ of the main Q-network are periodically assigned to the parameters θ’ of the target Q-network. Therefore, the target Q-network has memories and experiences from the historical performance of the main Q-network, which can guide the decision-making of the main Q-network and reduce its overestimation. In the training process, the predicted Q-value obtained by the main Q-network is represented as follows:

$$y_t^{\mathrm{predict}}=Q\left(s_t,a_t;{\theta }_t\right)$$

(16)

Based on the Markov decision process and optimal strategy, the target Q-value obtained by the target Q-network can be expressed as Equation (17) (Van Hasselt et al., 2016).

$$y_t^{\mathrm{DDQN}}=r_{t+1}+\gamma Q\left(s_{t+1},\underset{a}{\mathrm{argmax}}Q\left(s_{t+1},a_{t+1};{\theta }_t\right);{{\theta }_t}^{\text{'}}\right)$$

(17)

Where γ is the discount factor that represents the decay of future rewards.

To prevent overestimation in the algorithm, the target Q-network corresponds its input behavior to the action that the main Q-network can obtain the maximum Q value, which is the most critical improvement of DDQN. Therefore, the Q-value update formula can be expressed as Equation (18), and the loss function can be expressed as Equation (19). In this paper, the gradient descent method is used to optimize the neural network parameters.

$$\begin{gathered} Q\left(s_t,a_t;{\theta }_t\right)=Q\left(s_t,a_t;{\theta }_t\right)+\alpha \cdot r_{t+1} \\ \begin{array}{c}+\alpha \left(\gamma Q\left(s_{t+1},\underset{a}{\mathrm{argmax}}Q\left(s_{t+1},a_{t+1};{\theta }_t\right);{{\theta }_t}^{\mathrm{\text{'}}}\right)-Q\left(s_t,a_t;{\theta }_t\right)\right)\end{array} \end{gathered}$$

(18)

$$L_t\left({\theta }_t\right)={\rm E}{\left[r_{t+1}+\gamma Q\left(s_{t+1},\underset{a}{\mathrm{argmax}}Q\left(s_{t+1},a_{t+1};{\theta }_t\right);{{\theta }_t}^{\text{'}}\right)-Q\left(s_t,a_t;{\theta }_t\right)\right]}^2$$

(19)

The memory replay method with priority experience is adopted for training to improve the training efficiency. The pseudo code of the DDQN-based training algorithm is shown in Algorithm 3.

Algorithm 3: DDQN-based training algorithm

1: Initialize the experience replay memory buffer D with capacity nmem 2: Initialize the main Q-network for action value function with random parameters θ 3: Initialize the target Q-network for action value function with parameters θ’ = θ 4: Obtain the initial state s1 and action a1 5: for t = 1 to Niter do 6:   obtain st+1, rt 7:   store (st, at, rt, st+1) to D 8:   for t = 1 to T do 9:     Use action selection policy to choose action at 10:     Execute at in the environment state st, then obtain reward r and the next environment state st+1 11:     store (st, at, rt, st+1) to D 12:     If the number of experiences in D > a batch size nbatch, then 13:       Select a batch size of (st, at, rt, st+1) from D based on priority experience. 14:       Calculate $y_t^{\mathrm{predict}}\leftarrow Q\left(s_t,a_t;{\theta }_t\right)$, $y_t^{\mathrm{DDQN}}\leftarrow r_{t+1}+\gamma Q\left(s_{t+1},\underset{a}{\mathrm{argmax}}Q\left(s_{t+1},a_{t+1};{\theta }_t\right);{\theta }_t^{\prime }\right)$ 15:       Update θ using the gradient descent method according to $y_t^{\mathrm{predict}}$ and $y_t^{\mathrm{DDQN}}$. 16:       Reset θ’←θ every C steps. 17:       Calculate the temporal difference error ${\delta }_t\leftarrow \left(y_t^{\mathrm{DDQN}}-y_t^{\mathrm{predict}}\right)$. 18:       Update the priority of D based on$\left|{\delta }_t\right|$. 19:     end if 20:   end for 21: end for

The model training involves two processes: environment interaction process and main Q-network training processe. The training process is shown in Figure 3.

(1) Environment interaction process: the agent makes an action a_t based on the state s_t of the environment, then the environment outputs the next state s_t+1 and the reward r_t. The data (s_t, a_t, s_t+1, r_t) generated during each iteration is stored in the experience replay memory D. To improve the exploration ability of the algorithm, the policy introduced in Section 2.4 is adopted as the action decision-making policy of the agent. The amount of data stored in the experience replay memory buffer D is limited. When the buffer is full, the earliest stored data is removed and new data is stored.

(2) Main Q-network training processe: when the data stored in the experience replay memory buffer D is greater than a batch size (n^batch), a batch size data is selected from D according to the priority experience rule to train the main Q-network. The Q-value is predicted by the main Q-network, and the Q-value is evaluated by the target Q-network based on the Markov decision process. Finally, the parameters θ of the main Q-network are modified. To ensure that historical experience can be reflected in the evaluation process, the parameters θ’ are updated based on the parameters θ after each C-step cycle.

3.2. Dynamic Scheduling Decision-Making Method

Manufacturing resources in actual production mainly include processing equipment, transportation equipment (AGV), and jobs. Each type of manufacturing resource may consist of multiple individual entities. Tuples are defined to record the real-time status of each individual entity in each manufacturing resource. For example, if there are five processing equipment in the workshop, the processing equipment resource will consist of five processing equipment tuples.

Each equipment tuple contains multiple labels and content related to equipment status. The labels of the processing equipment tuples include “device number”, “assigned task”, “start node”, “end node”, and “end time”. The device number is the unique identifier of the equipment. The assigned tasks are in the form of a list of multiple [job number, process number], which records the information of the job and process to be processed on the equipment. The tuple packaging method is adopted to achieve lightweight expression of production information. The tuple information of these manufacturing resources reflects the production status in real-time, facilitating the statistical analysis of the real-time production environment status after disturbances.

The process of autonomous decision-making for dynamic scheduling problem is illustrated in Figure 4. According to the training process in Section 3.1, we train a model for intelligent agent to make scheduling decisions. When disturbances occur, the agent will make decisions and output the new scheduling plan in the form of a Gantt chart.

4. Event-Driven and Period-Driven Hybrid Dynamic Rescheduling

Rescheduling is used to solve scheduling problems with disturbances. In Section 3, a dynamic scheduling autonomous decision-making method based on DDQN is proposed to solve the problem of how to implement rescheduling. However, there are many disturbance factors in the production environment. If there are any disturbing factors and a rescheduling is performed, it will greatly undermine the stability of the scheduling system. Therefore, research on the rescheduling strategy and triggering mechanism is also crucial.

4.1. Rescheduling Triggering Mechanism

In practical manufacturing environments, some disturbances may cause the workshop to halt production at once, while others may result in the inability to execute the original production plan after a period of time. Therefore, an event-driven and period-driven hybrid dynamic rescheduling is used to handle such disturbances. The event-driven approach is used to address disturbances that have an immediate impact on the production. The period-driven approach is used to monitor the lag effects of disturbances and to address them in a timely manner at periodic intervals to prevent impacts on subsequent production. This approach improves both the responsiveness and stability of dynamic scheduling.

In the Figure 5, it shows the implementation process of the dynamic rescheduling trigger strategy combining period-driven and period-driven methods. Based on the early-stage static scheduling results, a dynamic scheduling process is implemented with a series of preparatory steps at the initial stage. These steps can be divided into two parts:

Step 1: GA is used to compute the static scheduling plan for the unfinished jobs in order to obtain a complete scheduling plan and Gantt chart.

Step 2: Based on the scheduling plan obtained from Step 1, the periodic trigger points are determined by dividing the total number of manufacturing processes in the training agent. Since multiple devices work in parallel, it is difficult to determine the periodic trigger points. Therefore, the completion time during training is used as the basis for determining the periodic trigger points.

After completing these two steps, the scheduling plan is executed sequentially according to periodic windows. During the execution process of the scheduling plan, it is judged at each periodic trigger point whether the rescheduling needs to be triggered. If there is no disturbance event, the production process will follow the initial scheduling plan and complete each window task in sequence. If a disturbance event occurs, it is judged whether the disturbance event needs to trigger the rescheduling immediately. If the disturbance event has no effect on the current window task, the event-driven rescheduling does not need to be triggered at this time. Instead, the impact of the disturbance event on the processing-related resource changes will be adjusted in the next periodic trigger point, followed by the re-determination of the scheduling plan. If the disturbance event affects the current window task of the scheduling plan, the event-driven rescheduling needs to be triggered immediately.

When a disturbance is triggered, the job that has already been completed is not affected. Whether the job being processed is affected by the disturbance needs to be discussed in detail. When the disturbance does not affect the job being processed, the job continues to be processed according to the original plan. If the disturbance affects the job being processed, the process is stopped and joins the rescheduling with the process steps of the unprocessed jobs.

4.2. Dynamic Rescheduling Implementation Method

A dynamic scheduling implementation method with judgment based on DDQN is proposed. In the initial stage, the current scheduling plan is calculated based on the optimization algorithm, and more time can be spent in this process to calculate better solutions. Production is carried out according to the established scheduling plan, and disturbance events are monitored in real-time. If a disturbance event is detected, the algorithm will make a series of judgments to determine whether and when to reschedule. This approach minimizes the number of reschedulings and improves the stability of dynamic scheduling. The following are explanations for the four judgment conditions shown in Figure 6.

① Real-time disturbance judgement: it is the first judgment condition for the scheduling system after receiving a disturbance and it will determine whether the disturbance needs to be handled immediately. It will be determined in this stage whether the disturbance event will affect the current production plan. In general, equipment failures and time delays will directly affect the original production plan, so event-driven rescheduling will be triggered. However, disturbance caused by order changes are usually additional orders, and these additional orders are generally arranged for follow-up production planning in the next cycle node. In addition, there are other disturbances that do not need to be handled immediately, such as when a faulty equipment is no longer involved in the tasks being executed in the current time window, the disturbance will be handled in the next cycle node.

② Periodic node arrival judgement: it used for determining if the progress has reached the trigger node of the cycle. If the disturbance event does not affect the execution of the current window plan and the current progress has not yet reached the pre-set cycle node, the original scheduling plan will be executed. When the cycle node is reached, the production status and production tasks are analyzed and processed.

③ Periodic node handling judgment: when the scheduling reaches a periodic node, it is used to determine whether there are any non-real-time disturbances from the previous cycle that need to be processed. If the previous execution period did not change the production state, the original production plan is executed. If there was a non-instantaneous disturbance in the previous period that will change the current production resource state, it triggers a periodic-driven rescheduling.

④ Task completion judgment: it used to monitor whether the scheduling task is completed. If the scheduling task is not completed, it will be carried out according to the original plan. If the scheduling task is completed, the program will be terminated.

When disturbances occur, a series of judgments will be made to determine when to reschedule. The specific process is shown in Figure 4-2. Once rescheduling is required, the trigger condition for rescheduling is input, and then the scheduling plan for the executing window and the remaining tasks are output.

In Figure 7, it shows the implementation rescheduling process under equipment failure. When equipment failure disturbances occur, current environmental state information is got. The agent quickly makes decisions on task scheduling for the upcoming execution window, and divides the cycle nodes for remaining processes. The remaining tasks after the upcoming execution windows waiting for process have low real-time requirements, so evolutionary algorithms can be used to obtain an excellent solution.

5. Data Experimentation and Analysis

5.1. Introduction to Experimental Data

This paper mainly studies the dynamic scheduling problem of FJSP. Therefore, Kacem dataset is select as the basic data for experimental research. There are four different data sizes in the dataset, namely Kacem 4×5, Kacem 8×8, Kacem 10×10, and Kacem 15×10. Different datasets contain different processing information, for example, Kacem 4×5 indicates that there are four types of jobs and five machines in the production environment. In Table 2, it is shown the specific parameters of the Kacem 4×5 dataset. The transportation time between two nodes is set as a random number between 0.5 and 1.5 according to the processing time value range of Kacem dataset. Table 3 shows the transportation time for Kacem 4×5. Similarly, the transportation time of the Kacem 8×8, Kacem 10×10, and Kacem 15×10 dataset is generated using above method, which is used for data experiments.

5.2. Model Training for Different Problems

The algorithm randomly selects job numbers from the Kacem 4×5, Kacem 8×8, Kacem 10×10, and Kacem 15×10 datasets. Two orders with a total of 20 jobs are generated for each dataset, and the first order is used for the initial training of the agent. Based on the trained main Q-network model parameters for order one, the second order is used for the continued training of the main Q-network using the method of breakpoint continuation training. Figure 8, Figure 9, Figure 10 and Figure 11 respectively show the training process for the two orders in the four different problems.

The results show indicate that in the first training process of all datasets, the main Q-network focuses on exploration in the early stages and gradually converges, and the rewards obtained gradually stabilize in the later stage. Due to the foundation of the first training, the second training can quickly maintain stable and good output. The main Q-network can make good decisions and obtain good rewards from the start, and maintain a relatively stable state in subsequent training. In the two training processes, the production environment and product type are consistent, but the order quantity is different. Therefore, it can be confirmed that the trained main Q-network has the ability to make different decisions for different orders.

Table 4 and Table 5 show the computational results for the proposed method and other methods for the 10 orders in Kacem 4×5 dataset. Table 6 and Table 7 show the computational results for different methods for the 10 orders in Kacem 8×8 dataset. Table 8 and Table 9 show the computational results for different methods for the 10 orders in Kacem 10×10 dataset. Table 10 and Table 11 show the computational results for different methods for the 10 orders in Kacem 15×10 dataset. Numbers 1 to 10 represent different order numbers. Eight heuristic rule methods (in Table 1), genetic algorithm, and trained model by the proposed method were compared. The performance of the algorithms was evaluated using the makespan and decision calculation time.

The following conclusions can be drawn from the comparative experiments of data from orders of different sizes. Evolutionary optimization algorithms represented by genetic algorithms can obtain good feasible solutions in most problems, but require more computational time. Although heuristic rules have fast decision-making speeds, the stability of the resulting solutions is insufficient. The trained DDQN decision model is able to make decisions quickly while obtaining good solution results in most problems.

5.3. Dynamic Scheduling Case Study

A 20-job order was randomly generated based on the Kacem 4×5 dataset. The order was scheduled using genetic algorithm to obtain the initial scheduling plan as shown in Figure 12 and Figure 13. These two figures represent the scheduling plan of jobs and equipment, respectively. In the figures, the blue block represents no-load transportation process, the green block represents load transportation process, and the white block represents processing process. For the job scheduling plan in Figure 12, the three numbers in parentheses respectively represent the job process, the processing equipment number, and the transportation equipment number. For the equipment scheduling plan in Figure 13, the numbers in parentheses for processing equipment respectively represent the job number and the job process number, and the numbers for transportation equipment represent the job number. The final blue vertical line represents the makespan on the scheduling plan, which is 60.46.

Figure 12 and Figure 13 show the scheduling plan obtained using genetic algorithm, which serves as the initial scheduling plan. Based on this case, dynamic scheduling verification experiments are conducted under the influence of disturbance factors. The disturbance factors include three typical sources of uncertainty: equipment failure, processing times lag, and order changes.

5.3.1. Equipment Failure Disturbance Case

Once a processing equipment is failure, it is necessary to allocate the tasks from the faulty equipment to other similar equipment to reduce the impact of equipment failures on production schedules.

Equipment failure is assumed to occur in processing equipment M1 at 25 seconds, as shown in Figure 14. At 25 seconds, the jobs being processed are (5,3) on M1, (1,1) on M3, and (12,3) on M5. The jobs currently being handled by the transportation equipment are (3,1) on T1, and (9,1) on T2. These processes are in progress when the disturbance is triggered.

The job 5 being processed by equipment M1 is non-conforming due to a failure of equipment M1. Therefore, job 5 must to be reprocessed. In the subsequent rescheduling, job 5 will be treated as a new job and processed from the raw material stage. The remaining equipment in operation will continue to complete the current processing step after the disturbance occurs. Therefore, the allowed rescheduling start time for this equipment is the moment when the current task is completed. If the equipment is not in operation when the disturbance occurs, then the allowed rescheduling start time for this equipment is rescheduled zero time.

Manufacturing resource information is used as a constraint for rescheduling, and these constraints are considered when making decisions in the main Q-network. Figure 15 shows the rescheduling results obtained from the main Q-network. It spends 2.44 seconds from the trigger of the disturbance to make the decision. The equipment failure disturbance occurred at 25 seconds. The original scheduling plan was executed before 25 seconds, and the rescheduling plan was executed after 25 seconds. Job 5 on faulty equipment M1 has been reprocessed in the rescheduling plan, as shown in Figure 15.

In addition, for the transportation process, the transportation equipment will continue to complete the current transportation task when a disturbance occurs. As shown in Figure 14., at the time of disturbance, the process (3, 1) is being transported to M4 by T1. The rescheduling plan shown in Figure 15 still arranges the process (3, 1) to transport to M4 by T1. However, for T2, the process (9, 1) is being transported to M1 at the time of disturbance. Process (9, 1) was affected by faulty equipment M1 and the assigned processing equipment was changed. Therefore, in the rescheduling plan shown in Figure 15, process (9, 1) was rescheduled to be transported to M3.

5.3.2. Processing Time Lag Disturbance Case

Based on the original production plan (in Figure 13), processing time lag is assumed to occur in processing time of process (17,1) on machine M3. In the original plan, process (17,1) should end at 20.84 seconds. However, the disturbance is expected to cause a delay of 15 seconds on top of the original processing time. Therefore, the triggering time of the disturbance is at 20.84 seconds. This phenomenon will affect subsequent operations and trigger event-driven rescheduling.

Figure 16 shows the rescheduling triggered by a processing time lag event. The triggering node is set to the normal completion time of process (17,1), which is 20.84 seconds. Other operations that are being processed at this time node include (19,2) and (20,1), which will continue to be completed. The delayed operation (17,1) will continue to complete the task according to the expected delay time of 15 seconds. The allowed start time node for subsequent operation (17,2) is set to 35.84 seconds, and the allowed start time for M3 is also set to 35.84 seconds.

The result of rescheduling triggered by a processing delay disturbance is shown in Figure 17. The triggering node is at 20.84 seconds. The original scheduling plan was executed before 20.84 seconds. After 20.84 seconds, the scheduling result determined by the trained master Q-network is executed. The decision-making time for this process is 2.5 seconds.

5.3.3. Order Change Disturbance Case

Since order change disturbance generally does not directly affect the current task being performed, this type of disturbance event will be processed at the cycle node. It will reduce rescheduling times and improve the stability of dynamic scheduling.

Based on the Kacem 4x5 dataset, orders for 40 jobs were randomly generated, and a genetic algorithm was used to calculate the production schedule for the orders. The initial scheduling plan obtained is shown in Figure 18. The cycle time was set to 60 seconds. Every 60 seconds, the system will check the current manufacturing resource information. If the manufacturing resource information has not changed, the cycle-driven rescheduling will not be triggered, and the production will continue according to the original plan. To facilitate the display of job information, the job number is only displayed on the processing equipment.

The result of rescheduling triggered by an order disturbance is shown in Figure 19. The order change occurred between 0-60 seconds, so the rescheduling was triggered at the first dynamic scheduling cycle node after the order change disturbance, which was at 60 seconds. At this moment, the newly added orders are included in the calculation range of the algorithm, and the tasks that need to be completed by the orders are reorganized. The original scheduling plan was executed before 60 seconds. After 60 seconds, the scheduling plan determined by the main Q-network was executed. The decision-making time was 12.45 seconds due to the large amount of calculation, which resulted in a corresponding increase in response time. However, it can still be considered that the algorithm quickly provided a scheduling solution.

6. Conclusion

In this paper, dynamic scheduling based on reinforcement learning under uncertain disturbance factors is studied. The real-time status composition of production workshop scheduling is analyzed, and a DRL environment model of FJSP is established. A high-quality rescheduling decision-making method and a highly responsive rescheduling trigger mechanism are proposed. The effectiveness of this algorithm is verified through data experiments and actual disturbances. The solution for dynamic scheduling under disturbance factors with high responsiveness and quick feedback is studied from multiple aspects, including DRL environment modeling, DRL algorithm design and training, rescheduling trigger mechanism, and data experiment verification. The conclusions of this paper are as follows:

(1) A real-time DRL production environment model consisting of production status, action, policy, and reward function is established. The status consists of the current number of completed processes, current time progress, processing equipment idle status, and transportation equipment idle status. The scheduling decision is used as the action model, and the scheduling result is used as the reward function.

(2) A dynamic scheduling decision-making method based on DRL is studied, and a training algorithm combining GA and DDQN is proposed. Based on GA, the reward benchmark is established in the initial training stage. The model is trained based on DDQN, and the autonomous scheduling decision-making after disturbance is realized.

(3) An event-driven and period-driven hybrid dynamic rescheduling is proposed based on the trained DRL decision model. The cycle node is based on the number of jobs in the DRL training model. This triggering mechanism effectively controls the rescheduling calculation times and improves the stability of the dynamic scheduling system.

The data experiment results show that the trained DRL dynamic scheduling decision-making model can provide timely feedback on the adjusted scheduling arrangements for different-scale order problems. In particular, compared with metaheuristic algorithms, the decision-making time is improved from minutes to seconds, which is a significant improvement. The decision results of the DRL dynamic scheduling model can provide better solutions for most problems. Dynamic scheduling decision-making process application verification is carried out based on disturbance factors such as equipment failure, processing time lag, and order changes. The proposed dynamic scheduling decision-making method and rescheduling trigger strategy can realize high responsiveness, quick feedback, high quality, and high stability of autonomous dynamic scheduling decision-making.