A Review on Reinforcement Learning in Production Scheduling: An Inferential Perspective

1. Introduction

Production scheduling is considered as one of the most critical element of manufacturing management in aligning production activities with business objectives, in ensuring a smooth flow of goods resources, and in supporting company’s ability to remain competitive in the marketplace. Scheduling algorithms have long been a subject of extensive research in various interdisciplinary domains, such as industrial engineering, automation, and management science, due to their important role in enhancing production efficiency and effectiveness [1]. The production scheduling tasks can be solved using three main types of step-by-step procedures such as exact algorithms, heuristic algorithms, and meta-heuristic algorithms [2,3]. Although an exact algorithm can theoretically guarantee the optimum solution, the NP-hardness of major problems makes them impossible to address effectively and efficiently [4]. Heuristics use a set of rules to create scheduling solutions quickly and effectively without consideration of global optimization. Furthermore, the creation of rules is heavily reliant on a thorough comprehension of the particulars of the situation [5,6,7,8]. Whereas meta-heuristics can produce good scheduling solutions in a reasonable amount of computing time, the way search operators create them significantly depends on the specific situation at hand [9,10,11,12,13]. In addition, the iterative search process poses challenges in terms of time consumption and applicability in real-time scenarios when dealing with large-scale problems. Reinforcement learning is a subfield within the broader domain of machine learning. RL is considered one of the most perspective approaches for robust cooperative scheduling, which allows production managers to interact with a complex manufacturing environment, learn from previous experience, and select optimal decisions. It involves the process of an agent autonomously selecting and executing actions to accomplish a given task. The agent learns via experience and aims to maximize the rewards it receives in certain scenarios. The primary goal of RL is to optimize the cumulative reward obtained by an agent through the evaluation and selection of actions within a dynamic environment [14,15,16,17,18,19,20]. The most current development in artificial intelligence technology has allowed successful application of RL in sequential decision-making problems with multiple objectives which are usable for robot scheduling and control [21,22]. The research on production scheduling using RL since 1998 has been evolving as advancing optimization techniques compared to metaheuristics. RL significantly improves the computational efficiency of addressing scheduling problems. Numerous studies of RL on production scheduling have been undertaken since its inception in 1998, establishing a substantial and valuable foundation (see, e.g., [23,24,25,26,27]). For this reason, it appears useful to employ bibliometric analysis to illustrate the current advancements and trends in this particular domain. Moreover, gaining a deeper understanding of prominent authors and collaborative countries in the given research field can serve as a source of motivation for academics and practitioners in their forthcoming endeavors.

Accordingly, the objective of this study is to gain a comprehensive impression of the existing research and emerging developments in RL applied to production scheduling. The main research question that this paper will address is: What are the main emerging research areas, the most important methodologies, and typical implementation domains within RL applied to production scheduling?

For this purpose, a consistent methodological framework for assessing bibliometric data has been designed, whose structure is described in subsequent section of this paper.

2. Materials and Methods

The bibliometric approach employed here presents a quantitative instrument for monitoring and representing scientific progress by examining and visualizing scientific information. The growing acceptance of bibliometric methods in several academic disciplines indicates that their use brings expected effects [28,29,30,31]. The present investigation is conducted utilizing the procedure comprising of five coherent phases, as depicted in Figure 1.

The bibliometric analysis of RL in the context of production scheduling is focusing here on its theoretical foundations and practical implications, between the years 1998 and 2023. The first two phases involve the collection of bibliographic data, which was gathered via a Web of Science (WoS) database - Core Collection of Clarivate Analytics. The search was restricted to this database due to its prominence as a comprehensive repository of scientific literature and its frequent utilization in academic assessments. The inclusion criteria encompassed publications that contain the terms “reinforcement learning” and “manufacturing” and “scheduling” using search term “Topic” and document types “Article”, “Review Article”, and “Early Access”. In the next two phase of the data sorting and presentation, the Microsoft Excel and VOSviewer software were employed to extract the necessary information, such as the annual scientific outputs, most relevant sources, most cited author, and keyword co-occurrence.

Moreover, this research follows an inferential approach, where the sample of population is explored to determine its characteristics [32,33]. Moreover, the inferential concept of scientific representation proposed by Suárez [34] was applied here to formulate research outputs. Its essence is to employ alternative reasoning to reach results that differ from the isomorphic view of scientific representation in the sense that empirical knowledge plays an important role in inductive reasoning [35,36].

Prior to the bibliometric analysis, a literature review on the recent development and research in the field is carried out in the following section.

3. Literature Review

RL techniques have gained significant popularity, hence affirming the level of interest in agent-based models. The objective of previous studies has focused on the application of deep reinforcement learning (DRL) in the domain of job shop scheduling. In recent years, there has been notable success in the application of RL to address many combinatorial optimization issues, including the Vehicle Routing Problem as well as the Traveling Salesman Problem [37,38,39,40,41,42]. Considering that a production scheduling task may be conceptualized as the environment within the framework of RL, an agent can acquire a policy of well-designed actions and states, and engage the extensive offline training through interaction with the environment. This innovative concept offers a fresh perspective on addressing scheduling challenges, particularly those characterized by uncertainty and dynamism, and necessitating stringent real-time constraints, as in the case of in a dynamic job shop scheduling problem [22,43,44,45,46,47].

In the context of production scheduling problems, value-based approaches are commonly utilized RL algorithms. These algorithms, ranked from most to least frequently employed, include Q-learning, temporal difference TD(λ) algorithm, SARSA, ARL, informed Q-learning, dual Q-learning, approximate Q-learning, gradient descent TD(λ) algorithm, revenue sharing, Q-III learning, relational RL, relaxed SMART, and TD(λ)-learning. In the field of DRL, many value-based approaches have been employed, such as DQN (Deep Q-Learning Networks), loosely-coupled DRL, multiclass DQN, and the Q-network algorithm [48,49,50,51,52,53,54,55,56,57,58].

Qu et al. [25] applied multi-agent approximate Q-learning to address the issue of conducting numerical experiments to showcase the efficacy of the methodology in both static and dynamic environments, as well as in various scenarios of a flow shop. The objective was to develop and execute optimized manufacturing scheduling in a manufacturing setting, considering realistic interactions among a workforce’s skill set and adaptive machines. Luo [59] employs Deep Reinforcement Learning (DRL) techniques to address the dynamic flexible job shop scheduling problem. The author is specifically focusing on scenarios including new work insertions, while his primary objective is to minimize the overall tardiness of the schedule. Luo et al. [60] were the first to employ hierarchical multi-agent proximal policy optimization (HMAPPO) as a solution for the constantly changing partial-no-wait multi-objective flexible job-shop problem (MOFJSP) with new job insertions and machine breakdowns.

Wang et al. [37] discusses that the production scheduling process involves manufacturing of several types of items using a hybrid production pattern that utilizes Multi-Agent Deep Reinforcement Learning (MADRL) model. Popper et al. [43] suggested that MADRL can be employed to optimize flexible production plants in a reactive manner, taking several criteria into account such as efficient and ecological target values. Du et al. [61] utilized the DQN algorithm to address the flexible task shop scheduling problem (FJSP) in the presence of varying processing rates, setup time, idle time, and task transportation. The method incorporates 34 state indicators and 9 actions to optimize the exploitation capabilities of the DQN component. Additionally, a problem-driven EDA component is integrated into the algorithm to augment the exploration capabilities.

Li et al. [62] utilizes a DRL approach for discrete flexible job shop problem with inter-tool reusability (DFJSP-ITR). To address the multi-objective optimization problem of minimizing the combined makespan and total energy consumption, a set of 26 generic state features, a genetic programming-based action space, and a reward function are proposed. Zhou et al. [63] suggests the utilization of online scheduling techniques that rely on RL with composite reward functions. This approach aims to enhance the efficiency and resilience of industrial systems. A novel algorithm for production scheduling based on deep reinforcement learning in complex job shops was applied in work of [18]. The authors in this article also presented the main advantages of this application such as better flexibility, global transparency, and global optimization. There are also other authors who paid attention to deep reinforcement learning, and developed diverse learning algorithms able to develop complex strategies and optimize production scheduling. For example, Luo et al. [26] developed on-line rescheduling framework for the dynamic multi-objective flexible job shop scheduling problem with new job insertions. This framework offers several advantages over others, e.g., it allows to minimize the total tardiness or maximize machine utilization rate. Another authors [27] applied deep Q-learning method to solve dynamic scheduling in smart manufacturing environment. Dynamic scheduling method on deep reinforcement learning has been also proposed in work [64], where proximal policy optimization was adopted to find the optimal policy of the scheduling in a real-world environment. It is also worth mentioning here that Wang and Usher [65] investigated in their work the application of widely used reinforcement Q-learning algorithm to be applied for agent-based production scheduling.

4. Research Findings and Results Description

Using the above-specified keywords and search criteria, 220 co-authored articles were found. The primary research areas that receive significant attention in the articles include Manufacturing Engineering, Industrial Engineering, Computer Science Interdisciplinary Applications, Operations Research Management Science, Electrical Electronic Engineering, Computer Science and Artificial Intelligence, Automation Control Systems, Computer Science and Information Systems, and other topics. The 220 publications in our sample were categorized into 34 distinct research topics. The eight primary research areas, along with their article distribution are displayed in Table 1.

4.1. Trends of Publications and Citations

Numbers of published articles and their citations usually provide sufficiently reliable information to anticipate further development of examined research domain. Considering the numbers of publications and citations in the field of production planning using learning algorithms keeping around 26 years of data, the trend analysis graph has been derived. For this purpose, the same search terms and keywords were applied as in case of identification of major research categories, but the types of documents were extended to all the types. The reason of changing document types was to find out the initial research initiatives in this domain. The same search conditions were applied in the rest of the paper to include the larger sample of publications for the purpose of the investigation. The annual distribution of publications (out of the total 297 items) and their citations from the same database during the period from 1998 to 2023 is illustrated in Figure 2.

An examination of the yearly scientific outputs between 1998 and 2017 amply demonstrated a relatively stable low number of articles published annually. Throughout these two decenniums, the need for RL in production scheduling in real conditions apparently did not appear. During the 2019–2023 era, there was a noticeable rise in the number of publications that were registered in the most recognized database for peer reviewed content. This phenomenon can be primarily attributed to the advancements in artificial intelligence. It is noteworthy to emphasize that if this exponential trend of increasing the number of publications continues, then one can anticipate that during the next decade the importance of RL in manufacturing scheduling will significantly increase.

4.2. Most Relevant Sources

An identification of the most relevant publications from an initial dataset presents common approach in bibliometric research since such sources usually publish influential research that attracts widespread interest. As a rule, the most productive journals have the greatest influence on the development of science in a particular field since they publish more articles and generate more citations [66,67,68]. As for as the relevancy of literature in the explored field, the top ten journals that have published the most articles, are identified here. Also, the top ten most cited scientific journals are mentioned in this sub-section. Figure 3 and Figure 4 categorize journals according to these two criteria in order to show that they represent journals that exhibit the utmost relevance to RL in production scheduling.

The International Journal of Production Research, Journal of Manufacturing Systems and Computers & Industrial Engineering along with the nine others are considered the most relevant and respected scientific publications in this field. It can also be noted that all twelve journals listed in Figure 3 and Figure 4, regardless of the results of the metrics used, can be empirically ranked as widely recognized for disseminating advanced research on reinforcement learning applied to production scheduling. In addition, their scientific rigor is also indicated by the fact that out of the twelve identified journals, eleven met high standards for quality and impact, and are indexed for Current Content Connect journals with a verifiable impact on steering research practices and behaviors [69,70].

4.3. Most Cited Authors

Since the1998s, many authors have made significant contributions to the development of this field. In this sub-section, the intention is to present some of those authors who made significant intellectual contributions to the research. The analysis of the most cited authors was performed using data from WoS - Core Collection database. In addition, co-citation analysis was carried out for each publication source (out of the total 297 items) to reveal the network between the studies. For this purpose, VOSviewer software [71] has been applied. To obtain relevant information and clear graphic representation of complex relations, the following filters were employed. Filter 1: Maximum number of authors per document – 10; Filter 2: Minimum number of documents of an author – 3; Filter 3: Minimum number of citations of an author – 10. Moreover, the full counting method has been applied meaning that the publications that have co-authors from multiple countries are counted as a full publication for each of those countries. The co-citation network of the selected sample of scholars using these settings is visualized in Figure 5.

This co-citation network map shows, among other things, scholars that have received the highest number of citations in the last 26 years. Of the 744 cited authors, 73 meet the above mentioned criteria. Each scholar from each included publication is represented by a node in this network. The size of each node indicates frequency of citation of the subject’s scholarly works. An edge is drawn between two nodes if the two scholars were cited by a common document. To rank influential scholars in the given domain based on their citation rates, the ten most cited authors were selected. Those ten authors are listed in Table 2.

4.4. Co-Occurrence Keywords

The goal of this part of the article is to identify areas of research in which the issue of production scheduling based on RL is of interest. For this purpose, the keyword analysis application has been employed to help separate important research themes which has received a high interest of researchers from less important ones. To obtain relevant and representative categories not including less significant ones, the following setting has been used: Minimum number of occurrence of keywords – 14. Based on this restriction, 20 keywords meet the threshold from the total 1 075 keywords. Those main keywords that were produced automatically from the titles in the papers on production scheduling based on RL along with their occurrence are shown in Figure 6.

As can be seen, topics can be divided into three clusters based on a computer algorithm, while each cluster has a different color as shown in Figure 6. When the colors are mixed, it means than the algorithm couldn’t make clear differences between the clusters, and the clustering results does not meet the analytical target from the practical perspective. In this case, the co-occurrence keywords network map allows to identify relevant research topics and their mutual relationships. Based on the obtained bibliometric results extracted from VOSviewer as well as our empirical experiences, ten related topics that are very close to the explored research domain were identified as shown in Table 3.

From this table one can easily find that the related terms closely correspond to the associated research disciplines that are identified in Table 1. Moreover, the results of this subsection seems to be consistent with practical observations and extant literature. For example, in recent years, there has been evidently increased interest in using deep reinforcement learning for optimization of real-time job scheduling tasks [72,73,74,75]. This fact can be correlated with the continuing trend of mass customization in the production of consumer goods [76,77]. As known, for mass customization is characteristic to meet dynamically changing user requirements in time, while customized products need to be completed by different deadlines. Accordingly, efficient real-time job-scheduling algorithms based on DRL become essential. The second important method that is ranked among the top 10 co-occurrences keywords is simulation. Its importance results from the fact that only operation times are required to produce an optimal schedule. But in some practical application other components (such as, e.g., dispatch time and suspend time) may play important role. Therefore, validation of proposed solutions through simulation is often used (see, e.g., [78,79,80,81]). The next important co-occurred keyword in Table 3 - smart manufacturing represents the implementation domain of production scheduling based on RL. Despite the fact that smart manufacturing has also become a buzzword, which also has its drawbacks, this conception is gradually being established as the new manufacturing paradigm. On the other hand, complexity of smart manufacturing network infrastructures becomes higher and higher, and the uncertainty of such manufacturing environment becomes a serious problem [25]. These facts lead to the necessity of applying advanced dynamic planning solutions that also includes production scheduling using RL. This paragraph simultaneously answers to the main research question formulated in Section 1.

5. Conclusion

The examination of bibliometric findings frequently indicates that an increase in the quantity of published articles is associated with recognition of progressive trends in the subject. Also for this reason, a bibliometric analysis is becoming more and more beneficial in a variety of academic fields since it makes mapping scientific information and analyzing research development objective and repeatable. The use of this method enables us to identify the networks of scientific collaboration, to establish connections between novel study themes and research streams, as well as show the connections between citations, co-citations and published productivity in the field.

The existing body of research on production scheduling primarily consists of studies conducted within the domains of Engineering Manufacturing, Industrial Engineering, Operation Research, Computer Science, Artificial intelligence, and Engineering Electrical Electronic. These scientific disciplines, notably ’Engineering Manufacturing´ and ´Industrial Engineering´ have exerted significant influence on development of production scheduling based on RL.

In addition to the above mentioned findings it would be needed to focus on other challenges to be considered in the future such manufacturing process planning with integrated support for knowledge sharing, increasing demand for improvements in ubiquitous “smartness” in manufacturing processes including designing and implementing smart algorithms, and the need for robust scheduling tools for agile collaborative manufacturing systems.