Reinforcement Learning for Optimal Replenishment in Stochastic Assembly Systems

1. Introduction

In the context of supply chain and inventory management, planning plays a critical role in the effectiveness of replenishment strategies [1]. Well-designed planning processes help maintain optimal inventory levels, balancing the risk of overstocking—which leads to increased storage costs—with the risk of stockouts, which can cause lost sales and diminished customer satisfaction [2]. By ensuring the timely availability of products, components, or raw materials to meet production schedules or customer demand, effective planning contributes directly to improved service levels and enhanced customer loyalty [3]. These challenges are further compounded under conditions of uncertainty, where variability in demand and supplier lead times can significantly disrupt replenishment decisions.

In replenishment management, planning is essential to maintaining a balance between supply and demand, minimizing costs, and ensuring customer satisfaction [4]. Effective planning enables the optimal implementation of replenishment policies [5], aligning inventory decisions with strategic business objectives such as cost reduction and high product availability. These policies adjust order quantities and stock levels based on real-time market conditions. The effectiveness of such planning depends largely on its ability to adapt to various sources of uncertainty arising from collaborative operations between manufacturers and customers, interactions with suppliers of raw materials or critical components, and even internal manufacturing processes [6]. The nature of this uncertainty is multifaceted [7], often resulting in increased operational costs, reduced profitability, and diminished customer satisfaction [8]. Numerous studies emphasize key sources of uncertainty in manufacturing environments, including demand variability, fluctuations in supplier lead times, quality issues, and capacity constraints [9].

Demand uncertainty significantly impacts supply chain design. While stochastic programming models outperform deterministic approaches in optimizing strategic and tactical decisions [10], most studies overlook lead time variability caused by real-world disruptions. Companies typically address supply uncertainty through safety stocks and safety lead times [11], which trade off shortage risks against higher inventory costs. The key challenge lies in finding the optimal balance between these competing costs. For a long time, lead time uncertainty received relatively little attention in the literature, with most research in inventory management focusing predominantly on demand uncertainty [12]. In assembly systems, component lead times are often subject to uncertainty; they are rarely deterministic and typically exhibit variability [13].

The literature on stochastic lead times in assembly systems has seen significant contributions that have shaped current approaches to inventory control under uncertainty[14]. A notable study by [15] investigates a single-level assembly system under the assumptions of stochastic lead times, fixed and known demand, unlimited production capacity, a lot-for-lot policy, and a multi-period dynamic setting. In this work, lead times are modeled as independent and identically distributed (i.i.d.) discrete random variables. The authors focus on optimizing inventory policies by balancing component holding costs and backlogging costs for finished products, ultimately deriving optimal safety stock levels when all components share identical holding costs. This problem is further extended in [16], which considers a different replenishment strategy—the Periodic Order Quantity (POQ) policy. In [17], the lot-for-lot policy is retained but a service level constraint is introduced. A Branch and Bound algorithm is employed to manage the combinatorial complexity associated with lead time variability. Subsequent studies [18,19], and [20] build on this foundation by refining models to better capture lead time uncertainty in single-level assembly systems, while also proposing extensions to multi-level systems and providing a more detailed analysis of the trade-offs between holding and backlogging costs.

Modeling multi-product, multi-component assembly systems under demand uncertainty is inherently complex. [21] proposes a modular framework for supply planning optimization, though its effectiveness depends on computational reductions and assumptions about probability distributions. For Assembly-to-Order systems, [22] develops a cost-minimization model incorporating lead time uncertainty, solved via simulated annealing. Several studies [23,24,25] and [26] address single-period supply planning for two-level assembly systems with stochastic lead times and fixed end-product demand. Using Laplace transforms, evolutionary algorithms, and multi-objective methods, they optimize component release dates and safety lead times to minimize total expected costs (including backlogging and storage costs). [2] later improved upon [18]’s work, while [27] extended the framework to multi-level systems under similar assumptions.

Existing models often rely on oversimplified assumptions about delivery times and demand, limiting their practical applicability [21]. This highlights the need for new optimization frameworks that better capture real-world complexities and component interdependencies in assembly systems. We enhance [15]’s method by incorporating : (1) stochastic demand models, (2) ordering and stockout penalty costs, and (3) MDP-based stochastic modeling. Deep reinforcement learning techniques, are employed to optimize solutions under delivery and demand uncertainties.

Over the years, various modeling approaches have been proposed to address uncertainty [28], including: Conceptual models: Theoretical approaches to understand the relationships between variables, Analytical models: use of mathematical formulas to optimize decisions, Simulations: reproduction of system behavior to test different policies and Artificial intelligence: use of algorithms to predict and optimize decisions. The choice of approach depends on key characteristics of the manufacturing context.

Production Planning and Control (PPC) must combine rigorous planning with technological flexibility to adapt to the complex dynamics of supply chains [29]. The integration of artificial intelligence (AI), such as reinforcement learning (RL), and digital tools is now essential to achieve these objectives[30]. This trend aligns with Industry 4.0, where AI and machine learning play a central role in improving industrial efficiency. Industry 4.0 represents a major transformation in production systems, and PPC is evolving toward self-managing systems that combine automation (e.g., robots and smart sensors) with decision-making autonomy (e.g., AI and machine learning) [29]. This paper [31] presents a systematic review of 181 scientific articles exploring the application of reinforcement learning (RL) techniques in PPC. It provides a mapping of RL applications across five key areas of PPC: Resource planning, capacity planning, purchasing and supply management, production scheduling and Inventory Management.

This study develops a discrete inventory optimization model for single-level assembly systems (multi-component, multi-period) under stochastic conditions. Overcoming existing limitations, we: Integrate multiple logistics costs,Relax restrictive assumptions (uniform delivery distributions, fixed demand, uniform storage costs), introduce component-based stockout calculation, employ deep reinforcement learning for efficient implementation. The model features integer decision variables for MRP compatibility while addressing dual uncertainties (demand/delivery). Current scope remains assembly-level systems. The problem involves optimizing replenishment policies under uncertainty. We chose to use a Deep Q-Network (DQN) algorithm, a Deep Reinforcement Learning (DRL) approach that learns an optimal replenishment policy through interactions with the environment. We avoid traditional optimization methods because they are often unsuitable for inventory management problems with uncertain delivery times and stochastic demand [32].

The first challenge in optimizing replenishment policies under uncertainty lies in the inadequacy of classical optimization methods. We chose to use a Deep Q-Network (DQN) algorithm, a Deep Reinforcement Learning (DRL) approach that learns an optimal replenishment policy through interactions with the environment. We avoid traditional optimization methods because they are often unsuitable for inventory management problems with uncertain delivery times and stochastic demand [32]. Overly Simplistic Assumptions: Classical methods (e.g., deterministic or stochastic optimization) typically assume that problem parameters—such as delivery times and demand—are either deterministic or follow simple, easily exploitable distributions [33]. However, in real-world scenarios, delivery times are often random with complex probability distributions, making classical modeling approaches highly impractical. Adaptability to Uncertainty: Unlike traditional methods, DQN does not require explicit modeling of distributions. Instead, it dynamically adapts to uncertainties by learning from experience, making it more robust in stochastic environments.

The second challenge lies in the complexity and high dimensionality of the problem, which involves multiple dynamic factors: time-varying inventory levels, uncertain delivery times, stochastic demand, and multiple cost structures (e.g., holding, shortage, and ordering costs). Traditional optimization methods struggle with such complexity: Linear Programming (LP) becomes inapplicable due to the explosion of variables and constraints in realistic scenarios [34]. Classical Dynamic Programming (DP) suffers from the curse of dimensionality [35], as it requires storing and computing excessively large value tables [36]. These limitations motivate the use of Deep Q-Networks (DQN) [37], which leverage neural networks to approximate the $Q(s,a)$ function efficiently. Unlike DP, DQN avoids explicit state enumeration and instead generalizes across states, making it scalable to high-dimensional problems [38].

The third critical aspect involves the system’s dynamics and adaptability requirements in real industrial environments, where practical challenges emerge such as fluctuating periodic demand and unpredictable delivery times, necessitating dynamic decision-making. Traditional models prove inadequate for these scenarios as they typically employ static approaches or require computationally intensive re-optimization processes [39]. In contrast, the Deep Q-Network (DQN) framework dynamically adapts its policy through continuous learning from system observations, enabling it to develop optimal sequential decision strategies that effectively minimize long-term operational costs without explicit re-optimization [40].

This paper is organized as follows: Section 1 provides an introduction to the research context and outlines relevant work in supply planning under uncertainty, emphasizing key challenges and gaps in existing approaches. Section 2 describes the problem in detail, including the characteristics of the inventory environment and the sources of uncertainty. Section 3 presents the proposed methodology, including the formulation of the problem as a Markov Decision Process (MDP). Section 4 introduces the Deep Q-Network (DQN) algorithm and explains its implementation for learning optimal replenishment policies. Section 5 discusses the experimental results and provides an analysis of the findings. Finally, Section 6 concludes the paper by summarizing the main contributions, acknowledging limitations, and suggesting directions for future research.

2. Problem Description

Replenishment planning in single-level assembly systems under stochastic demand and lead time uncertainty presents a complex optimization problem where component orders must be determined amid two key sources of variability: (1) uncertain demand for the finished product, and (2) random lead times for each component. The core optimization challenge involves minimizing the total expected costs comprising inventory holding costs, stockout penalties and ordering costs. Crucially, demand follows a known probability distribution, while each component’s lead time is characterized by its own distinct distribution. These stochastic elements create a cascading risk effect - the failure to secure any single component due to lead time variability can halt the entire assembly process. The fundamental objective is to develop an optimal ordering strategy that achieves robust system performance while maintaining cost efficiency under uncertainty, requiring careful consideration of both demand-side and supply-side stochasticity in an integrated framework.

Figure 1 illustrates a single-level Bill of Materials (BOM) for an assembly system, depicting the relationship between a finished product (FP) and its components (i₁, i₂, ..., i_n). The diagram shows that one unit of FP is assembled from multiple components, each with a specific consumption coefficient (c_i), representing the quantity required per unit of FP.

3. Methodology

To model the replenishment planning problem under lead-time uncertainty as a reinforcement learning (RL) task, we formulate it as a Markov Decision Process (MDP). The MDP framework captures the dynamics of inventory management, where the agent makes decisions on ordering and stock management at each time step. The reward function is designed to incorporate key cost components, including storage costs for each component, the shortage cost of finished products, penalties for stockouts of components, and the ordering costs associated with replenishment decisions.

3.1. MDP Environment

The environment in a Markov Decision Process (MDP) is everything external to the agent. It defines how the world responds to the agent’s actions and evolves over time.

Structure of the Environment

At each time step t:

• The agent observes the current state $S^{\left(t\right)}\in \mathcal{S}$.
• It chooses an action $a^{\left(t\right)}\in \mathcal{A}$.
• The environment responds:

  –

  It transitions to a new state $S^{(t+1)}\sim P(·|S^{\left(t\right)},a^{\left(t\right)})$.

  –

  It emits a reward $R^{\left(t\right)}=R(S^{\left(t\right)},a^{\left(t\right)})$.

Table 1. List of Variables and Notations.

Symbol	Description
PF	Finished product.
i	component.
t	Period.
$D^{\left(t\right)}$	Random demand for finished products at period t, modeled as a stochastic variable with a known probability distribution.
$L_i^{\left(t\right)}$	Delivery time of component i ordered at period t, a random variable with a known probability distribution.
$Q_i^{\left(t\right)}$	Quantity ordered of component i at period t.
$S_i^{\left(t\right)}$	Stock level of component i at the end of period t.
$Q{r}_i^{\left(t\right)}$	Quantity of component i received at period t.
$c_i$	Quantity of component i needed to produce one unit of finished product (consumption coefficient ).
$h_i$	Unit holding cost per period for component i.
$p_i$	Unit shortage cost for component i.
${\alpha }_i$	Empty stock cost for component i.
${\beta }_i$	Order placement cost for component i.
M	Maximum order quantity permitted for each component per period.
T	Planning horizon (number of periods).
N	Number of components needed to assemble the finished product.
$\mathcal{A}$	Space of actions.
$\mathcal{S}$	Space of states.

Table 1. List of Variables and Notations.

Symbol	Description
PF	Finished product.
i	component.
t	Period.
$D^{\left(t\right)}$	Random demand for finished products at period t, modeled as a stochastic variable with a known probability distribution.
$L_i^{\left(t\right)}$	Delivery time of component i ordered at period t, a random variable with a known probability distribution.
$Q_i^{\left(t\right)}$	Quantity ordered of component i at period t.
$S_i^{\left(t\right)}$	Stock level of component i at the end of period t.
$Q{r}_i^{\left(t\right)}$	Quantity of component i received at period t.
$c_i$	Quantity of component i needed to produce one unit of finished product (consumption coefficient ).
$h_i$	Unit holding cost per period for component i.
$p_i$	Unit shortage cost for component i.
${\alpha }_i$	Empty stock cost for component i.
${\beta }_i$	Order placement cost for component i.
M	Maximum order quantity permitted for each component per period.
T	Planning horizon (number of periods).
N	Number of components needed to assemble the finished product.
$\mathcal{A}$	Space of actions.
$\mathcal{S}$	Space of states.

Figure 2 represents a reinforcement learning process or a dynamic decision-making system in the context of inventory management. It is divided into four main components, arranged in a circular flow to illustrate the sequence of steps in the process. Each component is represented by a colored circle, and the transitions between them are shown with curved arrows.

States

The state of the system can be represented by a vector of the current stock levels for all components i.

$$S^{\left(t\right)}=[S_1^{\left(t\right)},S_2^{\left(t\right)},\dots ,S_N^{\left(t\right)}]$$

(1)

Actions

An action $a^{\left(t\right)}=(a_1^{\left(t\right)},a_2^{\left(t\right)},\dots a_{\left(i\right)}^{\left(t\right)}\dots ,a_{\left(N\right)}^{\left(t\right)})\in$$\mathcal{A}$ represents the decisions regarding the quantities to be ordered at each period t. Each $a_i^{\left(t\right)}$ corresponds to the quantity ordered for each component at the beginning of the period.

$$a^{\left(t\right)}=Q^{\left(t\right)}=[Q_1^{\left(t\right)},Q_2^{\left(t\right)},\dots ,Q_N^{\left(t\right)}]$$

(2)

This is the vector of quantities ordered for each component i at period $\left(t\right)$. $\mathcal{A}=\{0,1,2,\dots ,M\}$ denotes the initial action space (discrete values). Each component i can take any value in the interval $\left[\right[0,M\left]\right]$. To ensure that the algorithm adjusts ordered quantities based on the specific characteristics of each component and avoids restricted or similar order quantities, it is important to define a flexible action space. A poorly defined action space (i.e., limited or uniform order quantities) could reduce the algorithm’s adaptability.

To address this, we dynamically adjust the action range for each component, ensuring sufficient diversity in possible order quantities. This allows the algorithm to adapt orders based on each component’s specific needs.

If the action space were too limited or similar across components, the algorithm might lack the flexibility needed to optimize orders effectively. To prevent this, we assign different quantity ranges to each component based on its importance. Instead of using a fixed set of possible actions, we adapt order ranges according to consumption coefficients and maximum delivery times. Components with higher consumption and longer delivery times have wider action ranges, allowing the algorithm to order more or less as needed. This flexibility enhances the algorithm’s ability to optimize stock management.

So to adjust the action space for each component based on its consumption coefficient and maximum delivery time, it is useful to dynamically adapt the action space according to the consumption of each component $\left(c_{i}D\right)$ and its delivery period ($L_i$).

Scaling the Action Space

The more a component is consumed and the longer it takes to be delivered, the larger its action range should be for better stock management. This can be achieved by adjusting the upper limit of the action space for each component.

Action Space Modeling Logic

Instead of defining a fixed-size discrete space for each component (e.g., from 0 to $\mathbb{N}$), we can implement an adaptive action space tailored to each component’s characteristics.

Action Space for a Component

$a_i\in [0,ma{x}_{orderi}]$    with $ma{x}_{orderi}=$ Estimated requirement = $c_i\times maxD\times max{L}_i$

This strategy is cautious, as it ensures that the model accounts for scenarios where demand is at its peak and delivery times are at their longest. By considering only the maximum demand and maximum delivery time, we prevent underestimating needs, even in extreme conditions, by creating an action space that includes all possible situations. In cases of overestimation, the agent will learn an optimal policy that naturally avoids unnecessary actions.

Global Action Space

$$\mathcal{A}=[0,ma{x}_{order1}]\times [0,ma{x}_{order2}]\times \dots \times [0,ma{x}_{orderM}]$$

where × represents the Cartesian product.

Transition Function

The expression of the transition function in a replenishment planning problem under uncertainty of random delivery times is used to describe how the state of the system evolves from one period to another, depending on the realizations of the random variables (delivery times), inventory levels, decisions already made (order quantities), and demand D.

$$P=P\left(S^{\left(t\right)}\right|S^{(t-1)},a^{(t-1)})$$

(3)

Formally, the transition function can be written as

$$P=\prod _{i=1}^{N}P\left(S_i^{\left(t\right)}\right|S_i^{(t-1)},a^{(t-1)})$$

(4)

Inventory Level Formula

The inventory level for each component i at the end of period t is given by

$$S_i^{\left(t\right)}=S_i^{(t-1)}+R_i^{\left(t\right)}-c_i{D}^{\left(t\right)}$$

(5)

• Let D be a random variable representing the demand for the finished product. It follows a discrete distribution on $\{1,2,\dots ,v\}$, with known associated probabilities:

  $$P(D=k),fork\in \{1,2,\dots ,v\}.$$
• The random variable $L_i$ represents the delivery time of component ii, which follows a discrete probability distribution on the set $\{1,2,\dots ,m\}$ with the associated probabilities:

  $$P(L_i^{\left(t\right)}=k),fork\in \{1,2,\dots ,m\}.$$

Computation of Receipts

The quantity received $Q{r}_i^{\left(t\right)}$ is the sum of orders placed in previous periods that are delivered in period t.

$$Q{r}_i^{\left(t\right)}=\sum _{t^{\prime }=1}^t{Q}_i^{\left(t^{\prime }\right)}.1_{\{L_i^{\left(t^{\prime }\right)}=t-t^{\prime }\}}$$

(6)

• $1_{L_i\left(k\right)=t-t^{\prime }}$ : An indicator function that is equal to 1 if the order for component i placed in period $t^{\prime }$ is delivered in period t and 0 otherwise, i.e. an indicator that is equal to 1 if the delivery time $L_i^{\left(t^{\prime }\right)}$ for the order for component i placed in period k is exactly equal $t-t^{\prime }$, and 0 otherwise.

Reward Function

The reward for a period t is defined as the inverse of the total cost:

$$R^{\left(t\right)}=-C_{total}^{\left(t\right)}$$

(7)

3.2. Storage Cost

Storage cost is the cost associated with maintaining component inventory in the warehouse. It is calculated based on the number of units of the components and the unit storage cost. The following formula calculates the storage cost associated with components $i=1,2,3,\dots N$.

$$C_{Storage}^{\left(t\right)}=\sum _{i=1}^N{h}_i.max(S_i^{\left(t\right)},0)$$

(8)

• $h_i.max(S_i^{\left(t\right)},0)$ : Represents the storage cost for each component when the stock level is positive.
  We add the storage costs of the components only if their stock levels are positive ( $S_i^{\left(t\right)}>0$), that is, if the components are in stock.

3.3. Shortage Cost

The cost of shortage is calculated based on the inventory levels of the components (based on the number of missing component units) and the shortage cost per component unit $p_i$. The shortage cost is the cost associated with the lack of components needed to produce the finished product, which can lead to stockouts and lost sales. So the unavailability of components "component shortage" can lead to a disruption in the production of the finished product.

$$C_{shortage}\left(t\right)=\sum _{i=1}^N{p}_i.max(-S_i^{\left(t\right)},0)$$

(9)

• $p_i.max(-S_i^{\left(t\right)},0)$ : Represents the shortage cost for each component. This shortage is linked to the lack of component i, when its stock level is negative.

We therefore add the costs of component shortages, only if the stock is insufficient ($S_i^{\left(t\right)}<0$), a negative stock level for a component i indicates an inability to satisfy the production of the finished product.

3.4. Empty Inventory Cost (Zero Inventory)

Empty inventory represents a situation where replenishment is needed quickly. It is not a complete shortage (because there is still time to react), but it is a situation that could quickly lead to a shortage. The inventory cost is defined by

$$Cos{t}_{emptyinventory}=\sum _{i=1}^N{\alpha }_i{1}_{\{S_i^{\left(t\right)}=0\}}$$

(10)

3.5. Order Launch Cost

The order launch cost (or order placement cost) refers to all the expenses associated with issuing and processing an order. The order launch cost is paid each time an order of i is launched at t so if $Q_i^t>0$

$$Cos{t}_{placement}=\sum _{i=1}^N{\beta }_i{Q}_i^{\left(t\right)}{1}_{\{Q_i^{\left(t\right)}>0\}}$$

(11)

3.6. Total Cost

$$\begin{array}{c}\hfill C_{total}^{\left(t\right)}=\sum _{i=1}^N{h}_i.max(S_i^{\left(t\right)},0)+\sum _{i=1}^{N}p.max(-S_i^{\left(t\right)},0))+\sum _{i=1}^N{\alpha }_i{1}_{\{S_i^{\left(t\right)}=0\}}+\sum _{i=1}^N{\beta }_i{Q}_i^{\left(t\right)}{1}_{\{Q_i^{\left(t\right)}>0\}}\end{array}$$

(12)

The total cost function is the sum of inventory cost plus shortage cost and order release cost over the planning horizon. Inventory cost increases when inventories are high, while shortage cost increases when inventories are low.

3.7. Optimal Policy

A function that determines the optimal quantity $Q^{\left(t\right)}$ to order in each period to minimize the expected total cost over horizon T :

$${\pi }^{*}\left(S\right)=arg\underset{\pi }{min}\mathbb{E}\left[\sum _{t=1}^T\left(C_{\mathrm{total}}^{\left(t\right)}\right)\right]$$

(13)

The policy $\pi \left(S\right)$ is a function that defines the quantity to order for each state $S^{\left(t\right)}$ Your objective is to find the policy ${\pi }^{*}$ that minimizes the expected total cost across all periods $t\in \{1,2,\dots ,T\}$. So the policy defines how to choose the quantities to order based on the current state of the environment (e.g. current inventory levels, lead time probabilities, etc.).

Objective:

Minimize the cumulative reward function R over the planning horizon , which represents the total cost by adapting the ordering policy $\pi$ to minimize inventory and stockout costs. This model takes into account the stochastic demand for the finished product, random lead times of components with specified probability distributions, and seeks to determine the optimal order quantities $Q_i$ to minimize inventory, stockout and order release costs.

4. DQN (Deep Q-Network)

Deep Reinforcement Learning (Deep RL) is a combination of Reinforcement Learning (RL) and Deep Neural Networks (Deep Learning) [41]. It allows an agent to learn optimal strategies in complex environments using powerful nonlinear approximations. Deep RL can solve complex problems (games, robotics, NLP) thanks to the power of deep networks. Major challenges (instability, divergence) have been partially addressed by techniques such as DQN [42].

4.1. Structure of the DQN Algorithm

The implemented Deep Q-Network (DQN) algorithm is structured around four core components. The first is the Replay Buffer, which stores past transitions in the form $(state,action,reward,next\_state,done)$ and enables the agent to learn from a randomized mini-batch of experiences, thereby reducing temporal correlation and stabilizing learning. The second component is the Q-Network, a feedforward neural network consisting of an input layer, a hidden layer with ReLU activations, and an output layer that approximates Q-values for each possible action. The third module is the DQN Agent, which initializes the Q-Network and Target Network, sets essential hyperparameters (e.g., learning rate, discount factor $\gamma$, and exploration rate $ϵ$), and manages the optimization and interaction with the replay buffer. Finally, the Action Selection mechanism adopts an epsilon-greedy strategy to balance exploration and exploitation, selecting either a random action with probability $ϵ$ or the action with the maximum estimated Q-value. This architecture provides a modular and efficient foundation for deep reinforcement learning in environments with discrete and multi-dimensional action spaces.

Algorithm 1 Replay Buffer

1: Initialize ReplayBuffer with capacity 2: Create a buffer with max length = capacity 3: function Push(state, action, reward, next_state, done) 4:     Store $(state,action,reward,next\_state,done)$ in the buffer 5: end function 6: function Sample(batch_size) 7:     Randomly select batch_size samples from the buffer 8:     Convert to tensors and return 9: end function 10: function Length 11:     Return the number of elements in the buffer 12: end function

Algorithm 2 Q-Network Forward Pass

1: Initialize QNetwork with state_dim, action_dim, hidden_dim 2: Define three fully connected layers: 3:    fc1: Input layer 4:    fc2: Hidden layer 5:    fc3: Output layer 6: function Forward(state) 7:     $x\leftarrow$ ReLU(fc1(state)) 8:     $x\leftarrow$ ReLU(fc2(x)) 9:     Return fc3(x) 10: end function

Algorithm 3 DQN Agent

1: Initialize state_dim, action_dims, hidden_dim, lr, gamma, epsilon, epsilon_min, epsilon_decay 2: Compute num_actions as product of action_dims 3: Initialize Q-Network and Target-Network 4: Copy weights from Q-Network to Target-Network 5: Initialize optimizer with learning rate $lr$ 6: Initialize Replay Buffer with capacity = 5000

Algorithm 4 Select Action

1: function SelectAction(state) 2:     if random() < epsilon then 3:         Choose a random action 4:     else 5:         Convert state to tensor 6:         Compute Q-values using Q-Network 7:         Select action with max Q-value 8:         Convert it to MultiDiscrete action 9:     end if 10:     Return action 11: end function

4.2. Comprehensive Comparison of Inventory Planning Methods

Table 2 compares classical models, optimization methods, machine learning approaches, and metaheuristics for inventory and replenishment planning.

Table 3 provides a comprehensive summary of the key parameters used in both the inventory environment and the Deep Q-Network (DQN) agent, outlining their values and respective roles in the learning and decision-making processes.

5. Results and Discussion

The environment parameters define the operational characteristics of the inventory system, such as the number of components, penalty and holding costs, demand and lead time uncertainties, and the structure of consumption. These parameters ensure the simulation accurately reflects the complexities of real-world supply chains. The agent parameters govern the learning process, including the neural network architecture, learning rate, exploration behavior, and training configuration. The inclusion of a replay buffer further enhances learning stability by breaking temporal correlations in the training data. The DQN algorithm is employed to learn optimal ordering policies over time. By interacting with the environment through episodes, the agent receives feedback in the form of rewards, which reflect the balance between minimizing shortage penalties and storage costs. To facilitate the simulation and make the analysis more interpretable, simple parameter values were used in the first table, including, limited lead time possibilities, and a small range of demand values. Additionally, the reward function was normalized (reward divided by 1000) to simplify numerical computations and stabilize the learning process. Despite this simplification, the implemented environment and reinforcement learning algorithms are fully generalizable: they can handle any number of components (n), multiple lead time scenarios, and a range of demand values (m), making the model adaptable to more complex and realistic settings. However, scaling up to larger instances requires significant computational resources, as the state and action spaces grow exponentially. Therefore, a powerful machine is recommended to run simulations efficiently when moving beyond the simplified test case.

The algorithm is developed using the Python programming language, which is particularly well-suited for research in reinforcement learning and operations management due to its expressive syntax and the availability of advanced scientific libraries. The implementation employs NumPy for efficient numerical computation, particularly for manipulating multidimensional arrays and performing vectorized operations that are essential for simulating the environment and computing rewards. Visualization of training performance and inventory dynamics is facilitated through Matplotlib, which provides robust tools for plotting time series and comparative analyses. The core reinforcement learning model is implemented using PyTorch, a state-of-the-art deep learning framework that supports dynamic computation graphs and automatic differentiation. PyTorch is used to construct and train the deep Q-network (DQN), manage the neural network architecture, optimize parameters through backpropagation, and handle experience replay to stabilize the learning process. Collectively, these libraries provide a powerful and flexible platform for modeling and solving complex inventory optimization problems under stochastic conditions.

Performance and Policy Evaluation

Figure 3 shows the evolution of rewards over training episodes. Initially, the agent exhibits high variability in performance due to random actions and limited experience. As training progresses, particularly after episode 100, the rewards stabilize and approach zero, indicating that the agent has learned a policy that effectively minimizes inventory-related costs.

Figure 4 illustrates the agent’s learned ordering policies over time. Early in training, the ordered quantities fluctuate significantly, reflecting exploration. Eventually, the policy converges toward more stable ordering behaviors. Component 2 is ordered most frequently and in larger quantities, suggesting its higher importance or greater risk of shortage. Component 3 is ordered less often, possibly due to lower consumption or more favorable lead times.

The evolution of average stock levels is presented in Figure 5. Initially, the stock levels are volatile and include frequent shortages (negative values). Over time, the agent learns to anticipate demand and lead times better, leading to smoother inventory levels. The final stock levels—higher for component 2—reflect its strategic importance and confirm the ordering strategy learned.

The results clearly demonstrate the capability of the DQN algorithm to adaptively optimize inventory policies under uncertainty. The convergence of rewards, the consistency in ordering patterns, and the stabilized stock levels collectively validate the effectiveness of the learning process.

Lead Time and Risk Analysis

The three components in the inventory system exhibit distinct patterns in both consumption and lead time variability, which significantly influence their associated risks. Component $i_1$, with the highest consumption coefficient, demonstrates a relatively stable delivery profile: 80 % of its orders are fulfilled within 1 or 2 periods, making it the least risky in terms of delays. In contrast, Component $i_2$, which has a moderate consumption rate of 2, faces greater uncertainty— 30% of its orders experience a delay of 3 periods, and 40% are delayed by 2 periods. This combination indicates a substantial risk of stockout if not managed carefully. Component $i_3$, despite its lower consumption rate of 1, shares the same 30% probability of a 3-period delay as $i_2$ but has only a 10% chance of being delivered within 1 period, the lowest among all components. This makes $i_3$ the most vulnerable to supply disruptions. Overall, the risk of delivery delays is highest for $i_3$, followed closely by $i_2$, while $i_1$ remains the most reliable in terms of lead time performance.

5.1. Impact of Lead Times on Ordered Quantities

When a component has a longer and more uncertain lead time, the agent tends to adapt its ordering strategy to mitigate the risk of stockouts. This often results in placing orders more frequently in anticipation of potential delays. Additionally, the agent may choose to maintain a higher inventory level as a buffer, ensuring that production is not interrupted due to late deliveries.

Table 4 presents a detailed analysis of the relationship between each component’s consumption level, its lead time uncertainty, and the resulting ordering behavior by the agent. Components with higher consumption but more reliable lead times, such as $i_1$, allow for stable and moderate ordering. In contrast, components like $i_2$ and especially $i_3$, which are subject to longer and more uncertain delivery delays, require the agent to compensate by increasing order frequency . This strategic adjustment helps to avoid shortages despite the variability in supplier lead times. The table summarizes these insights for each component.

To better understand the agent’s ordering behavior, Table 5 addresses key questions regarding the observed stock levels for each component. Although one might expect ordering decisions to align directly withconsumption coefficient, this is not always the case. In reality, the agent adjusts order quantities and stock levels by taking into account both the demand and the uncertainty in lead times. As shown below, components with higher risk of delivery delays tend to be stocked more heavily, even if their consumption coefficient is relatively low, while components with reliable lead times require less buffer stock.

Summary of Effects

Table 6. Summary of lead time effects on stock levels.

${\mathit{i}}_{\mathit{i}}$	${\mathit{c}}_{\mathit{i}}$	Lead Time	Agent’s Strategy
$i_1$	High	Stable (80% ≤ 2 periods)	Moderate stock, regular orders
$i_2$	Medium	30% risk of 3-period delay	Higher stock to avoid shortages
$i_3$	Low	Highly uncertain (30% in 3 periods)	Higher stock than expected

Table 6. Summary of lead time effects on stock levels.

${\mathit{i}}_{\mathit{i}}$	${\mathit{c}}_{\mathit{i}}$	Lead Time	Agent’s Strategy
$i_1$	High	Stable (80% ≤ 2 periods)	Moderate stock, regular orders
$i_2$	Medium	30% risk of 3-period delay	Higher stock to avoid shortages
$i_3$	Low	Highly uncertain (30% in 3 periods)	Higher stock than expected

Final Observation: The agent adjusts decisions based on lead time risks, not just consumption rates.

5.2. Impact of Uncertain Demand on Order Quantities

The expected demand for the finished product is:

$$E\left[D\right]=(1\times 0.2)+(2\times 0.5)+(3\times 0.3)=2.1$$

Table 7. Expected Demand per Component.

i	${\mathit{c}}_{\mathit{i}}$	Formula	Expected Required Quantity
$i_1$	3	$3\times 2.1$	6.3
$i_2$	2	$2\times 2.1$	4.2
$i_3$	1	$1\times 2.1$	2.1

Table 7. Expected Demand per Component.

i	${\mathit{c}}_{\mathit{i}}$	Formula	Expected Required Quantity
$i_1$	3	$3\times 2.1$	6.3
$i_2$	2	$2\times 2.1$	4.2
$i_3$	1	$1\times 2.1$	2.1

This suggests that, in an ideal case without lead time risks, the order quantities should follow the ratio :

$$Q_1>Q_2>Q_3$$

Variability in Demand and Its Effect

In addition to lead time uncertainties, the agent must also consider variability in demand and cost-related trade-offs when determining order quantities. The decision-making process is not only influenced by the likelihood of delayed deliveries but also by how demand fluctuates over time and how different types of costs interact. Table 8 outlines the key factors that affect the agent’s ordering behavior and explains their respective impacts on the optimal policy formulation.

Interplay Between Demand Uncertainty and Lead Time Risks

If the observed order quantities do not strictly follow the expected pattern $Q_1>Q_2>Q_3$, this may be due to the agent anticipating delivery delays and adjusting order sizes accordingly. Such adjustments also reflect an effort to minimize shortage costs, potentially resulting in overstocking components with uncertain lead times. Additionally, the agent may adopt a dynamic policy that evolves over time in response to past shortages, modifying future decisions based on observed system performance. Demand uncertainty forces the agent to carefully balance risk and cost, and while component $C_1$ should theoretically be ordered the most due to its high consumption, the impact of lead time variability and the need to hedge against delays can significantly alter this behavior.

Good Points in Our Model:

• Model convergence: The total reward stabilizes after approximately 100 episodes, indicating that the agent has found an efficient replenishment policy.
• Improved ordering decisions: The ordered quantities for each component become more consistent, avoiding excessive fluctuations observed at the beginning.
• Reduction of stockouts: Despite some variations, the average stock levels tend to remain positive, meaning the agent learns to anticipate demand and delivery lead times.
• Adaptation to uncertainties: The agent appears to adapt to random lead times and adjusts its orders accordingly.

6. Conclusion

This study presents significant advancements in the application of reinforcement learning techniques to replenishment planning under uncertainty. A novel discrete-time Markov Decision Process (MDP) model is introduced for single-level assembly systems, accounting for random delivery lead times and stochastic demand. The optimization criterion integrates stockout costs, inventory holding costs, and ordering costs. To address this problem, a custom simulation environment is developed to model the dynamics of a replenishment system under uncertainty. An adapted Deep Q-Network (DQN) algorithm is then applied to derive optimal replenishment policies for each component. In addition, the integration of local search—through multiple action spaces specific to each component—accelerates convergence and significantly improves solution quality. Experimental results demonstrate the effectiveness and robustness of the proposed DQN-based approach.

Research Outlook : Future research should focus on extending the model to multi-level assembly systems, which poses additional challenges due to increased structural complexity and interdependencies among components. A key direction involves reformulating the total expected cost function to accommodate nonlinearities more explicitly. Further investigation may also include the integration of interpolation techniques and queueing mechanisms to improve production planning and scheduling under various sources of uncertainty. A critical need persists for the ongoing refinement of supply planning models to more effectively manage uncertainty, especially within complex, multi-tiered production systems. While this work contributes to addressing demand and lead time variability, future efforts should aim to incorporate broader uncertainty dimensions and enhance the scalability of the proposed methods. Advancing the balance between theoretical rigor and industrial applicability will be crucial for driving practical improvements in supply chain performance.

Ethical approval was not required for this study as it does not involve human participants or sensitive data.