Adaptive Traffic Signal Control Using Deep Reinforcement Learning with Noise Injection

1. Introduction

Traffic congestion is a phenomenon with a large impact around the world [1,2,3]. In the United States alone, cities such as Chicago, Boston, and New York have recorded the highest levels of congestion worldwide, with consequences for public health, induced by pollutant emissions, stress, and the time spent by drivers in congestion [4,5,6,7]. Mexico City ranks fourth in traffic congestion and also faces the variety of social problems derived from this phenomenon [7,8,9]. Traffic congestion also generates significant economic losses [10], for example, in the USA, the cost of traffic congestion resulted in a loss of $74 billion nationwide [7,11]. By 2030, this phenomenon is expected to be present in many more cities due to the growth of the population and the use of vehicles [12,13].

To reduce vehicle congestion at intersections, control of traffic lights is necessary [14,15]. However, the normal use of this technology, with fixed-time (FT) phases, cannot be very efficient because this strategy cannot respond to changing traffic situations [16]. The addition of sensors [17], allowed the design of Adaptive Traffic Signal Control (ATSC) systems [18]. During the early years of ATSC implementations, several techniques were proposed to optimize vehicular traffic. Among the most important are: Split Cycle Offset Optimization Technique (SCOOT) [19], Self-Organizing Traffic Lights (SOTL) [20,21,22], Real-Time Hierarchical Optimized Distributed and Effective System (RHODES) [23], and Sydney Coordinated Adaptive Traffic System (SCATS) [24], which were attempts to set more efficiently the timing of traffic lights. Although these techniques were functional during the 1980s and 1990s, they are not adequate to deal with the more complex scenarios that arise with increasing vehicular traffic.

Artificial Intelligence (AI) has significantly enhanced the performance of ATSC. Since 2015, there are more than 2500 publications related to ATCS, including mathematical modeling and AI applications [25]. Techniques such as Fuzzy Logic (FL) or Genetic Algorithms (GAs) have helped to reduce traffic congestion by designing more intelligent phase-timing patterns of traffic lights [26,27]. Other new AI techniques based on Machine Learning (ML) are also used. In particular, a widely used subset is Deep Learning (DL), which has achieved important advances in the area of vehicular traffic [28]. Deep Reinforcement Learning (DRL) arises from the combination of DL with Reinforcement Learning (RL) algorithms that are used in the learning stage [29,30]. RL incorporates neural networks (NN) that, considering vehicle demand, identify characteristics of traffic behavior that are very helpful in phase-timing design [31,32,33,34].

In general, DRL techniques can be classified as centralized, where a set of traffic lights communicate and provide feedback between them for better decision making [35], and decentralized, where traffic lights act independently of each other [36]. There is also research that proposes hybrid solutions using both paradigms [37]. Due to the expected increase in the number of connected and automated vehicles (CAVs), which do have a different interaction with infrastructure or vehicular traffic, analysis of technologies to control intersections also allows to propose behaviors that would not be possible in regular scenarios [38].

The complexity of DRL techniques in ATSC is related to the size of the traffic network to be analyzed and the intrinsic difficulties of the algorithmic implementation of the chosen DRL model. Complexity is greater in centralized techniques because they manage several intersections simultaneously. The results obtained to date indicate that there are still pending issues, such as long execution times, convergence problems, and the large number of variables to be considered, that limit their application to real situations [31,33,39,40,41]. There are DRL techniques that have been successfully applied to a set of single intersections. However, the reported results correspond to cases where intersections and traffic conditions are similar [42,43].

Single intersection studies use variables such as queue length, waiting time, vehicle spacing, specific vehicle type, or number of vehicles approaching the intersection as reference information, and work with the complexity of traffic and the given choice of variables [31]. They usually take traffic lights at the intersection as a single global agent [44]. The research in [27,45,46], shows a number of works where optimization is solved with metaheuristic-based algorithms.

Although most ATSC studies with DRL treat intersections as exclusive global agents or apply decentralized strategies to multiple intersection systems [47,48,49,50], few of them deal with the agent "anxiety" to prematurely change phase with small stochastic variations. There are some studies that use a decentralized approach at single intersections, modeling each traffic light as an independent agent and including the number of traffic light changes (flickers) as a specific variable [51]. This approach could allow better local coordination and traffic stability by solving the convergence problems that constrain traditional models, such as excessive changes in traffic lights that can affect traffic safety and stability [52].

It is still necessary to apply DRL models in single intersections with the intention of correcting erratic decision making and improving the stability of vehicular phases by combining flickers, waiting times, or vehicular queue sizes, as has been mentioned in [53,54,55]. Works such as [54,56,57], also confirm that these combinations have not yet been sufficiently addressed and suggest that, despite the need to deal with scenarios with multiple intersections, problems with single intersections need to be solved by expanding their complexity scale. Recently [56], mentioned that the use of stable phase mechanisms using variables such as flickers, has not been fully explored. In summary, the use of DRL models and algorithms in simple intersections has open problems that still need to be addressed.

In this paper, we present a novel optimization method in which we involve three elements: vehicle queue size, flickers, and Advantage Actor-Critic (A2C) RL with Noise Injection (NI), using a descentralized approach that takes traffic lights as individual agents, generating communication between the different traffic lights installed at a single intersection. This type of combination of variables, agents and communication at a single intersection has not been used before and justifies exploring its performance.

The aim of this work is to develop and evaluate a decentralized DRL framework for traffic signal control at a single intersection, incorporating a novel adaptive reward function. Although focused on a single intersection, this study is highly relevant, as it provides a foundational testbed for complex traffic control strategies, demonstrates the practical benefits of integrating multiple traffic factors, and establishes principles that can be extended to multi-intersection networks.

This work differs from existing approaches by focusing on traffic signal control at a single intersection using a decentralized DRL framework, demonstrating that even in a single-node scenario, decentralization enhances local adaptability. Unlike most studies, the proposed model integrates into the reward function traffic related variables, flickers and queue length, together with A2C learning results, providing a more comprehensive assessment of traffic dynamics. The use of A2C improves learning stability and convergence compared to traditional DRL methods. In this way, each intersection operates as an independent agent adapting to local traffic conditions. In addition, the adaptive reward function optimizes key metrics such as queue lengths and traffic balance. Overall, this methodology enables autonomous and adaptive phase optimization, reduces congestion and delays, and is scalable to multi-intersection networks while maintaining decentralized control. Finally, by grounding the model in realistic traffic data and simulations, the approach offers practical applicability and a closer approximation to real-world traffic management, setting it apart from many theoretical or simplified studies.

The main contributions of the proposed study are summarized as follows:

1.

A decentralized DRL framework with NI (ATSC-DRLNI) for single-intersection ATSC in which each traffic light operates as an independent agent adapting to local traffic conditions.

2.

A stability-oriented reward function that integrates queue length, flickers, and A2C learning feedback in an effort to mitigate premature phase changes and support smoother convergence.

3.

A comparative evaluation against seven baseline methods in Simulation of Urban MObility (SUMO) platform using real traffic data from a Mexican city for calibration and validation, suggesting improvements in queue reduction and phase stability.

4.

An effort toward narrowing the gap between simulation-based ATSC research and potential real-world applications that emphasizes the role of stability-related variables in decentralized DRL settings.

The article is organized as follows: In Section 2, a brief overview of the ML and the techniques to be implemented will be given. The general problem formulation, as well as the details of the proposed model and algorithm, are presented in Section 3. Simulations details and the results of the experiments performed to verify the effectiveness of the present model are presented in Section 4. Next, is Section 5, where the analysis and discussion of the highlights of this research is established. Finally, Section 6 presents the conclusions and future work on this line of research.

2. Preliminaries on Machine Learning

In this section, we will briefly explain some RL and DL preliminaries, to show how the DRL and its algorithms work, their functions and categories, and to describe the actor-critic problem [29].

2.1. Artificial Neural Networks (ANNs)

Artificial Neural Networks (ANNs) are statistical models belonging to ML that identify data characteristics and, from them, formulate predictions for decision making [58,59]. ANNs have the following form:

$$\widehat{y}=\left(\sum _{i=1}^n{x}_i·w_i+b\right)$$

(1)

where the input estimates x are multiplied by a weight value w given for each input with a bias b added. The process is repeated for the input and outputs of each layer of the ANN.

A type of ANN, named Convolutional Neural Network (CNN), is commonly used for image-oriented processing and training, where images can be transformed into one-dimensional and grayscale vectors, which are then processed pixel by pixel. Typically, CNN networks consist of several layers: fully connected layers, pooling layers, convolution layers, and nonlinear layers. The fully connected layers and the convolutional layers have parameters, while the nonlinear layers and the pooling layers do not have parameters [60,61].

2.2. Reinforcement Learning (RL)

RL is a discrete time Markov Decision Process (MDP) that allows an agent to make future decisions based on current observation [62,63]. A MDP is defined by 5-tuples $\{S,A,T,R,\gamma \}$, where S represents the set of states contained in the environment, A is the set of actions that an agent can take within the environment, T is a transition function, which is the probability of movement between the states of the environment, R is the reward/penalty function that guides the learning, and $\gamma \in [0,1]$ is a discount factor that impacts the weight of rewards or penalties in RL-oriented policies. This process is usually executed by trial and error and depends on the amount of information that is given to the model, its size, and the learning time.

The ultimate goal of using RL is to obtain the highest cumulative sum of agent’s learning, after an extended period of training [64]. Formally defined, the overall reward $G_t$ is the maximization at time t of the sum of rewards $R_t$ evaluated at different future times, according to

$$G_t=R_{t+1}+R_{t+2}+R_{t+3}...+R_T,$$

(2)

where T is a previously established final time step. This type of maximization is known as reward maximization.

A variation in the RL approach consists in the use of a discount factor, in which agents weigh the rewards at future times using a discount factor.

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}.$$

(3)

There are other forms oriented to maximize the cumulative reward by maximizing the actual value of the reward $R\left(t\right)$ for any given time t.

2.3. Q-Learning (DQN)

To obtain the optimal reward, Q-Learning is used that is based on evaluating the different agent’s actions using $\epsilon$-greedy, a learning policy based on the exploration of the agent possible actions. At initial times, random actions are explored as a trial-and-error learning activity [65,66]; as time progresses, the best-performing rewards are exploited and the best actions are selected. A $\epsilon$-greedy algorithm is easy to understand and provides greater accuracy in learning, given a training time. The updating function $Q{L}_t$ is of form

$$Q{L}_t(s,a)=Q{L}_{t-1}(s,a)+\alpha \left[R_t+\gamma \underset{\forall a^{\prime }}{max}\{Q{L}_t(s,a^{\prime })-Q{L}_{t-1}(s,a)\}\right],$$

(4)

where a are the actions, $\alpha \in [0,1]$ defines the learning rate or step size; if $\alpha =0$, the agent will never learn from the new actions, conversely if $\alpha \to 1$ the agent will favor recent information, as the value of $Q{L}_t(s,a)$ is modified more quickly. The innovation term depends on the reward $R_t$ and the choice of the action a that maximizes the growth of $Q{L}_t(s,a)$, weighted by $\gamma \in [0,1]$, a discount factor, which in Q-Learning determines the significance of the upcoming actions. If $\gamma$ is set to 0, the agent discards future rewards entirely and only considers current rewards, on the other hand, if $\gamma$ is set to 1, the agent will focus on getting rewards in the long run. A third parameter $\epsilon$ can be used to introduce randomness in the search space of the $\epsilon$-greedy action selection procedure of the Q-learning algorithm. In the action selection phase, a given action is selected based on the Q-values available [67,68].

2.4. Advantage Actor-Critic (A2C)

RL contains algorithms that are regularly divided into three groups: actor-only, critic-only and actor-critic, where the words actor and critic are synonyms for policy and function values [69,70,71]. Actor-only methods work with a set of parameterized policies that can be used directly in the optimization processes, while critic-only methods use time-difference (TD) learning, which has a smaller variance in expected estimates. This method is based on the choice of greedy actions, which suggests the choice of functions with the highest value in the expected performance. However, this process is computationally expensive when the action space is continuous. Advantage Actor-Critic (A2C) methods combine the advantages of actor-only and critic-only methods. This choice is capable of producing continuous actions and preserves the convergence properties of the policy gradient [70].

2.5. Neural Networks in Reinforcement Learning

RL algorithms can be implemented without NN, using tabular and iterative methods, which make them more intuitive but generate scale and precision problems when large amounts of data and many variables are considered. In the case in our ATSC application, the use of ANNs, using standard weight-adjustemt algorithms such as back-propagation, is appropriate.

To obtain robustness to variations during training, Noise Injection (NI) is added, which consists of the addition of noise in the input data of a ANN during the training phase to improve performance [72,73]. This noise is represented by a random disturbance $ϵ$, with a Gaussian distribution. The shape of the input with noise is represented as:

$$\tilde{x}=x+ϵ,\mathrm{where}ϵ\sim \mathcal{N}(0,{\sigma }^2)$$

(5)

where $\tilde{x}$ consists of the sum of the original data input x plus the random noise obtained from a normal distribution with mean 0 and variance ${\sigma }^2$. The input to the ANN is then $\tilde{x}$ instead of x.

3. Materials and Methods

In this section, we formulate the ATSC problem as a DRL task, making a clear definition of all elements, such as scenario behavior, reward function, actions, and states. We present the functions used, such as Experience Replay (ER) and DQN, and explain the training with NI and DRL. The general architecture is illustrated in Figure 1, where the model is named ATSC-DRLNI, while Table 1 contains the notation used in this proposal.

3.1. System Description

The ATSC-DRLNI model is based on the joint implementation of techniques belonging to DRL, and includes the CNN design, the data set (synthetic or real) processing, and the use of RL algorithms, in this case, ER and $\epsilon$-greedy policy. The general architecture is shown in Figure 1, and will be explained below and in the following sections.

The DL scheme is divided into two streams, one where the inputs are processed by four CNN layers connected in series, and another where the output of the third CNN is passed through a NI-layer, where gaussian noise is incorporated, and whose output is added to the output of the fourth CNN. The NI-layer will improve the behavior of the network, specifically with regularization, robustness, and performance. This NI-layer implements a rectified linear unit function (ReLU), that generates a flow with a discrete probability distribution. The final output vector is used in: i) the reward function, ii) to update the CNN weights with back-propagation at each step using the DQN algorithm, and iii) to create the batches and buffers to be used for ER learning. Finally, the $\epsilon$-greedy algorithm selects the actions.

3.2. Actions

The set of feasible actions include all possible directions and paths that the traffic light can adopt and, therefore, that an agent is free to choose from. For example, suppose that a traffic light change may occur within a given time frame T, and the agent starts a cycle with the green phase of the traffic light ($G_l$), then, depending on the state of traffic, the agent will move to the yellow phase ($Y_l$), and finally to the red phase of the traffic light ($R_l$). To maintain a stable vehicular flow, after a time for $R_l$ the agent should switch back to the $G_l$ repeat the cycle, noting that only one phase can be active at any time. Action ($G_l$, $Y_l$, $R_l$) of agent A at time t is denoted by $A_t$.

3.3. States

The state at time t, where an action is taken, is formed as a vector whose elements contain information from the environment, which is given by the specific features obtained from the traffic light sensors in use (length of vehicles, vehicle density, the distance between stopped and moving vehicles, etc). In our case, we use as information the total number of vehicles at a distance D from the intersection under analysis and associate this number also with the actions of the traffic light signals, e.g., $G_l$, $Y_l$ and $R_l$, and their respective time frames. Figure 2 illustrates the data flow, from states to Q-values.

3.4. Reward

The reward function is an essential component of the RL algorithm design, since it is the feedback index that evaluates the quality of the agent’s performance.

We implement a reward function $R_t$ at time t based on the combination of the vehicle queue, flickers, and the agent action $A_t$ as follows:

$$R_t=-{\displaystyle \frac{1}{p\left(A_t\right)}}·{\displaystyle \frac{Q_t}{Q_{norm}}}-{\displaystyle \frac{F_t}{F_{norm}}},$$

(6)

where $p\left(A_t\right)$ is the probability of action $A_t$, decided in the A2C policy described below, $Q_t$ the size of the vehicle’s queue in the scenario considered, $F_t$ the number of flickers at time t. $Q_{norm}$ and $F_{norm}$ are appropriate factors to scale the involved variables.

This reward function has an additional penalty included when there are no vehicles and a filcker is produced, in that case

$$R_t\leftarrow R_t-F_p$$

(7)

with $F_p$ the chosen penalty. The proposed reward then weighs the length of the queue and the number of flickers. A positive change in the reward indicates an improvement in the intersection condition, and conversely, a negative change in the reward shows deterioration. The reward function is used by the agents, in this case the traffic lights involved in the scenario that will be explained later on, and the interaction with the environment determines the policy of each agent. This reward function uses the DQN function presented before. To detect changes in the reward, the following time difference is introduced

$$S_t=R_t-R_{t-1}$$

(8)

3.5. The Neural Network Structure

The structure of the proposed NN model is illustrated in Figure 2. This NN is designed in such a way that it can process any state and produce a proper action. The input received comes from video cameras installed at several points before the intersection that, once processed, provide information about the number of vehicles at a certain distance D of the intersection, for the considered scenario, as will be explained later on. There are four CNN hidden layers aiming to maintain an adequate performance both in the NN and in the pre-processing. These layers include a NI-layer with a ReLU activation function. The number of output nodes for obtaining the Q-values functions is equivalent to the number of actions, which, in turn, are the triggers for obtaining the best reward. The NI-layer provides a statistical probability distribution by means of GN, which is added to the result of the last CNN. The output layer vector is the set of Q-values used to minimize the updating function $Q_t$ in Equation (4). Algorithm 1 describes the A2C policy optimizacion in the ATSC-DRLNI model.    

Algorithm 1: ATSC-DRLNI policy optimization

The policy ${\Pi }_0$ is initialized with a random stochastic value to calculate the function $S_0$. Then, given a state $S_t$ and an action $A_t$ carried out, the policy for time t, ${\Pi }_t$ is determined. The error term from one iteration to the next is computed in step 3 of the algorithm, where the goal is to adjust $S_t$ using $R_t$, the current value of the reward, and the previous value $S_{t-1}$.

We update the actor by adjusting the action probabilities for state $S_t$ based on the error term. If ${\delta }_t$ is positive, we must raise the probability of taking action $A_t$, because it increases the reward, conversely, if it is negative, we must lower the probability of taking action $A_t$, because choosing it results in a poorer performance in the objective function. This is done by the actor who knows the actions.

Updating the critic will adjust the estimated value of $S_t$ using $S_{t-1}$ and $\alpha$, the learning ratio of the critic, not the actor. Finally, the new state must be set and the algorithm is repeated from step 2.

4. Results

In this section, the scenarios and dataset for the simulation are established. To verify the performance of our proposal, simulation results comparing them with other well established methods available in the open literature are presented. A summary of the statistics of the general behavior obtained is also included.

4.1. Setup

To validate the functionality of the methodology, several simulations were carried out for different behaviors of vehicular traffic. The simulations were coded with the Simulation of Urban MObility (SUMO) sumo-gui version 1.10.0, Python 3.8.8, and OpenIA Gym version 0.7.4 and executed in a cluster with the following technical data: MSI Intel(R) Core(TM) i7- 10870H CPU @ 2.20GHz, 16 GB DDRA RAM, VRAM 8GB DDR6 NVIDIA RTX3600, and 3060 NVIDIA CUDA Core. Relevant intersection data used for the simulations are shown in Table 2.

4.2. Simulation Environment and Dataset

The traffic intersection depicted in Figure 3 is the environment used to train the RL algorithm. It represents an intersection in Aguascalientes, a city in Mexico, whose data are provided by the local Dirección de Tránsito y Vialidad and were obtained by an intelligent traffic system provided with cameras and sensors. Vehicle’s directions in the intersection are $N,S,E,W$, indicated as $N\leftrightarrow S$ and $E\leftrightarrow W$, implying that both directions are valid at the same time. Vehicle directions also includes turns, as vehicles can go from S to W ($S\to W$) and from N to E ($N\to E$).

Training simulations were executed in SUMO platform. The intersection has four phases with the following directions: Phase 1 (north-south) and (south-north); Phase 2 (east-west) and (west-east); Phase 3 (east-south) and (north-east); Phase 4 (east-north) and (west-north). As mentioned, all the input data information are from an intersection in Aguascalientes City, Mexico. Sensor information was available every 30 s, and data for a complete week were used, and training was performed using 4 hour subsets. For a complete 4-phase cycle (green-yellow-red), the average vehicle flow is 400. The vehicles generated by the simulator are probabilistically defined and matched to the data-set provided by the Dirección de Tránsito y Vialidad.

There are two vehicle traffic conditions, one corresponding to normal situations and another, for specific times of the day, to the case where demand is higher and congestion is to be expected. A combination of data including both conditions was used for system training in all simulations. Two scenarios were simulated: in the first one, phases 1 and 2 are optimized to give preference to straight traffic, while the second optimization is based on phases 3 and 4, which favor more complex maneuvers (turns to right and left) that are, in general, more prone to cause congestion. Special configurations for the right and left lanes, where vehicles execute turns, were also defined with the intention of representing real situations at urban intersections, where turns are a critical issue in traffic handling. The distance $D=800m$ selected to detect incoming vehicles at the intersection, represents a typical urban environment with relatively short distances between intersections and poses an additional challenge for the synchronization of traffic lights.

Table 3. Intersection scenario 1 parameters

Parameter	Value
Number of lanes	10
Traffic Flow	Random trips
Simulation platform	SUMO
Yellow phase duration	5 s
Initial Green phase duration	30 s
Distance D to intersection to detect vehicles	800 m
Value of $Q_{norm}$	5,000 vehicles
Value of $F_{norm}$	100 flicker
Value of $F_p$	0.2 flicker

Table 3. Intersection scenario 1 parameters

Parameter	Value
Number of lanes	10
Traffic Flow	Random trips
Simulation platform	SUMO
Yellow phase duration	5 s
Initial Green phase duration	30 s
Distance D to intersection to detect vehicles	800 m
Value of $Q_{norm}$	5,000 vehicles
Value of $F_{norm}$	100 flicker
Value of $F_p$	0.2 flicker

Table 4. Intersection scenario 2 parameters.

Parameter	Value
Number of lanes	14
Traffic Flow	Random trips
Simulation platform	SUMO
Yellow phase duration	5 s
Initial Green phase duration	30 s
Distance D to intersection to detect vehicles	800 m
Value of $Q_{norm}$	6,500 vehicles
Value of $F_{norm}$	100 flicker
Value of $F_p$	0.2 flicker

Table 4. Intersection scenario 2 parameters.

Parameter	Value
Number of lanes	14
Traffic Flow	Random trips
Simulation platform	SUMO
Yellow phase duration	5 s
Initial Green phase duration	30 s
Distance D to intersection to detect vehicles	800 m
Value of $Q_{norm}$	6,500 vehicles
Value of $F_{norm}$	100 flicker
Value of $F_p$	0.2 flicker

Once trained, the ATSC-DRLNI model was applied to the two environments shown in Figure 4, where the same 4 phases, traffic conditions and scenarios used for the training still apply. All data were obtained from the same source. The results shown below are obtained from simulations of these two environments and correspond to scenario 1, where phases 1 and 2 are optimized to give preference to straight traffic, as better results were obtained with this scenario.

4.3. Selected Methods for Comparison

Seven other traffic control methods were selected from the open literature to use for comparison purposes with the proposal of this work, which are listed below. In all of them, the same scenarios and reward function, where applicable, were used.

• Fixed-Time (FT): This is a classical non-adaptive method used for traffic light control that serves as a baseline for comparisons.
• Multi-Agent Deep Reinforcement Learning (MADRL): Multi-agent algorithm based on DRL. It uses gradient policies and trains agents in a centralized manner, where collective decision-making is based on real-time information feedback to agents. There are also descentralized versions, where agents are autonomous with decision-making not depending on other agents.
• Multi-Agent Recurrent Deep Deterministic Policy Gradient (MARDDPG): decentralized algorithm for agent training where each agent shares policies and information with other agents. Individual agents are capable to execute decision independently.
• Single-Agent Based Algorithm (SABA): DQN based descentralized algorithm where agents do not relate between them, although they try to predict what other agents may decide.
• Longest Queue First (LQF): This method favors lane selection where the green light is already active and gives priority to vehicles close to the intersection, a situation that may produce inconsistent phase cycles.
• Q-Learning (DQN): RL-based method for action learning that uses a predefined number of Q-values that are related to the number of actions. Actions are stored in a table that contains elements, such as vehicles queue, waiting time of vehicles in lanes close to the intersection and the current phase.
• Self-Organizing Traffic Lights (SOTL): This method uses a self organizing algorithm to determine the phases of the traffic lights.

4.4. Simulation Results

Simulations use the parameters shown in Table 5, that are the values that yielded the best results in terms of the final value of the metrics, convergence, and stability are included in Table 6.

The results of applying the model are represented using different metrics: Average Waiting Time (AWT), Average Queue Length (AQL), Average Reward (AR), and traffic light changes (flickers), this one divided by $F_{norm}$. Figure 5, Figure 6 and Figure 7 show and example of one simulation of 5,000 episodes, where the AR, AQL, and flickers are plotted against the number of episodes for all the eigth models considered. In all cases, they are plotted using a moving average with a window of 10 episodes and a shadow area representing the standard deviation.

In Figure 5 the final value of the AR obtained with ATSC-DRLNI is higher than that of any other model; it is 38% better than the closest model, SABA. The AR value begins at -3.18 and ends at -1.80, showing a large variation in the first 300 episodes. SABA and LQF, non RL models, achieve a much faster convergence, however, their final AR values are -3.02 and -3.08, respectively. In the case of DQN, it has acceptable initial AR values, however, it shows a poor learning capacity, finishing with a value of -4.3, similar to those obtained with MARDDPG and MADRL.

The queue length for the eight models is presented in Figure 6 expressed in seconds/hour. For ATSC-DRLNI, the value converges to 2,426 vehicles. This value is significantly smaller (40%) than those achieved by SABA, SOTL, LQF and FT, which attain intermediate queue length results. The other models, MARDDPG, MADRL and DQN have the worst queue length results.

With respect to the adimensional number of flickers in Figure 7 the best results were obtained by SOTL, followed by ATSC-DRLNI, which compares favorable with the other six models, where LQF and SABA have a performance only slightly worse than ATSC-DRLNI.

A summary of the results for the metrics considered is included in Table 7, which includes mean values, final values, and standard deviation during the last episode.

Three different metrics to show learning stability, traffic balance, and system stability were also obtained based on the following definitions:

$$\begin{array}{cc}\hfill Learning\_stability& =1-min\left(1,{\displaystyle \frac{{\sigma }_{R_{late}}}{|{\mu }_{R_{late}}|+ϵ}}\right)\hfill \end{array}$$

(9)

with

$$\begin{array}{cc}\hfill R_{late}& \{r_i:i\ge 3,000\}\mathrm{reward}\mathrm{in}\mathrm{final}\mathrm{episodes}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mu }_{R_{late}}& ={\displaystyle \frac{1}{|R_{late}|}}\sum _{r\in R_{late}}r\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\sigma }_{R_{late}}& =\sqrt{{\displaystyle \frac{1}{|R_{late}|-1}}{\sum }_{r\in R_{late}}{(r-{\mu }_{R_{late}})}^2}\hfill \end{array}$$

(12)

and $ϵ=0.1$ a regularization factor.

$$\begin{array}{cc}\hfill Traffic\_balance& =1-{\displaystyle \frac{|{\mu }_{Q_{NS}}-{\mu }_{Q_{EW}}|}{max(0.1,{\mu }_{Q_{NS}}+{\mu }_{Q_{EW}})}}\hfill \end{array}$$

(13)

with ${\mu }_{Q_{NS}}$ medium size of vehicle queue in the North-South direction and ${\mu }_{Q_{EW}}$ medium size of vehicle queue in the East-West direction.

$$\begin{array}{cc}\hfill System\_stability& =1-{\displaystyle \frac{{\sigma }_{flicker}}{max(0.1,{\mu }_{flicker})}}\hfill \end{array}$$

(14)

where ${\mu }_{flicker}$ is the mean value of the flickers, and ${\sigma }_{flicker}$ its standard deviation.

The values for these three metrics are shown in Figure 8 where, in all cases, the closer the value to 1, the better the metric. It can be noted that the ATSC-DRLNI performs best in terms of traffic balance and worse in terms of learning stability, an issue that will be commented below. Its performance in terms of System Stability is similar to the other methods.

5. Discussion

Findings show a consistent positive correlation between the waiting time and the size of the queue length, similar to that shown in other studies [53,55], indicating that the inclusion of only one of two variables in the reward function is correct. The use of the number of flickers in the reward function showed a positive overall impact in the obtained results, in contrast to other methods for single intersections [47,48,49,53], where the reward function focuses only on waiting time and/or queue length, without considering this variable.

The results presented suggest that optimization using the ATSC-DRLNI model can be a viable option to handle traffic lights and alleviate traffic congestion in real scenarios. From a theoretical point of view, results also align with those in [53,54,55,56,57], as our simulation results indicate the convenience of using a combined reward function in single intersections.

There is research that has also used flickers [53,55], in a DRL context with multiobjective models, showing reductions up to 18% in waiting time. However, these studies do not combine queue length and flickers in the reward function.

The good performance of the ATSC-DRLNI model in balancing traffic (Figure 8-b), and in queue reduction (Figure 6) are a direct consequence of the NI layer that, by introducing controlled disturbances during training, forces agents to discriminate between spurios fluctations in demand and real tendencies in congestion. This, in turn, prevents "over-reactions" in the control policy to small queue increments, a behavior that in other models (such as standard DQN) produces unstable phase changes. System stability, as shown in Fig. Figure 8-c, corroborates that agents have learned a more robust control policy, that avoids overfitting with the data training set.

It is worth mentioning that the ATSC-DRLNI model has some limitations that must be addressed. First, simulations take longer than the other models considered in the comparison. Second, there are some environmental factors, such as pedestrians, special vehicles (ambulances, police cars, or autonomous vehicles), that are not included. Third, evaluation of multiple intersections and more complex scenarios is still missing.

Future research must focus on scenarios with multiple intersections, trying to model the ATSC at each intersection as a descentralized agent. The use of MADRL may be useful in this situation [74,75]. Finally, field testing is necessary to obtain a real assessment of ATSC strategies [31,47,74].

6. Conclusions

This research investigates the application of DRL to optimize ATSC and emphasizes the trade-off between minimizing queue lengths and suppressing signal instability at complex intersections. We developed and simulated a training environment on the SUMO platform, allowing the proposed ATSC-DRLNI model to learn signal timing policies through an A2C framework. Experimental results suggest that the agent skillfully balances queue size and phase flickers, outperforming seven benchmarks—including multiagent approaches such as MADRL and MARDDPG—with a 40% reduction in queue length and sustained stability. However, although these results highlight the considerable potential of decentralized traffic management, they are limited to the simulated contexts described here.

A key contribution of this study is the analysis of how the A2C algorithm improves communication between agents and learning stability, enabling simultaneous optimization of queue lengths and signal flashes under congested traffic conditions. However, the disparity between the optimizations derived from simulation and practical implementation remains a major obstacle, especially with regard to physical constraints and unmodeled stochastic elements. To address this disparity, future research focused on validating these control strategies in increasingly realistic simulations or hardware-in-the-loop (HIL) is needed.

In addition, the methodology proposed in this paper could bridge the gap between theoretical autonomous traffic systems and their practical implementation, although it requires rigorous safety evaluations before real-world deployment. Future work will focus on creating multi-objective reward functions that explicitly penalize signal flickering without sacrificing wait time minimization, as well as integrating vulnerable road users, such as pedestrians, into state representations. Overall, this study posits deep reinforcement learning as a powerful tool for designing scalable traffic management paradigms, while highlighting the need for further empirical corroboration for the transition from simulations to reality.