Optimizing Traffic Signal Control Using Reinforcement Learning Methods: Hybrid Approach

1. Introduction

Urban traffic congestion has become a critical challenge in modern cities due to the rapid increase in vehicle ownership and the limitations of existing infrastructure. Inefficient traffic signal control (TSC) systems contribute significantly to increased travel time, fuel consumption, and environmental pollution. Traditional traffic signal control strategies, such as fixed-time or actuated control, are often unable to adapt effectively to dynamic and stochastic traffic conditions.

In recent years, reinforcement learning (RL) has emerged as a promising approach for intelligent traffic signal control [3,4]. By modeling the traffic environment as a sequential decision-making problem, RL focuses on sequential decision-making problems, where an agent learns optimal behaviors through trial-and-error interactions with its environment. This paradigm allows for adaptive and data-driven optimization of signal timings, improving overall traffic flow and reducing congestion.

Several classical RL algorithms have been applied to TSC, including Q-learning, SARSA, and Monte Carlo methods. Q-learning, an off-policy algorithm, is known for its fast convergence and ability to approximate optimal policies. However, it may suffer from instability due to overestimation of action values. On the other hand, SARSA, an on-policy method, provides more stable learning but often converges more slowly. Monte Carlo methods, which rely on episodic returns, offer unbiased value estimation but typically exhibit high variance and require longer training periods.

Despite their individual strengths, these algorithms also present inherent limitations when applied independently to complex and dynamic traffic environments. This motivates the need for hybrid approaches that can leverage the advantages of multiple learning strategies while mitigating their weaknesses.

In this paper, we perform a simulation of an intelligent traffic signal control system based on a hybrid reinforcement learning framework. The proposed approach combines Q- learning, SARSA, and Monte Carlo methods into a unified update mechanism, aiming to achieve a balance between convergence speed, stability, and long-term performance. The traffic environment is modeled as a stochastic intersection with dynamic vehicle arrivals, where the RL agent controls the traffic light phases. The performance of the proposed hybrid method is evaluated against classical RL algorithms using key traffic metrics such as average delay, waiting time, throughput, and queue length[12].

The main contributions of this work can be summarized as follows:

• A hybrid RL approach that integrates Q-learning, SARSA, and Monte Carlo updates into a single framework.
• A simulation-based traffic signal control model using a dynamic and stochastic environment.
• A comprehensive performance evaluation using multiple traffic efficiency metrics.
• A comparative analysis demonstrating the effectiveness of the proposed method over classical RL approaches.

2. Background and Preliminaries

In this section, we introduce the essential notions and theoretical background of RL used in this paper [1]. To provide a rigorous foundation for the proposed framework, we present the key mathematical definitions, including the formal representation of the environment, state space, action space, and reward function.

2.1. Reinforcement Learning

RL is a machine learning technique that enables optimal solutions to sequential decision problems through trial and error. It’s a process where an agent (controller) optimizes its behavior by interacting with its uncertain environment. The agent makes decisions and takes actions, and the environment provides feedback, facilitating state changes. The goal is to learn the optimal behavior in an environment to maximize the reward.

The RL process begins with the agent observing the current state of the environment, selecting an action, receiving a reward from the environment, adjusting its policy, and repeating the cycle. The agent and the environment interact at discrete time steps $t=0,1,2,\dots ,T$, where the agent observes the state at time step t, denoted by $s_t\in S$, produces an action $a_t\in A$, receives a reward $r_{t+1}\in \mathbb{R}$, and transitions to a new state $s_{t+1}$. This interaction is further illustrated in Figure 1.

The agent aims to learn a policy $\pi$ to maximize total cumulative reward over time. Unlike unsupervised learning, which seeks hidden structures in unlabeled data without focusing on rewards. Thus, RL is the third ML paradigm. Along with supervised and unsupervised learning, dedicated to maximizing an agent’s total rewards from its environment.Formally, RL problems can be formulated in the framework of Markovian Decision Process (MDPs).

2.2. MDP: Markov Decision Process

In this section, we present the fundamental principles of Markov Decision Processes (MDPs), a mathematical framework for modeling sequential decision-making problems commonly encountered in RL.

A MDP consists of five elements as follows $(\mathit{S},\mathit{A},\mathit{P},{\mathit{r}}_{\mathit{t}},\gamma )$ where :

• $\mathit{S}$ is the space of environmental states.
• $\mathit{A}$ is the space of actions the agent can take.
• P is a transition function $P:\mathit{S}\times \mathit{A}\times \mathit{S}\to [0,1]$ (the probability of ending in state $s^{\prime }$, given that action a is taken in state s.can be represented by a conditional probability mass function denoted by $\mathbb{P}(s^{\prime }\mid s,a)$).
• $r_t$: $\mathit{S}\times \mathit{A}\to \mathbb{R}$ The reward function: $r(s,a)$ provides the expected reward conditioned on being in a specific state s and taking a particular action a.

  $$r_t=\mathbb{E}[r_{t+1}\mid S_t=s,A_t=a,S_{t+1}=s^{\prime }],$$

  (1)
• $\gamma$ is a discount factor that specifies how much immediate rewards are favored over future rewards.$\gamma \in [0,1]$, When $\gamma$ is equal to 1, it indicates that future rewards are considered as important as present rewards. Conversely, when $\gamma$ is equal to 0, it means that only present rewards are valued.

Through interactions with the environment, the agent generates a trajectory $\tau$, consisting of a sequence $s_0,a_0,s_1,a_1\dots s_T,a_T$. At the end of a trajectory, our agent will have accumulated a sum of rewards what we refer to as the return. The overall goal of the agent is to maximize the cumulative reward ${\displaystyle R_t\left(\tau \right)={\displaystyle \sum _{k=0}^T}{\gamma }^k{r}_{t+k}}$ along a trajectory up to time t by choosing the actions obtained along this trajectory $\tau$. Note that when ($T<\infty$) is finite, it is referred to as a finite episode, whereas if T is infinite, it is referred to as a continuing episode ( $\gamma <1$).

2.3. Policies and value function

We define a policy as a function $\pi :S\to A$ that assigns a probability distribution over the action space for each state. If there is only one action associated with each state, it is called a deterministic policy. In the general case, $\pi (a_t\mid s_t)$ denotes the probability of choosing action $a_t$ in state $s_t$.

The ultimate goal of RL is to find the optimal policy ${\pi }^{★}$ that maximizes the expected cumulative discounted reward, formally defined as:

$${\pi }^{★}=arg\underset{\pi }{max}{\mathbb{E}}_{\pi }[{\displaystyle \sum _{t=0}^{\infty }}{\gamma }^t{r}_t\mid S_t=s],$$

(2)

the agent seeks to learn a behavior that not only reacts greedily to immediate feedback but also anticipates long-term consequences term.

To evaluate an agent’s performance from a specific initial state under a particular policy, we use a metric called the value function. This function measures the quality of a state by estimating the total rewards expected from the trajectory. It indicates how beneficial it is to be in a certain state. Most RL algorithms estimate a value function $V_{\pi }$, which is always associated with a policy $\pi$, as the expected rewards depend on the actions taken. The state-value function is denoted as follows:

$$V_{\pi }\left(s\right)={\mathbb{E}}_{\pi }[R_t\mid S_t=s]={\mathbb{E}}_{\pi }[{\displaystyle \sum _{k=0}^T}{\gamma }^k{r}_{t+k}\mid S_t=s],(\forall s\in S).$$

(3)

In RL, the value function applies to all policies, and when the policy is optimal, it is referred to as theoptimal value function. Additionally, we define the Q-value function, which represents the value of starting from a state s, taking an action a, then following policy $\pi$ until reaching a terminal state. The action-value function is denoted:

$$Q_{\pi }(s,a)={\mathbb{E}}_{\pi }[R_t\mid S_t=s,A_t=a]={\mathbb{E}}_{\pi }[{\displaystyle \sum _{k=0}^T}{\gamma }^k{r}_{t+k}\mid S_t=s,A_t=a],(\forall (s,a)\in (S,A)).$$

(4)

A fundamental property of the value function is its satisfaction of the following recursive relationship:

$$\begin{array}{cc}\hfill V_{\pi }\left(s\right)& ={\mathbb{E}}_{\pi }[r_{t+1}+\gamma R_{t+1}\mid S_t=s]\hfill \\ \hfill V_{\pi }\left(s\right)& =\sum _a\pi (a\mid s)\sum _{s^{\prime }}p(s^{\prime }\mid s,a)[r(s,a,s^{\prime })+\gamma V_{\pi }\left(s^{\prime }\right)]\hfill \end{array}$$

This relationship is called the Bellman equation for $V_{\pi }$ (a similar form can be derived for the Q-function).

$$Q_{\pi }(s,a)={\mathbb{E}}_{\pi }[r_{t+1}+\gamma R_{t+1}\mid S_t=s,A_t=a]$$

$$=\sum _{s^{\prime }}p(s^{\prime }\mid s,a)r(s,a,s^{\prime })+\gamma \sum _{s^{\prime }}p(s^{\prime }\mid s,a)\sum _{a^{\prime }}\pi (a^{\prime }\mid s^{\prime })Q_{\pi }(s^{\prime },a^{\prime })$$

$$Q_{\pi }(s,a)=\sum _{s^{\prime }}p(s^{\prime }\mid s,a)[r(s,a,s^{\prime })+\gamma \sum _{a^{\prime }}\pi (a^{\prime }\mid s^{\prime })Q_{\pi }(s^{\prime },a^{\prime })]$$

These 3 concepts: policy, state-value function and action-value function are the foundation of all RL algorithms we will discuss. Solving a RL problem involves finding a policy that achieves high long-term rewards.

The value function provides a partial ordering over policies, allowing us to determine whether one policy $\pi$ is better than or equal to another policy ${\pi }^{\prime }$.

$$\pi \ge {\pi }^{\prime }\mathrm{if}{V}_{\pi }\left(s\right)\ge V_{{\pi }^{\prime }}\left(s\right)\forall s\in S.$$

If a policy is better than or equal to all other policies, it is referred to as the optimal policy. The optimal policy is denoted as ${\pi }^{★}$.

Given that the optimal action-value function $Q^{*}(s,a)$ is the one which gives the maximum expected return, the optimal policy ${\pi }^{*}$ can be found by choosing the action $a^{*}$ that gives the maximum Q-value for the given state s:

$$a^{*}=arg\underset{a\in \mathcal{A}}{max}{Q}^{*}(s,a)$$

(5)

The optimal Q-function follows the Bellman equation and it can be learned through an iterative update process:

$$Q_{t+1}(s,a)=Q_t(s,a)+\alpha {\delta }_{t+1}$$

(6)

where $0<\alpha <1$ is the learning rate, and ${\delta }_{t+1}$ is the temporal difference (TD) error:

$${\delta }_{t+1}(s,a)=r_t+\gamma \underset{a_{t+1}\in \mathcal{A}}{max}{Q}_t(s_{t+1},a_{t+1})-Q_t(s,a)$$

(7)

We denote $y_t=r_t+\gamma {max}_{a_{t+1}\in \mathcal{A}}{Q}_t(s_{t+1},a_{t+1})$ as the TD-target.

3. Traffic Signal Control Problem

Traditional Traffic Signal Control (TSC) approaches are rooted in classical traffic flow theory and have long served as the backbone of urban traffic management [8]. However, these methods do not respond well to real-time traffic changes. The max-pressure method tries to reduce congestion by balancing traffic between lanes, but it assumes that the roads after the intersection have unlimited space, which is not realistic. Overall, these methods cannot adapt well to changing traffic conditions and limit their effectiveness in real-world deployment scenarios.

3.1. TSC Types

TSC frameworks play an essential role in managing urban mobility, safety improvement, and optimizing intersection efficiency.These approaches can generally be divided into three main categories: fixed-time (pre timed) control, actuated control and adaptive control, each of which has several variants. This Figure 2 presents the main categories of traffic signal control methods and their common subtypes. trees, positioning

3.2. Reinforcement Learning for TSC

In RL-based traffic signal control, the decision-making process relies on several key components that interact dynamically to optimize traffic flow. The overall functioning of RLin a traffic signal control system can be explained by the following main steps.

• Environment: The traffic network constitutes the environment in which the RL agent operates. It may include intersections, vehicle and pedestrian flows, and external disturbances like weather or sudden congestion. The RL agent does not directly control this environment but learns to react to its dynamics by optimizing signal timings (Figure 3).
• RL Agent: The RL agent is the brain of the system, it receives the incoming traffic data and decides the best signal phases. It learns to minimize congestion, delays and emissions through continuous interaction with the environment depending on the method used.
• Action: Traffic light control systems offer options such as timing adjustment and synchronized green light lane design choices. Once an action is selected, the system performs signal adjustments (e.g. green time extension, phase skipping, cycle length adjustment, coordination of multiple intersections) as per the agent output.
• Environment Update: After applying the control decisions, the environment evolves. Vehicles move, congestion shifts, and new traffic enters, resulting in a new state that becomes the input for the agent’s next decision.
• Reward Calculation: The system evaluates the agent’s decision using predefined metrics. Common reward functions are reduction in waiting time, queue lengths, fuel consumption, emissions, and enhancement of pedestrian safety and traffic balance.
• Reward Signal Feedback: The computed reward is fed back into the RL algorithm, refining its policy. Through this iterative process, the agent improves its control strategy, enabling real-time, adaptive signal management.

3.3. RL Algorithms for Traffic Signal Control

There are three main types of RL algorithms for traffic control: value-based, policy-based and actor-critic methods. Each of them has a different mechanism to optimize the traffic performance.

3.3.1. Value-Based Algorithms

This RL type uses a value function to estimate expected returns of actions in specific traffic states, selecting actions to maximize expected rewards. It has proven effective for discrete tasks like phase selection and green-time allocation, although scalability can be problematic in larger or continuous action spaces. Common examples include Q-learning (QL) and deep Q-networks (DQNs).

• Q-Learning: An off-policy algorithm that iteratively updates an action-value function independently of the current policy. Suitable for single-intersection control, it learns state-action values $Q(s,a)$ via:

  $$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[R(s_t,a_t)+\gamma \underset{a}{max}Q(s_{t+1},a)-Q(s_t,a_t)\right]$$

  (8)
  Q is efficient in discrete environments but struggles with high-dimensional networks.
• Deep Q-Networks: DQN enhances Q-learning by approximating the Q-function with deep neural networks (NN). Techniques such as experience replay and target networks improve training stability, enabling coordination across multiple intersections. The update rule is given by:

  $$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[R(s_t,a_t)+\gamma \underset{a}{max}Q(s_{t+1},a;\theta )-Q(s_t,a_t;\theta )\right]$$

  (9)

3.3.2. Policy-Based Algorithms

This algorithm optimizes policy directly, bypassing value estimation, making it suitable for continuous action spaces and enabling fine-grained control such as dynamic phase timing. It offers flexibility in complex environments but demands precise hyperparameter tuning, with PPO noted for its stability and efficiency.

• Proximal Policy Optimization (PPO) is a RL algorithm that constrains policy updates to maintain training stability. It is commonly employed in multi-intersection control systems and effectively balances exploration and exploitation within large-scale networks. The clipped objective function may be expressed as:

  $$\begin{array}{cc}\hfill L^{\mathrm{CLIP}}\left(\theta \right)& =\mathbb{E}\left[min\left(r_t\left(\theta \right)A_t,\mathrm{clip}\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right)A_t\right)\right]\hfill \end{array}$$

  (10)

  where : $r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{\mathrm{old}}(a_t\mid s_t)}}$ and $A_t$ denotes the advantage function [9].

3.3.3. Actor-Critic Algorithms

Actor-critic approaches represent a particularly compelling hybrid family within the reinforcement learning landscape, as they seek to combine the strengths of bothvalue-based and policy-based methods. The central idea rests on the cooperation of two distinct components: an actor, responsible for proposingactions given the current state, and a critic, whose role is to assess the quality of those decisions. This duality endows the architecture with enhanced learning stability while preserving sufficient computational efficiency to tackle large-scale traffic control problems. Three representative variants are examined below:

• Deep Deterministic Policy Gradient: The DDPG was specifically designed for continuous action spaces, making it well-suited to tasks such as the dynamic adjustment of signal phase durations at intersections. Its operation relies on two complementary, iterative updates. Firstly, the critic refines its estimate of the state-action value:

  $$\mathit{CriticUpdate}:Q(s_t,a_t)\leftarrow R(s_t,a_t)+\gamma Q\left(s_{t+1},\mu \left(s_{t+1}\right)\right)$$

  (11)
  Secondly, the actor updates its policy by maximising the gradient of this value:

  $$\mathit{ActorUpdate}:{\nabla }_{\theta }J\left(\theta \right)=\mathbb{E}\left[{\nabla }_{a}Q\left(s_t,\mu (s_t;\theta )\right){\nabla }_{\theta }\mu (s_t;\theta )\right]$$

  (12)
  While powerful, DDPG is sensitive to tuning.
• Soft Actor-Critic (SAC): To address this sensitivity, SAC enriches the objective function by incorporating an entropy term. This modification naturally encourages broader exploration of control strategies, thereby improving robustness in potentially non-stationary multi-agent environments. The resulting optimisation objective takes the form:

  $$J\left(\pi \right)=\sum _t\mathbb{E}\left[R(s_t,a_t)+\alpha H\left(\pi (·\mid s_t)\right)\right]$$

  (13)

  where the coefficient $\alpha$ governs the trade-off between exploration and exploitation.
• Advantage Actor-Critic (A2C): ntroduces yet another perspective through the notion of advantage, which quantifies the extent to which a given action outperforms the average expected return in a given state:

  $$A(s_t,a_t)=R(s_t,a_t)-V\left(s_t\right)$$

  (14)
  By substituting this centred signal for the raw reward in the policy update, A2C reduces gradient estimate variance, yielding more stable learning dynamics:

  $${\nabla }_{\theta }J\left({\pi }_{\theta }\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }log{\pi }_{\theta }(a_t\mid s_t)A(s_t,a_t)\right]$$

  (15)
  The A2C achieves strong results in real-time adaptive control, particularly in collaborative multi-agent architectures where variance reduction plays a decisive role in ensuring convergence.

4. Results and Discussion

4.1. Experimental Setup

In our study, we propose an simulation of an intelligent traffic signal control system using a RL framework. The environment models a single intersection with stochastic vehicle arrivals in both horizontal and vertical directions. Four learning strategies are evaluated: Q-learning, SARSA, Monte Carlo (MC), and the proposed hybrid Method.

Each algorithm is trained over multiple episodes, and its performance is evaluated using key traffic metrics. Including average delay, waiting time, throughput, queue length, and signal phase durations (green and red times). To enhance readability and reduce noise, the results are smoothed using a moving average, and ariability is represented using standard deviation bands.

4.2. Proposed Method

The proposed hybrid approach draws on the complementary strengths of three well-established RL algorithms: Q-Learning, SARSA, and Monte Carlo, with the goal of achieving what no single method can easily deliver on its own: fast convergence, stable learning behavior, and reliable long-term reward estimation.

Each algorithm contributes a distinct capability. Q-Learning brings strong off-policy learning, enabling the agent to rapidly approximate optimal action values. SARSA, through its on-policy update mechanism, promotes greater stability by avoiding overly aggressive updates in stochastic environments like traffic control. Monte Carlo methods complement both by relying on complete episode returns, which leads to more accurate long-term value estimates.

To integrate these contributions into a coherent learning rule, we define a hybrid Q-value update based on a weighted combination of the three estimators:

$$Q_{\mathrm{new}}(s,a)=(1-\alpha )Q(s,a)+\alpha \left({\beta }_1{Q}_{QL}(s,a)+{\beta }_2{Q}_{SARSA}(s,a)+{\beta }_3{Q}_{MC}(s,a)\right)$$

(16)

where $\alpha \in [0,1]$ is the learning rate, governing how much weight is given to new information relative to the current estimate. The coefficients ${\beta }_1$, ${\beta }_2$, and ${\beta }_3$ control the relative influence of each method and are subject to the normalization constraint:

$${\beta }_1+{\beta }_2+{\beta }_3=1$$

(17)

Rather than fixing these weights throughout training, we introduce an adaptive scheduling strategy that adjusts their values dynamically as learning progresses. Let $t\in [0,1]$ denote the normalized training progress:

$$t={\displaystyle \frac{\mathrm{current}\mathrm{episode}}{\mathrm{total}\mathrm{episodes}}}$$

(18)

The weights then evolve as time-dependent functions ${\beta }_1=f_{QL}\left(t\right)$, ${\beta }_2=f_{SARSA}\left(t\right)$, and ${\beta }_3=f_{MC}\left(t\right)$, each reflecting the natural role of its associated method at different stages of training. In the early episodes, ${\beta }_2$ dominates to favor the conservative and stable updates of SARSA, which is especially valuable in highly dynamic environments. As training advances, ${\beta }_1$ gradually increases, allowing Q-Learning to drive faster convergence toward an optimal policy. In the final stages, ${\beta }_3$ becomes predominant, leveraging the improved reliability of complete episode returns for fine-grained value refinement.

At each update step, the weights are renormalized to ensure ${\beta }_1+{\beta }_2+{\beta }_3=1$ is always satisfied. This phased, adaptive design allows the agent to benefit from each learning paradigm precisely when it is most effective, resulting in a more robust and well-rounded policy for complex control tasks such as adaptive traffic signal management [2].

4.3. Traffic Performance Evaluation

To evaluate the effectiveness of the proposed traffic control system, several key performance metrics are considered.

1.

The Average Delay measures the mean time vehicles spend waiting before passing through the intersection, reflecting overall congestion levels;

2.

The Throughput represents the number of vehicles successfully processed per unit time, indicating the efficiency of traffic flow;

3.

The Queue Length corresponds to the number of vehicles waiting in each lane or approach, providing insight into spatial congestion dynamics;

4.

The Waiting Time captures the total time each vehicle spends in the system before being served, offering a microscopic view of individual vehicle experience.

Finally, the Reward is defined within the reinforcement learning framework as a scalar feedback signal that encourages desirable behaviors such as minimizing delay and queue buildup while maximizing throughput. Together, these metrics provide a comprehensive evaluation of both system-level efficiency and agent-level learning performance.

4.4. Results

4.4.1. Learning Curves

The 4 RL algorithms were trained over $N=3000$ episodes on the single-intersection TSC environment and evaluated across 6 metrics:cumulative reward, average queue length, vehicle throughput,delay per vehicle, and convergence speed. Results are summarised in Table 1.

As shown in Figure 4a, Q-Learning and SARSA plateau early at final rewards, reflecting the bias inherent in single-step bootstrapping under stochastic traffic arrivals. Monte Carlo achieves the highest reward ($-\mathrm{10,753}$) through unbiased full-return estimates, but at the cost of elevated training variance and reduced throughput ($392.9$ veh/ep), as high-variance policy updates compromise real-time phase-switching decisions (Figure 4c).

The proposed hybrid method attains a final reward of$-\mathrm{11,386}$, a 7.2% improvement over both TD baselines, while achieving the highest throughput among all evaluated algorithms ($396.5$ veh/ep, $+3.6$ veh/ep over Monte Carlo).

In terms of delay, the Hybrid reduces waiting time by $7.3\%$ relative to Q-Learning and SARSA, with only a $5.8\%$ gap compared to Monte Carlo (Figure 4d). This trade-off of marginal increase in delay for significant gain in throughput is particularly advantageous in high volume intersection management where the primary operational aim is to maximise the flow of vehicles.

The convergence profiles (Figure 4f) confirm that while the Hybrid converges at episode 2938, it reaches a higher normalised reward ceiling than Q-Learning and SARSA, consistent with the established bias–variance trade-off in RL [2,10].

These results support the adaptive 3-component weighting strategy of the hybrid algorithm. The hybrid starts with Q-Learning (early exploration) as the dominant update signal and smoothly transitions to Sarsa (stable mid-training updates) and eventually Monte Carlo (accurate long-horizon returns). This allows the hybrid to leverage the complementary strengths of the baselines and to overcome their individual shortcomings.

4.4.2. Convergence Analysis

All RL algorithms (Figure 5) improve progressive during training, though with notable differences in speed and stability. Q-learning converges fast but with early fluctuations, SARSA learns more smoothly but slower, and Monte Carlo suffers from high variance due to its episodic nature. The proposed hybrid approach outperforms all three by combining fast convergence with greater stability, confirming the benefit of integrating multiple learning paradigms.

4.5. Comparative Analysis of RL Algorithms

Table 2 summarizes the performance of four RL algorithms (Q-Learning, SARSA, Monte Carlo and a hybrid approach) evaluated for intersection signal control across five metrics. All four methods converged around episode 10 000, indicating similar learning horizons.

In terms of stability, measured by standard deviation of waiting time (Figure 6c), Hybrid exhibited the most stable behavior ($\sigma =29.37$ s), while Monte Carlo and the Q-Learningmethod exhibited more variability ($\sigma =33.77$ s and $\sigma =33.51$ s respectively). SARSA was in between $\sigma =32.61$

All algorithms perform relatively similar in throughput which shows a slight trade-off between delay minimization and overall flow efficiency. Maximum queue lengths were comparable among the methods, with all algorithms except Hybrid reaching 6 vehicles, while Hybrid reached only 5 vehicles.Figure 6d and Figure 6a further illustrate these differences over the course of training. The throughput curves (Figure 6d) confirm that all methods stabilize at similar levels, while the waiting time and queue length plots (Figure 6c and Figure 6b) highlight the lower variance achieved by Q-Learning. The delay comparison (Figure 6a) corroborates the tabulated values, showing Hybrid consistently producing the shortest delays after convergence.

5. Discussion

The results show that the use of multiple RL techniques results in better performance under dynamic traffic conditions. The hybrid method takes advantage of complementary strengths of:

• Q-learning: fast convergence,
• SARSA: stability,
• Monte Carlo: long-term return estimation.

This combination enhances robustness and adaptability, making the hybrid approach suitable for intelligent TSC.

Overall, the hybrid RL approach outperforms classical methods in each of the metrics assessed. It offers a promising solution for adaptive TSC in stochastic and dynamic environments.

• Average Delay: The hybrid method achieves the lowest average delay among all evaluated algorithms. This indicates its superior capability in minimizing congestion and optimizing traffic signal timing. Q-learning performs competitively but may occasionally lead to suboptimal decisions due to value overestimation.
• Throughput: The throughput results indicate that the hybrid approach allows a higher number of vehicles to pass through the intersection. This reflects a more efficient allocation of green signal phases. SARSA also provides consistent throughput but remains slightly less efficient than the hybrid method.
• Queue Length: Queue length is a key indicator of congestion levels. The hybrid method maintains shorter and more stable queue lengths compared to other approaches. Monte Carlo exhibits higher variability, resulting in fluctuating queue sizes.
• Waiting Time: The total waiting time is significantly reduced using the hybrid approach. This improvement highlights the ability of the proposed method to dynamically adapt signal control decisions based on real-time traffic conditions.

However, this study is limited to one intersection and a discrete state representation. Future work can extend this framework to multi-intersection networks and integrate deep reinforcement learning techniques to handle more complex and high-dimensional environments.

6. Conclusion

This paper presented a comparative study of 4 RL algorithms: Q-Learning, SARSA, Monte Carlo, and a Hybrid approach, applied to adaptive traffic signal control at an isolated intersection. Through extensive simulation over 10,000 training episodes, all methods were evaluated across 5 key performance indicators: average waiting time, waiting time standard deviation, peak queue length, vehicle throughput, and convergence speed. The results confirmed that all four algorithms successfully learned effective signal-switching policies without any prior knowledge of traffic dynamics, validating reinforcement learning as a suitable framework for intelligent intersection management.

From among all tested techniques, the introduced Hybrid algorithm showed uniform and well-balanced results when tested against all measures. Through the use of a phase-based schedule, which incorporates the exploration ability of Monte Carlo, quick convergence capability of Q-Learning, and on-policy stability of SARSA, the introduced hybrid technique can adapt to the dynamic conditions of training as opposed to sticking to a single update strategy. The adaptability of this approach is most relevant in non-stationary environments, such as dynamic real-world traffic situations.

There are many avenues for further investigation including extension of the assessment into multi-intersection networks, incorporation of architectures based on deep RL (e.g., DQN or PPO) for more sophisticated state representation, and use of the framework in simulation environments such as SUMO and VISSIM to test its practical application. The contributions made in this study are significant in providing an alternative and flexible approach to traditional model-free solutions and are thus likely to be considered a valuable starting point for future investigation.