Traffic Signal Control with Communicative Deep Reinforcement Learning Agents: a Case Study

1. Introduction

Traffic congestion is becoming an increasingly critical issue in medium to large cities, affecting the quality of life for citizens [1].

A primary factor contributing to traffic congestion is the surge in population growth. As the population increases, so does the number of vehicles on the road. According to the United Nations, 54% of the world’s population resides in urban areas, and this percentage is expected to rise to 66% by 2050 [2]. This growth directly correlates with increased traffic congestion [2]. As cities continue to expand, the layout of intersections can become increasingly complex. Most blocks can only accommodate a limited amount of vehicles at a stoplight before traffic gets backed up. In densely populated conditions, traffic congestion is exacerbated by narrow road widths, limited vehicle parking, traffic signals and maintenance operations [3]. Global warming, which directly influences wind patterns, rainfall, and other extreme atmospheric events, plays a significant role in accelerating the deterioration of pavement conditions. This environmental phenomenon is not only a contributing factor to the degradation of our city infrastructures but also a consequence of it. The interplay between global warming and deteriorating pavement conditions forms a complex cycle that exacerbates both traffic congestion and environmental issues [4].

In this scenario, the management of ground vehicle traffic is an issue of great importance and the target of intense research activity. One essential element of the problem is traffic signal control (TSC), i.e., the management of the traffic lights at intersections to optimize vehicle traffic in urban road networks. To model this problem, each signalized intersection can be conceptualized as an agent. The available actions for these agents are the various traffic light switches. These agents can be trained to optimize the timing of these actions, thereby smoothing the traffic flows and improving overall traffic management.

One current approach to this problem relies on the application of reinforcement learning algorithms [5].

In a nutshell, reinforcement learning (RL) is a machine learning method based on an agent which learns to perform a task by interacting with an external environment and by receiving a reward signal [6]. In the present case, the external environment is the traffic network and the reward is usually some statistical quantity, such as the average of the vehicle queue length at traffic light intersections [7]. Such a paradigm can be summarized as a multi-agent reinforcement learning (MARL) [8]. In adaptive traffic signal control (ATSC), agents learn how to control the switch of multiple traffic lights to minimize vehicle queuing at signalized intersections [9].

From the viewpoint of each learning agent, the environment is non-stationary. This non-stationarity arises from two main factors: firstly, during the learning process, each agent updates its policy, thereby altering its behavior. This results in a change in the perceived environmental dynamics for each individual agent. Secondly, within the framework of a traffic network, the overall flow of vehicles can fluctuate due to factors such as the volume of vehicles in circulation and the rate of vehicle flow. These changes can occur far away from the signalized intersection but can still affect the vehicle distribution at the intersection, thereby changing the perception of the environment for the agent controlling that intersection.

Convergence constitutes a challenge for MARLs [10]. Modification of the behavior of an agent during the learning phase can create the need for a consequent modification of the behavior of the other agents. This, in turn, may trigger further changes in the original agent’s strategy. These effects can yield unsteady dynamics in which the agents’ behaviour changes continuously, in an attempt to adapt to variations caused by other agents, without converging to a steady state. At the same time, MARL offers new opportunities as agents may share knowledge either locally or universally [11], i.e. communicate observations and, possibly, strategies, and imitate or directly learn from other learning agents, which may accelerate the learning process and subsequently result in more efficient ways to achieve an effective strategy [12].

The four Multi-Agent Reinforcement Learning methods evaluated in this work, namely MA2C, IA2C, IQL-DNN, and IQL-LR, have been extensively studied and represent popular approaches [13]. They differ in terms of agent communication and reward expression. The agents operate based on local observation in all cases. In the IA2C, IQL-DNN, and IQL-LR algorithms1, the agents are not allowed to communicate and the reward measures the global traffic. In contrast, in the MA2C case, agents are allowed to communicate by observing the strategies of nearby agents as they learn from a reward function that measures only local traffic. Due to the scope of the reward computation, MA2C exhibits a blend of cooperative and competitive characteristics, unlike other MARL algorithms which are predominantly cooperative in a game-theoretic sense. As a baseline, an algorithm with a greedy approach is adopted, releasing the longest vehicle queues when they are detected near the traffic lights [13].

The primary objective of the present study is to validate the theoretical framework of the algorithms and assess their performance in simulated real-world settings in terms of effectiveness, robustness, and applicability. These settings, two subsets of urban road networks of the city of Bologna, Italy (namely the Andrea Costa and Pasubio areas) are replicated in SUMO2.

The generalizability of the findings is evaluated through pseudo-random training. In this approach, the starting and ending points of each vehicle in the traffic network simulation are randomly determined. The randomness is regulated by seeds, which remain constant throughout different episodes before the training phase happen and are then modified. The assessment process involves testing the acquired strategies with different seeds, thereby utilizing various origin-destination (OD) pairs for each vehicle. This training protocol strikes a balance between completely random traffic conditions and choosing a single seed for the entire training, introducing a novel element.

1.1. Related Work

In the early eighties, the success of SCOOT (Split Cycle Offset Optimisation Technique) [15], developed in the UK, and SCATS (Sydney Coordinated Adaptive Traffic System) [16], developed in Australia, led to the popularity of ATSC. Both SCOOT and SCATS are real-time adaptive traffic control systems that automatically adjust traffic signal timings based on traffic conditions.

Several techniques have been applied to optimization problems of this kind in recent years, as fuzzy logic [17], swarm intelligence [18], genetic algorithms [19], and reinforcement learning (RL). RL includes q-learning, double q-learning, dueling networks, and Actor-Critic [7].

MARL-based algorithms for ATSC have been presented with a variety of approaches. Recent surveys on the topic highlight progress and downsides. These surveys include [20], which provides a classification of agent-based models on the dynamics time scale; [9], which summarizes the current issues for reinforcement learning applications in adaptive traffic signal control: increase of state–action complexity, problems in real-world implementations and proper comparison and evaluation of achieved results; [21] which focuses on robustness analysis of the approaches included in the survey; [22] which explores the extensive modeling of the beliefs, desires, learning, and adaptability of individuals and the optimization problems. Besides, the survey points out some limitations in terms of calibration and validation procedure, agents’ behavior modeling, and computing efficiency.

In summary, general issues are the size of search spaces, state observability, non-stationarity and reward design. Many studies necessitate meticulous tuning. Furthermore, the risk of an agent’s policy overfitting and failing to generalize to other traffic scenarios is a significant limitation when deploying such algorithms in real-world scenarios.

Model based approaches, in which state transitions are modeled by a probability distribution conditioned on the current state-action pair, are more efficient in terms of convergence and data exploitation. Building accurate probability distributions is a challenging task in the context of ATSC. Model free methods (with deterministic state transition) are more widely used in literature. The algorithms evaluated here are model free, following this trend [13].

In the following sections, technical details and performance analysis will be extensively discussed.

1.2. Outline

Section 2.1 provides an intuitive description of how a traffic network can be viewed as an instance of multi-agent reinforcement learning (MARL), while Section 2.2 formalizes these concepts from a single-agent perspective. The extension to the multi-agent case is discussed in Section 2.2.1. In Section 2.3, the theoretical foundations of MA2C and IA2C are presented and framed within a Partially Observable Decision Process (POMDP) scheme. The actor-critic machinery is reviewed first from a single-agent viewpoint (Section 2.4.1) and then from a multiple-agent perspective (Section 2.5), followed by the presentation of IA2C (Section 2.5.1) and MA2C (Section 2.5.2) formalism. Such formalism includes the need of considering past state history to predict adequate agents’ strategies as detailed in Section 2.3.

The implementation of the described algorithms is elaborated in Section 2.6 and Section 2.6.1. Section 3 delineates the conducted experiments, while Section 4 provides an overview of recent theoretical tools for assessing convergence and stability guarantees in multi-agent reinforcement learning (MARL) systems. Section 5 presents a summary of the results regarding general vehicle flow. Conclusions and future directions are reported in Section 6.

2. Material and Methods

Figure 1 shows a typical experiment setting: a traffic network including multiple signalized intersections controlled by corresponding agents. Every intersection contains one or more crossroads, each including a number of lanes.

2.1. System Overview

All the methods considered in this work (IQL-DNN, IQL-LR, IA2C, and MA2C) employ agents which observe the nearby vehicle flow 3 and return an action vector. The returned actions determine traffic light switching. In MA2C agents also observe the behavior of their neighboring agents, i.e. the strategy of agents located in nearby intersections. Agents trained with MA2C receive more information than the agents trained with other methods. In all cases they operate on the basis of partial information (they are not able to observe the entire state of the environment).

Another peculiarity of MA2C lies in the way the reward is calculated. In IQL-DNN, IQL-LR and IA2C, the DP reward is averaged over the whole traffic network; in MA2C, instead, the reward takes into account local traffic, i.e. the traffic in the vicinity of each agent and its neighbors. The two methods each have advantages and disadvantages. Using a measure of the overall traffic forces the agents to cooperate, but drives the learning process of each agent with a measure that is only weakly correlated to the agent’s actions. Adopting a measure of the local traffic, on the other hand, does not guarantee cooperation, but rewards each agent based on a measure that more directly depends on its actions. The latter method can promote competition between distant agents, yielding convergence towards better strategies in each group of agents. These differences among IQL-DNN, IQL-LR, IA2C and MA2C also impact the stability of the learning dynamics, as highlighted in tests.

2.2. System Formalization

From an agent’s standpoint, the problem of coordinating signalized intersections can be formalized as it follows: every agent (i.e. every signalized intersection) aims to minimize the amount of queuing vehicles (reward 4 ) by observing their motion in its neighborhood (i.e. by observing its neighborhood state) and ultimately learns how to balance its actions (control of traffic light switching) with the other agents. The IA2C agent type observes its neighborhood and, seeking its best possible strategy (policy 5 ), encodes in its reward computation the vehicle queuing at traffic lights in the whole traffic network (environment). On the other hand, the MA2C agent type restricts the environment reward to its neighbourhood (Figure 2) and embeds the observation of its neighbors’ policy (fingerprints) in its state.

2.2.1. Multi-Agent Formalization

Formally, a network of agents can be represented by a graph $G(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ (vertices) is the set of the agents and $\mathcal{E}$ (edges) is the set of their connections. Agent i and agent j are considered neighbors if the number of edge crossings needed to move from i to j is less than or equal to a chosen threshold. In the adopted formalism, the neighborhood of agent i is denoted ${\mathcal{N}}_i$ and its local region is ${\mathcal{V}}_i$ = ${\mathcal{N}}_i\cup i$. The distance between any two agents is denoted as $d(i,j)$ with $d(i,i)=0$ and $d(i,j)=1$ for any $j\in {\mathcal{N}}_i$.

The following list summarizes how the above introduced entities are defined in this setting:

• Agent: signalized intersection
• Environment: traffic network
• Reward:
  The one-step reward for agent i at time t cumulates the queues (number of vehicles with speed less than 0.1 m/s) at the lanes corresponding to a certain signalized intersection in the interval $\left[t,t+\Delta t\right]$:

  $$r_{t,i}=\sum _{l\in L_i}queu{e}_{t+\Delta t}\left[l\right]$$

  (1)

  where $L_i$ is the set of lanes converging at a signalized intersection (agent) i. The one-step reward is accrued over multiple steps to form the Reward perceived by agent i (this step is different in MA2C, with respect to the other algorithms, as detailed in the following sections).
• State (IA2C, IQL-DNN and IQL-LR):
  The state of the IA2C, IQL-DNN and IQL-LR agents at time t, $s_t$, is computed by:

  $$s_{t,i}={\left\{wav{e}_t\left[l\right]\right\}}_{l\in L_i}$$

  (2)

  where $wav{e}_t$ [veh] measures the total number of vehicles in the set of incoming lanes ($L_i$) at time t, within 50m from a signalized intersection (agent i).
• Action: $u_{t,i}$ is the action of agent i at time t (traffic light setting)
• Policy:
  agent i’s policy at time t, ${\pi }_{t,i}$, is a the probability distribution over its available actions
• Fingerprints:
  policy vector set of agent i’s neighbor agents (${\pi }_{t,-i}$)
• State (MA2C):

  $$s_{t,i}=[\alpha ·s_{t,i}^{IA2C},{\pi }_{t,-i}]$$

  (3)

  where $\alpha$ is a multiplication factor limiting the observation of neighbor agents ($\alpha <1$) and $s_{t,i}^{IA2C}$ is the state as defined in eq. (2).

In this work, the states of all the considered agents will simply be referred to as $s_t$ when the context prevents confusion.

2.3. Background

In [13], all algorithms were formulated under the assumption of a Markov environment. As outlined in Section 2.2.1, the observation for each agent depends on a subset of the entire state space. To expound further, in IA2C, IQL-DNN, and IQL-LR, the observation is confined to the vehicle dynamics proximal to each individual agent. In contrast, in MA2C, each agent possesses an additional capability to observe both the vehicle dynamics and the strategy employed by its neighboring agents, providing a broader perspective, albeit still incomplete. The system should be framed in the framework of Partially Observable Decision Process (POMDP). A common way to allow the agent to observe its past states is by adopting a Long Short-Term Memory Network [23] as detailed in Section 2.6.1.

MA2C and IA2C rely on the history of past state observations stored in an LSTM network. In this framework the goal of the algorithm is to learn limited-memory stochastic policies $\pi (u_t\mid h_t)$, i.e. policies that map sufficient statistics of a sequence of states $h_t$ to probability distributions on actions. For IQL-DNN and IQL-LR, refer to [13] for a fully detailed description of the algorithms.

2.4. Markov Decision Process

Reinforcement learning (RL) can be used to find optimal solutions for many problems. A Markov decision process (MDP) constitutes an adequate framework for Markovian approaches such as IQL-DNN and IQL-LR. An MDP can then be defined as a tuple $\left(\mathcal{S},\mathcal{U},P,R,\gamma \right)$ with its contents defined as follows:

• $\mathcal{S}$ is a set of states, where $s_t\in \mathcal{S}$ denotes the agent’s state at time t.
• $\mathcal{U}$ is a set of available actions, where $u_t\in \mathcal{U}$ denotes the action the agent performs at time t.
• $p:\mathcal{S}\times \mathcal{U}\times \mathcal{S}\to [0,1]$ is a transition function and denotes the probability of arriving at state $s_{t+1}$ when action $u_t$ is performed in state $s_t$.
• $r:\mathcal{S}\times \mathcal{U}\times \mathcal{S}\to \mathbb{R}$ is a reward function where $r_{t+1}$ is the reward when the agent transitions from state $s_t$ to state $s_{t+1}$ after performing action $u_t$.
• $\gamma \in [0,1]$ is a discount factor.

The transition function p is often referred as the model of the system as it captures its dynamics. All the algorithms considered in this work are model free, i.e. every action $u_t$ yields a state change $s_t\to s_{t+1}$ with probability 1, and all other transitions have probability 0. In general $\mathcal{U}$ may depend on $s_t$ but in the studied case (signalized intersections) $\mathcal{U}$ doesn’t depend on $s_t$.

2.4.1. Single Agent Recurrent Policy Gradients

For IA2C and MA2C, which rely on partial observability and are equipped with a memory as discussed in Section 2.3, the MDP described in Section 2.4 is not an adequate description. In this case the environment still produces a state $s_t$ at every time step but state transitions are governed by a machinery dependent upon its previous actions $u_{1:t}$ and the previous states $s_{1:t}$ of the system.

In this more general setting, the agent has a memory of its experience consisting of finite episodes. Such episodes are generated by the agent’s behaviour in the environment: in each time step t, the agent observes $s_t$, performs action $u_t$, and receives a reward $r_t$ whose expectation depends solely on $u_t$ and $s_t$. We define the observed history $H_t$ as the sequence of states and actions from the beginning of the episode up to time t: $H_t=\left[s_0,u_0,\dots ,s_{t-1},u_{t-1},s_t\right]$.

With an optimal or near-optimal policy for a POMDP, the action $u_t$ is taken depending on the preceding history.

Let $R\left(H_t\right)$ be a measure of the total reward accrued during a sequence of states, and let $p(H_t\mid \theta )$ be the probability of a state sequence given policy-defining weights $\theta$. An optimal policy will minimize

$$\tilde{\mathcal{L}}\left(\theta \right)=\int p(H_t\mid \theta )R\left(H_t\right)d{H}_t$$

(4)

This, in essence, indicates the expected reward over all possible (selectively remembered) sequences, weighted by their probabilities. Using gradient descent to update parameters $\theta$:

$$\begin{array}{cc}\hfill {\nabla }_{\theta }\tilde{\mathcal{L}}\left(\theta \right)& ={\nabla }_{\theta }{\mathbb{E}}_{H_t}\left[R\left(H_t\right)\right]=\hfill \\ \hfill =\int {\nabla }_{\theta }p\left(H_t\right)R\left(H_t\right)d{H}_t& =\int p\left(H_t\right){\nabla }_{\theta }logp\left(H_t\right)R\left(H_t\right)d{H}_t\hfill \end{array}$$

(5)

using the likelihood-ratio "trick" [24] and the fact that ${\nabla }_{\theta }R\left(H_t\right)=0$ as the accrued reward $R\left(H_t\right)$ does not depend on $\theta$. Accordingly, the loss gradient can be rewritten as:

$${\nabla }_{\theta }\tilde{\mathcal{L}}\left(\theta \right)={\mathbb{E}}_{H_t}\left[{\nabla }_{\theta }logp\left(H_t\right)R\left(H_t\right)\right]$$

(6)

where $p\left(H_t\right)$ is the only term depending on $\theta$.

In the above computations, the whole state-action sequence is denoted with $H_t$. The entire sequence is unnecessary, and the agent only needs to learn sufficient statistics (referred to as history hereafter) $h_t=S\left(H_t\right)$ of the events, representing the limited memory of the agents’ past. Thus, a stochastic policy $\pi$ and history $h_t$ can be redefined by the position $\pi \left(u\mid h_t;\theta \right)=\pi \left(u|S\left(H_t\right);\theta \right)$6, and implemented as a Long Short-Term Memory (LSTM) network with weights $\theta$. This produces a probability distribution $\pi (u\mid h_t)$, from which actions are drawn. Accordingly, $p\left(H_t\right)$ can be further subdivided:

$$p\left(H_T\right)=p\left(s_0\right)\prod _{t=1}^{T}p\left(s_t\mid u_{t-1},s_{0:t-1}\right)\pi \left(u_{t-1}\mid h_{t-1}\right)$$

(7)

As in the Markov policy gradient case, all the p terms will be split into a sum of logarithms and be removed, since they do not depend on $\theta$.

Performing Monte Carlo sampling for the whole episode length, the loss $\tilde{\mathcal{L}}\left(\theta \right)$ can be approximated as in the following formula:

$$\tilde{\mathcal{L}}\left(\theta \right)\underset{\sim }{\propto }\sum _{i=0}^{N_{sa}-1}\sum _{i=0}^{N_{st}-1}log\pi \left(u_{t,i}|h_{t,i}\right){\widehat{R}}_{t,i}$$

(8)

where $N_{sa}$ is the number of sampled episodes and $N_{st}$ is the number of time steps in an episode.

Introducing batch learning and iterating over sampled episodes, the training procedure aims to minimize the following loss.

$$\mathcal{L}\left(\theta \right)=\sum _{t=0}^{t_B-1}log\pi \left(u_t|h_t\right){\widehat{R}}_t$$

(9)

where the episode index has been omitted.

In equation (9), $t_B$ is the number of time steps in the batch; ${\widehat{R}}_t$ is the aggregated reward over the batch, given by ${\widehat{R}}_t={\sum }_{\tau =t}^{t_B-1}{\gamma }^{\tau -t}{r}_{\tau }$, where the usual $\gamma$ discount factor has been introduced [6]; $r_{\tau }$ is sampled from $\mathbb{E}\left[r_{\tau }^{\pi }\right]$.

2.4.2. Single Agent Recurrent Advantage Actor Critic

In accordance with the formulation in Section 2.4.1, the derivation of the loss functions for Recurrent Advantage Actor-Critic (A2C) in the context of a POMDP is undertaken. The Actor’s loss is:

$$\mathcal{L}\left(\theta \right)=\sum _{t=0}^{t_B-1}log{\pi }_{\theta }\left(u_t|h_t^{\pi }\right)A_t$$

(10)

while the Critic’s loss is:

$$\mathcal{L}\left(\psi \right)={\displaystyle \frac{1}{2}}\sum _{t=0}^{t_B-1}{\left(R_t-V_{\psi }\left(h_t^V\right)\right)}^2$$

(11)

where:

• $h_t^{\pi }$ and $h_t^V$ are the histories of past events registered by the two LSTM networks used to compute the regressors ${\pi }_{\theta }$ and $V_{\psi }$
• $R_t={\widehat{R}}_t+{\gamma }^{t_B-t}{V}_{{\psi }^{-}}\left(h_{t_B}^V\right)$ is the n-step return [6], with $n=t_B-t$; the regressor for state value at $t_B$ (corresponding to the first time step of the next batch) is evaluated using the current parameters ${\psi }^{-}$
• $A_t=R_t-V_{{\psi }^{-}}\left(h_t^V\right)$ is the sampled advantage for actions $\{u_t,...,u_{t_{B-1}}\}$

In the above definitions, ${\psi }^{-}$ is the set of the Critic’s parameters computed at the last learning step (current Critic’s parameters); accordingly, in the next sections ${\theta }^{-}$ will denote the set of the Actor’s parameters computed at the last learning step (current Actor’s parameters) 7. Notably, the inner states of the Critic’s LSTM are not updated when computing $V_{{\psi }^{-}}\left(h_{t_B}^V\right)$ as $t_B$ corresponds to the first time step of the next batch. This prevents the state sequence (history) from being compromised when the state value is computed while processing the next batch.

2.5. Multi-Agent Policy-Gradient Systems

2.5.1. Independent Advantage Actor-Critic (IA2C)

In IA2C, each agent i learns its own policy ${\pi }_{{\theta }_i}$ and value function $V_{{\psi }_i}$ collecting experiences from its own DP. The observation of agent i is restricted to ${\mathcal{V}}_i$, i. e. $s_{t,{\mathcal{V}}_i}:={\left\{s_{t,j}\right\}}_{j\in {\mathcal{V}}_i}$.

The loss functions for the Actor and Critic are:

$$\mathcal{L}\left({\theta }_i\right)=\sum _{t=0}^{t_B-1}log{\pi }_{{\theta }_i}\left(u_{t,i}|h_{t,{\mathcal{V}}_i}^{\pi }\right)A_{t,i}+\beta \sum _{u_i\in {\mathcal{A}}_i}{\pi }_{{\theta }_i}log{\pi }_{{\theta }_i}\left(u_i\mid h_{t,{\mathcal{V}}_i}^{\pi }\right)$$

(12)

$$\mathcal{L}\left({\psi }_i\right)={\displaystyle \frac{1}{2}}\sum _{t=0}^{t_B-1}{\left(R_{t,i}-V_{{\psi }_i}\left(h_{t,{\mathcal{V}}_i}^V\right)\right)}^2$$

(13)

where:

• $A_{t,i}=R_{t,i}-V_{{\psi }_i^{-}}\left(h_{t,{\mathcal{V}}_i}^V\right)$
• $R_{t,i}={\widehat{R}}_t+{\gamma }^{t_B-t}{V}_{{\psi }_i^{-}}\left(h_{t_B,{\mathcal{V}}_i}^V\right)$
• ${\widehat{R}}_t={\sum }_{\tau =t}^{t_B-1}{\gamma }^{\tau -t}{\widehat{r}}_{\tau }$
• ${\widehat{r}}_t={\displaystyle \frac{1}{\left|\mathcal{V}\right|}}{\sum }_{i\in \mathcal{V}}{r}_{t,i}$
• $h_{t,{\mathcal{V}}_i}^{\pi }=S^{\pi }\left(H_{t,{\mathcal{V}}_i}\right)$
• $h_{t,{\mathcal{V}}_i}^V=S^V\left(H_{t,{\mathcal{V}}_i}\right)$
• $H_{t,{\mathcal{V}}_i}=\left[\left\{s_{0,j}\right\},u_0,...,\left\{s_{t-1,j}\right\},u_{t-1},\left\{s_{t,j}\right\}\right]$ with $j\in {\mathcal{V}}_i$

In the above, ${\theta }_i$ and ${\psi }_i$ are the Actor and Critic’s network parameters, and a ’-’ superscript indicates their current values; $-i$ as a parameter subscript refers all the agents except agent i.

When training begins, all actions have similar probability. To encourage exploration, the entropy loss of policy ${\pi }_{{\theta }_i}$ has been added in (12), weighted by the parameter $\beta$.

Restricting the observation of agent i to ${\mathcal{V}}_i$ affects both $V_{{\psi }_i}$ and ${\pi }_{{\theta }_i}$ via $H_{t,{\mathcal{V}}_i}$. On the one hand, this yields partial observability while on the other hand, since $R_{t,i}$ is dependent on the whole $s_t$, it allows ${\mathcal{V}}_i$ and ${\pi }_{{\theta }_i}$ to converge towards optimum values specific to agent i. This is especially important for $V_{{\psi }_i}$ as, without a restriction on the agent’s observation, all the agents will seek the same global optimum $V^{*}$, while every ${\pi }_{{\theta }_i}$ has its own specification in terms of type and number of available actions.

2.5.2. Multi-Agent Advantage Actor-Critic (MA2C)

MA2C [13] improves the stability of the learning process by allowing some communication among agents belonging to the same neighborhood; a spatial discount factor weakens the reward signals from agents other agents than i in the loss function, and agents not in ${\mathcal{N}}_i$ are not considered in the reward computation. Equations (12) and (13) become:

$$\begin{array}{c}\hfill \mathcal{L}\left({\theta }_i\right)=\sum _{t=0}^{t_B-1}log{\pi }_{{\theta }_i}\left(u_{t,i}|{\tilde{h}}_{t,{\mathcal{V}}_i}^{\pi },{\pi }_{t-1,{\mathcal{N}}_i}\right){\tilde{A}}_{t,i}\\ \hfill +\beta \sum _{u_i\in {\mathcal{A}}_i}{\pi }_{{\theta }_i}log{\pi }_{{\theta }_i}\left(u_i|{\tilde{h}}_{t,{\mathcal{V}}_i}^{\pi },{\pi }_{t-1,{\mathcal{N}}_i}\right)\end{array}$$

(14)

$$\mathcal{L}\left({\psi }_i\right)={\displaystyle \frac{1}{2}}\sum _{t=0}^{t_B-1}{\left({\tilde{R}}_{t,i}-V_{{\psi }_i}\left({\tilde{h}}_{t,{\mathcal{V}}_i}^V,{\pi }_{t-1,{\mathcal{N}}_i}\right)\right)}^2$$

(15)

In the above equations:

• ${\tilde{A}}_{t,i}={\tilde{R}}_{t,i}-V_{{\psi }_i^{-}}({\tilde{h}}_{t,{\mathcal{V}}_i}^V,{\pi }_{t-1,{\mathcal{N}}_i})$
• ${\tilde{R}}_{t,i}={\widehat{R}}_{t,i}+{\gamma }^{t_B-t}{V}_{{\psi }_i^{-}}\left({\tilde{h}}_{t_B,{\mathcal{V}}_i}^V,{\pi }_{t_B-1,{\mathcal{N}}_i}\right)$
• ${\widehat{R}}_{t,i}=$${\sum }_{\tau =t}^{t_B-1}{\gamma }^{\tau -t}{\tilde{r}}_{\tau ,i}$
• ${\tilde{r}}_{t,i}={\displaystyle \frac{1}{\mid {\mathcal{V}}_i\mid }}(r_{t,i}+{\sum }_{j\in {\mathcal{V}}_i,j\ne i}\alpha r_{t,j})$
• ${\tilde{h}}_{t,{\mathcal{V}}_i}^{\pi }=\left\{h_{t,i}^{\pi }\right\}\bigcup \alpha \{h_{t,j}^{\pi },j\in {\mathcal{N}}_i\}$
• ${\tilde{h}}_{t,{\mathcal{V}}_i}^V=\left\{h_{t,i}^V\right\}\bigcup \alpha \{h_{t,j}^V,j\in {\mathcal{N}}_i\}$
• ${\tilde{h}}_{t,{\mathcal{V}}_i}^{\pi }={\tilde{S}}^{\pi }\left({\tilde{H}}_{t,{\mathcal{V}}_i}\right)$
• ${\tilde{h}}_{t,{\mathcal{V}}_i}^V={\tilde{S}}^V\left({\tilde{H}}_{t,{\mathcal{V}}_i}\right)$
• ${\tilde{H}}_{t,{\mathcal{V}}_i}=\left[\left\{s_{0,i}\right\}\bigcup \alpha \left\{s_{0,j}\right\},u_0,...,\left\{s_{t-1,i}\right\}\bigcup \alpha \left\{s_{t-1,j}\right\},u_{t-1},\left\{s_{t,i}\right\}\bigcup \alpha \left\{s_{t,j}\right\}\right]$
  with $j\in {\mathcal{V}}_i$

The spatial discount factor $\alpha$ penalizes other agent’s rewards8, and $D_i$ is the limit of agent i’s neighborhood.

Equation (15) yields a more stable learning process since (a) fingerprints ${\pi }_{t-1,{\mathcal{N}}_i}$ are input to $V_{{\psi }_i}$ to bring in account ${\pi }_{{\theta }_{-i}^{-}}$, and (b) spatially discounted return ${\tilde{R}}_{t,i}$ is more correlated to local region observations $\left({\tilde{s}}_{t,{\mathcal{V}}_i},{\pi }_{t-1},{\mathcal{N}}_i\right).$ By inspection of equations (12, 13, 14, 15), the peculiarities of MA2C with respect of IA2C can be summarized as: 1) MA2C uses fingerprints, 2) MA2C uses a spatial discount factor, 3) the overall reward (${\widehat{R}}_{t,i}$ in MA2C) is computed on ${\mathcal{V}}_i$ instead of the whole set of agents $\mathcal{V}$.

2.6. System Architecture

Figure 3 provides an overview of the system.

As stated in the previous sections, the goal is to minimize the vehicle queues measured at signalized intersections. To this end, the following steps are repeated: 1) An ANN takes as an input the perceived state of the environment and provides a policy. For MA2C, the environment state includes the policies of its neighbor agents (fingerprints). 2) An action is sampled from the provided policy and passed to the traffic simulator, which performs a simulation step and returns the new environment state and last episode flag (a flag indicating whether the episode is over) to the ANN. The simulator also stores actions, rewards, and the last episode flag in a buffer. These two steps are repeated until the buffer (here denoted batch) is full. 3) the ANN uses the stored rewards to change its parameters and improve its policy.

2.6.1. ANN Detail

For IQL-DNN and IQL-LR, see to the multi-perceptron ANN details in [13]. For MA2C and IA2C, states, actions, next states and rewards are collected in batches called experience buffers, one for each agent i: $B_i={\left\{\left(s_t,u_t,s_{t+1},r_t\right)\right\}}_i$. Batches are stored while the traffic simulator performs a sequence of actions. Each batch i reflects agent i’s experience trajectory. Figure 4 shows the IA2C and MA2C architectures.

This architecture is used for each agent so 2$\mid \mathcal{V}\mid$ ANN’s are instantiated. In IA2C (left), the first layer is formed by a variable number of input neurons that encode wave measures, i.e. the total number of approaching vehicles along each incoming lane, within 50m. The second layer is formed by 128 neurons. The third layer is formed by 64 LSTM units. The first layer is fully connected to the second layer. The third layer is fully connected to itself and to the fourth layer.

In the case of the MA2C method (Figure 4., right), the first layer is formed by a variable number of input neurons encoding wave measures and by a variable number of neurons encoding the action vectors of nearby agents. The second layer is formed by two groups of 128 and 64 neurons that receive connections from the two groups of sensory neurons described above. The third layer is formed by 64 LSTM units. The first layer is fully connected to the second layer; the third layer is fully connected to itself and to the fourth layer.

The network architecture reflects the A2C formalism [25]; each graph represents two different networks, one for the Actor (policy) and one for the Critic (state value), with their respective parameters here denoted as $\theta$ and $\psi$. The fourth layer of the Actor network is formed by N neurons with a softmax activation function, which encode the probabilities of choosing one of the N corresponding possible red-green-yellow transitions of the corresponding traffic light. The fourth layer of the Critic network is a single neuron that encodes the expected reward. The neurons of the second and third layer use the rectified linear unit (ReLU) activation function. For ANN training, an orthogonal initializer [43] and an RMSprop 9 gradient optimizer have been used. To prevent gradient explosion, all normalized states are clipped to [0, 2] and each gradient is capped at 40. Rewards are clipped to [-2, 2].

3. Calculation

We evaluate MA2C, IA2C, IQL-DNN and IQL-LR in SUMO-simulated traffic environments replicating two districts in the Bologna area (Andrea Costa and Pasubio) [26]. While these traffic networks are similar in the required number of agents, they differ significantly in their topology, allowing evaluation of how topology influences the learning process. Every episode of the SUMO simulation consists of 3600 time steps. For a pre-determined interval of time, a vehicle is inserted in the traffic network at each time step a with a random origin-destination (OD) pair.

To ensure a fair evaluation of the algorithms, identical settings for experimental parameters (except those parameters specific to each algorithm [13]) have been adopted. The criterion used to evaluate the algorithms’ performance is the queue at the intersections, which is linked to the DP reward by equation (1). Queues are estimated within SUMO for each crossing and then passed as rewards. In testing, the algorithms have been compared with a greedy approach (hereafter denoted Greedy) which simply selects the traffic light phase associated with the largest $wave$ over all incoming lanes. For the pseudo-random experiments reported in the following sections, the seed is kept constant during collecting training samples, while different seeds are used in the 8 testing episodes 10. For the fully stochastic experiment (reported in Section 4), the seeds are kept constant only during each episode; when testing, seeds are changed for each of the 8 episodes, like in the pseudo-random case. For brevity, “veh" is a shorthand for “vehicles”.

The DP is instantiated with the settings listed in Table 1.

The size of the batch indirectly sets the n parameter of the n-step return appearing in Equations (12), (13), (14), and (15) and has been chosen to balancing the complementary characteristics of TD and Monte-Carlo methods [6]. With a forward step limit at 1000800 and the settings reported in Table 1 for $T_s$, $\Delta$t and $\mid B\mid$, a complete learning process results in 278 episodes (i.e. 5004 learning steps).

3.1. Algorithms

The presented tests do not rely on a structured configuration of the vehicles in motion 11. In fact, assuming a traffic macro-behavior in terms of general vehicle flows is beneficial when the flows effectively reflect the real traffic macro-behavior; but this is seldom the case in practice and such assumptions might instead cause overfitting [27] originating from spurious regularities. To avoid this issue, pseudo-random Origin Destinations (OD) pairs are assigned to each vehicle. This allows the learning process to achieve a more general optimum as the equilibrium among the agents doesn’t depend on any specific assumption on the flows or boundary condition. Our experimentation consists of the following steps:

• Loading traffic networks with a varying number of vehicles.
• Evaluating the training graphs of IQL-DNN and IQL-LR.
• Comparing the IA2C and MA2C learning curves
• Comparing MA2C with IA2C and Greedy
• Evaluating the behavior of the learning curves for IA2C and MA2C
• Evaluating the effects of the learned policies in terms of traffic volumes

3.2. Traffic Networks

Figure 5 (left) shows the Andrea Costa neighborhood of Bologna [26].

The round/elliptic (purple) shaded spots identify the signalized intersections (agents). The right side of the figure shows the way each agent is connected to the others. The set of all the agents connected to a single agent constitutes its neighborhood.

Figure 6 (left) shows the Pasubio neighborhood of Bologna [26]. As in the Andrea Costa case, the right hand side of the figure shows how the agents have been connected.

The vehicle insertion procedure over each episode of 3600 time steps is divided into the following alternatives:

• One vehicle inserted per time step for the first 2000 time steps, then no more vehicles for the remaining time steps, totalling 2000 vehicles
• One vehicle inserted per time step for all 3600 time steps, totalling 3600 vehicles
• Two vehicles inserted per time step for the first 2000 time steps, then no more vehicles for the remaining time steps, totalling 4000 vehicles

3.3. IQL-DNN and IQL-LR

In agreement with the results reported in [13] 12, IQL-DNN and IQL-LR algorithms perform rather poorly in the traffic networks, even in the simplest case (2000 vehicles). Figure 7 and Figure 8 show the performance of the two algorithms in terms of the overall average queue13. The graphs show that the algorithms have a divergent trend and then stabilize around 500.000 interactions to a value significantly worse than the original setting 14. In view of this result IQL-DNN and IQL-LR have been excluded from further experimentation with real traffic networks, as was done in [13]15

3.4. MA2C and IA2C Performance on the A. Costa Traffic Network

Figure 9 and Figure 10 show the minimization of the average queue on the traffic network loaded with 2000 and 3600 vehicles. While both the MA2C and IA2C learning processes achieve a similar minimum, the curves appear very different: while MA2C achieves a stable point from iteration $\sim 100$ onward, IA2C is much less steady. Particularly in Figure 10 (load of 3600 vehicles), IA2C achieves its optimum (which, as shown in Figure 15 is the only success out of 10 learning runs) only in the very end of the process.

Table 2 and Table 3 show the average queue length for 8 tests, with different random seeds for each test. The average and standard deviation across these tests are also shown. MA2C appears better than IA2C in Table 2, and evidently outperforms Greedy.

The average queue measured is similar to the achieved minimum in the training process, showing that even though the system was trained with pseudo-random simulations, the final result doesn’t appear to overfit on specific sets of origin-destination (OD) pairs. This is especially important because it verifies that the learning process focuses only on dynamics occurring in the vicinity of the traffic lights. RL is known to have a tendency to overfit when the training dataset is a subset of all potential cases [27]; training with real data is challenging for this application as transforming integrated traffic flow data into OD matrices requires mixed heuristic-deductive reasoning [29].

In the next sections, this relevant finding will be confirmed by further experimentation.

3.5. MA2C and IA2C on the Pasubio Traffic Network

Figure 11 shows the training graph for the case of 2000 vehicles circulating in the Pasubio traffic network. The figure shows that MA2C is able to achieve an improvement while IA2C showed a much more oscillatory behavior during training. As a result, when measuring performances with 8 different test seeds, the average queue measured are very different in the two cases (Table 4 and Table 5). Notably, in Table 4, MA2C exhibits a slightly worse behaviour than the Greedy approach.

The case of 3600 vehicles was more challenging for MA2C and IA2C (Figure 12). While optimization of IA2C failed across 10 learning runs, MA2C was successful in 6 runs out of 10 (Figure 15). In testing, MA2C was more effective than the Greedy policy, which showed a larger average and a much larger variance in the 8 tests.

3.6. Further Tests

To further evaluate the capabilities of MA2C in managing an increased vehicle flow, 2 vehicles per time step for the first 2000 time steps have been inserted, totalling 4000 vehicles in the traffic network. We carried out 10 training attempts, for both the Andrea Costa and Pasubio traffic networks. Across these experiments, MA2C behaved similarly. The loss increased in the initial training stage, followed by learning in the following steps, but without substantial improvement in terms of the average queue of vehicles as shown in Figure 13.

Though this behaviour highlights MA2C’s limitations in dealing with high volumes of traffic, the next sections will show that MA2C still represents a better alternative than the other algorithms, even in this case.

4. Convergence Stability

This section is devoted to the convergence analysis of the learning process: given the IA2C and MA2C algorithms, the goal is to evaluate if there are theoretical guarantees for the loss minimization process towards a relative minimum. To this end, three issues should be considered: first, the non-Markovian nature of the decision process; second, the non-stationary aspect of the problem: besides the inherent non-stationary nature of the environment, when the agents’ policies evolve during the learning process, from the viewpoint of each agent, the environment changes, due to the changing policies of the other agents; third, each agent has access only to a subset of the overall system state, namely the state of the neighboring agents.

As depicted in Figure 2, IA2C and MA2C differ in the computation of the DP reward, which, in turn, enters in the computation of the loss: while IA2C considers the contribution of all the agents in the traffic network, MA2C restricts its scope to each agent’s neighborhood. This makes the stochastic game determined by the IA2C agents fully cooperative, while MA2C is mixed cooperative-competitive. The input of the neural network adopted to determine value and policy is different in the two cases: both the approaches refer each agent’s neighborhood but only MA2C uses as an input the learning policies from the other agents. Presently, there is no theory general enough to encompass the complexity of IA2C and MA2C. In the following sections, the theory framework behind general Multi-Agent Reinforcement Learning (MARL) problems is surveyed, emphasizing the theoretical limits of the convergence guarantees. These limits hinder their application to IA2C and MA2C.

4.1. Theoretical Evaluation of the Learning Process

In [30] an approach is proposed that links instances of NMDP to probabilistic-deterministic finite automata yielding regular decision processes (RDP). Following this proposed method, it is possible to define a Markov abstraction [30] that reduces the problem to solving an MDP instead of an RDP. In general, LSTM cells have the potential of relying on infinite histories, but in episodic RL the history length is limited by the length of episodes and can be modeled with finite automata.

Formally, a Markov abstraction over a finite set of states S is a function $\alpha :H\to S$ such that, for every two histories $h_1\in H$ and $h_2\in H$ and every action $u\in \mathcal{U}$, $\alpha \left(h_1\right)=\alpha \left(h_2\right)$ implies $D(·|h_1,u)=D(·|h_2,u)$, where $D(·|h,u)$, the dynamics function, describes the probability of observing a certain pair of observation and reward (or episode termination) in the next time instant, given a certain history h and action u. Since the dynamics are the same for every history mapped to the same state, there is an MDP $M_{\mathcal{P}}^{\alpha }$ with dynamic function $D^{\alpha }$ (the induced MDP) which can be solved in place of the original RDP $\mathcal{P}$ [30]. It is straightforward to see that the machinery due to the presence of an LSTM cell in the ANN shows no significant difference with the Markov abstraction defined in [30] for the case of a single-agent DP. For the multi-agent case, there is no such correspondence, as any such abstraction must account for the number of agents participating in the stochastic game and any limits on the rate of change of their policies.

Work on convergence of actor-critic decentralized MARL has been reported in [27,31,32] and in [33], which focuses on vanilla Actor-Critic. In [31] the proposed theoretical tools extend the approach described in [27] to the case of deterministic policies. As confirmed in [32] and [31] at the present [27] represents the state of the art on theoretical studies on stability and convergence of the learning process. The non-stationarity issue is not extendable to the multi-agent case as "theoretically, if the agent ignores this issue and optimizes its own policy assuming a stationary environment [...] the algorithms may fail to converge, except for several special settings. Empirically independent learning may achieve satisfiable performance in practice" ([34]). in [27] there are theoretical tools able to determine that the policy parameter ${\theta }_i$ (referring agent agent i) obtained from the presented A2C algorithm converges almost surely to a point in the set of asymptotically stable equilibria. Besides the above mentioned non-stationarity issue, in [27] there are limitations preventing the theory to be applied to the IA2C and MA2C cases. The most stringent limitations regard the state observation that in [27] is complete, while in both IA2C and MA2C is partial16, and the adoption of linear function approximation while IA2C and MA2C rely on non-linear ANNs.

To summarize, a theory able to formally provide convergence guarantees for the learning process in IA2C and MA2C is still lacking, due to difficulties in extending the link between NMDP and MDP to MARLs and some simplificative hypotheses when formally dealing with non-stationarity and observation (see [27] for details) that cannot be applied to IA2C and MA2C.

4.2. Experimental Evaluation of the Learning Process

Some heuristic analysis of the learning process is possible by inspecting the average queue curve during learning.

A tendency to diverge from asymptotic behaviour, oscillations, or spikes show that the agents have not found a steady balance: even when a solution is found, queues at each signalized intersection can both shrink and grow during a single episode. As discussed in the previous section, when ANNs are coupled with Reinforcement Learning, a challenging task arises from pursuing a changing target during the learning process. In the multi-agent case, targets are even more elusive: an agent, e.g. agent i, converges smoothly towards an optimal policy as long as ${\theta }_{-i}$ stay constant. This is not the case in a collective learning process where other agents’ changing policies ${\pi }_{{\theta }_{-i}}$ affect their behaviour. This in turn affects (from the decentralized view point) the expected return of agent i. Eventually, this sequence of causal relationships can make the learning process inconsistent. This issue can happen even if agent i finds an ideal policy: changes of ${\pi }_{{\theta }_{-i}}$ might cause the agent i to slip away from its optimum. Ideally, all the agents find their optima at roughly the same time.

As an example of this problem, Figure 14 shows the behaviour of the seven agents in the Andrea Costa case, when loaded with 2000 vehicles as described in Sec. Section 3.2. The top graphs show the behaviour in time of the Actor’s loss for agents $a_0,...,a_6$ during the learning process. The bottom graph shows the reward, as the averaged cumulative queues evolve in time. The left (red) shaded region highlights the inconsistency issue: at the beginning of the shaded region agent $a_1$ is slightly unstable (while the other agents appear to be close to a relative balance) and eventually drags away the other agents from their converging path. In the (green) shaded region on the right, instead, all the agents appear to move together to a common relative minimum; this is especially evident observing agents 0, 1 and 4.

To limit the occurrence of such inconsistencies it is beneficial to maintain the learning process as smooth as possible; this is achieved by keeping small $D_{KL}({\pi }_{{\theta }^{-}},{\pi }_{\theta })$17 by updating the target of the optimization process after each learning step.

The overall consistency of the algorithms during training can be evaluated by observing that the properties of the environment change significantly depending on quantitative factors. For this reason, the results obtained with IA2C and MA2C are compared in three experimental settings with varying number of vehicles introduced in the traffic networks: Figure 15 shows the number of training successes across 10 learning attempts for the cases with 1000, 2000 and 3600 vehicles in both the A. Costa and Pasubio networks.

As evident in Figure 15, MA2C is more stable than IA2C. The learning processes tend to become more inconsistent when the vehicle load increases, with the notable exception of the A. Costa case loaded with 2000 vehicles, which shows much more problematic than the case with 3600 veh. It is interesting to observe that, while this case is more unsteady than the 3600 case, the graph of the A. Costa case loaded with 2000 vehicles displays a much less oscillatory behaviour once convergence is achieved, as highlighted in the Figure 16 (shaded region).

Figure 16 shows that, when loaded with 1000 or 2000 vehicles, the achieved convergence stays stable while, when loaded with 3600 vehicles, the agents “keep struggling" even after reaching a balance among them, due to the excessive amount of vehicles circling on the traffic network.

4.3. Generalization

We performed further experiments to evaluate the generalization capability of the training process.

4.3.1. Random Learning

To test the tendency of the considered algorithms to focus on the signalized intersection dynamics, an attempt of training using totally random vehicle flows has been performed. This test differs from the previous ones as the random process seeds change at every simulation, instead of staying constant through the whole learning process. Even when adding only one vehicle per time step, no algorithm among those considered was able to achieve consistent positive results. On the Pasubio network with totally random training, only MA2C was able to optimize a policy within 10 attempts; this convergence is shown in Figure 17.

When testing this solution on random test seeds, no better performance was achieved with respect to the pseudo-random case. This result has involved finding a “happy medium” that lies between the extreme of imposing completely random traffic conditions and the extreme of imposing a single seed for the entire training. The solution identified, which offers the optimal balance between the stability of the learning phase and the generality of the solution, involves keeping the seed constant for all episodes within the minibatch, and then varying it thereafter.

Table 6 shows that the MA2C performance after training on random traffic is slightly worse than when trained with pseudo-random traffic.

This experiment confirms that pseudo-random training is a well-balanced approach: its performance when tested with other random seeds (changing the vehicle distribution in terms of origin-destination pairs) shows that it avoids overfitting to spurious traffic circumstances while also presenting a more manageable learning problem than that posed by total-random training.

4.3.2. Policy Cross-Evaluation

This section presents another strategy to evaluate MA2C attitude towards overfitting. So far a certain solution has been evaluated only with its respective test cases; for instance, a policy learned with 2000 vehicles has been tested with the same number of vehicles (though in different vehicle distributions). This section presents the result of the learning step with varying vehicle loads. As an instance, MA2C has been tested to see how well the policy learned with 2000 vehicles performs with 3600 vehicles. These experiments have relevance as they contribute to help assessing the quality of pseudo-random training by allowing evaluating how much the agents are able to focus on the vehicle dynamics at their surroundings, resisting overfitting to other spurious features such as the vehicle load and distribution in time and space. The tables in the next sections report the results of the experiments.

Training MA2C with 2000 Vehicles:

Table 7 shows the results of MA2C compared to the greedy approach. Out of the six experiments reported, MA2C outperforms the greedy approach in five, with the notable exception of via Pasubio in a 2000 vehicle case. It is interesting that, though trained with 2000 vehicles, MA2C continues to perform well when dealing with 3600 vehicles, indicating that the learned policy does focus on the dynamics of the vehicle in the agents’ proximity. While MA2C failed to learn a policy for 4000 vehicles in 2000 time steps, the chaotic traffic related to this case can be alleviated by using the policy learned with 2000 vehicles, which performs slightly but noticeably better than Greedy.

Training MA2C with 3600 Vehicles:

Table 8 shows the results when training MA2C with 3600 vehicles. In all the experiments MA2C showed better performances than Greedy. It is noticeable that in the via Pasubio case, MA2C trained with 3600 performs better that MA2C trained with 2000 vehicles, even when tested with 2000 vehicles.

5. Results

It was found the need to position the algorithms within the framework of a Partially Observable Markov Decision Process (POMDP). The equations were subsequently reformulated within this framework, shedding light on which algorithms are truly compliant with the underlying Decision Process.

Further experimental results show that MA2C outperforms both IA2C and the Greedy approach by exhibiting a more robust behaviour when aiming to reduce the vehicle queues at signalized intersections.

For this purpose, attention is directed towards the A. Costa experiment featuring 2000 vehicles. As outlined in the preceding sections, the typical traffic simulation covers 3600 time steps.

• In the first part of the simulation (time steps [0, 2000]) a vehicle is pseudo-randomly inserted on the map for each time step and follows a pseudo-random path.
• In the second part of the simulation (time steps [2000, 3600]) no vehicle is inserted. Eventually, all the vehicles circulating on the map leave through one of the exit lanes or end their journey by reaching their destination.

Figure 18 shows the number of running vehicles in the time span [0, 3600] for the cases Andrea Costa and Pasubio:

The curve referring to the beginning of the training ("no sync" in Figure 18) - when a synchronization among the agents is yet to be achieved - shows that due to heavy queuing at the traffic lights, several vehicles stay on the road after time step 2000. In the graph, the curve keeps rising while vehicles are injected and tends to slowly decrease afterwards. When the network is trained ("sync" in Figure 18), the amount of vehicles running fades quickly towards zero after time step 2000.

Figure 19 refers the Andrea Costa case and shows the behaviour of the ratio between the number of running vehicles with no sync over the same quantity in the sync case. The ratio increases and shows three distinct regions: a linear behaviour with two different slopes for $t\in [0;2000]$, an exponential behaviour for $t\in [2000;2500]$, and a more irregular behaviour for $t\in [2500;3600]$. The ratio at $t=2000$ is $\sim 3.2$, at $t=2500$ is $\sim 28.5$, while at $t=2500$ is $\sim 70$. The Pasubio case shows a similar pattern.

6. Conclusions

The development of multi-agent reinforcement learning methods for real-world problems represents a promising yet nascent research area [35]. The complexities arise when attempting to strike a balance in a cooperative-competitive stochastic game. The aim of this study is to validate the theoretical framework of the algorithms proposed in [13] and assess its effectiveness, robustness, and applicability in real-world scenarios. The theoretical framework of the algorithms has been critically reviewed and, given the overall state of partial observability, the Decision Process has been reframed within the context of a Partially Observable Markov Decision Process (POMDP), which is inherently non-Markovian. This is due to the observation space, which forces each agent to a subset of the total set of agents’ observations. The theory has been reformulated accordingly, as outlined in Section 2.3 and Section 2.4.1. This reformulation sheds light on the poorer performance of the class of Markovian algorithms (IQL-DNN, IQL-LR) compared to non-Markovian ones [13]. The study confirms these findings, as detailed in Section 3.3. Subsequently, the evaluation of IA2C and MA2C, compliant with the partially observable setting, takes place. In Section 3.4, these algorithms undergo testing in two areas of Bologna (Italy) replicated in SUMO, a software tool for ATSC problems.

The analysis of the learning process and the comparison of results obtained by varying the number of vehicles indicate that MA2C is sufficiently effective. Specifically, it is demonstrated that in cases where the load is 2000 vehicles, MA2C generates a traffic decrease of a factor $\sim 70$ at the end of the simulation. These results are replicated in both the studied areas.

Examining the results obtained by training the system with pseudo-random vehicles and evaluating it with total random traffic, reveals that this training strategy is resistant to overfitting. This finding represents a significant step towards addressing real-world traffic problems with MARL algorithms, eliminating the need to know specific traffic flows in advance for training while resulting in adequate compromise between stability of the learning phase and generality of the solution.

The limitations of the evaluated algorithms become apparent at high traffic loads ($>3600$ veh.); in these scenarios, both MA2C and IA2C perform comparably to a simple greedy approach. Future research may reveal whether there is a sharp transition to this behavior concerning the vehicle load.

Promising areas for future research include alternatives to LSTM cells, such as Echo State Networks (ESN) and Liquid State Machines, which can handle the chaotic behaviors of vehicle dynamics. These approaches also offer a faster training stage. Other directions aim to identify the dependency of the algorithm on the extent of the agents’ neighborhood and on specific settings such as the traffic network topology, the vehicle flow distributions, and origin-destination pairs.

As a final consideration, all simulations show that the process of balancing the agents’ actions is influenced by the number and interconnections among the agents (i.e., the signalized intersections), as well as by the traffic volume, which could be a typical situation in medium to large cities. To simplify the task, it may be beneficial to divide traffic network graphs into subgraphs. This strategy can be more efficiently implemented on less densely connected traffic networks, such as areas on the outskirts of cities, where agents tend to cluster in small groups.