Reinforcement Learning-based Resource Allocation Scheme of NR-V2X Sidelink for Joint Communication and Sensing

1. Introduction

In the future, vehicles on the road will need to frequently exchange information with surrounding vehicles, pedestrians, and road traffic infrastructure, which drives the development of Vehicle-to-Everything (V2X) technology. Today’s vehicles are equipped with numerous sensors and communication systems, having transformed from traditional vehicles into intelligent vehicles. Through V2X network, there is potential to achieve Advanced Driver Assistance System(ADAS), improving driving safety and comfort. Recently, the latest beyond 5G and 6G standards have introduced new requirements for V2X, including enhanced demands for sensing accuracy, precision, and resolution, alongside the existing communication criteria of latency, reliability, capacity, and coverage.[1]. Therefore, the joint communication and sensing(JCS) system that utilizes a signal to simultaneously achieve two functions have attracted much attention.

In previous systems, communication and radar sensing were separate systems using different frequencies and hardware resources. However, with increasingly scarce spectrum resources, there is a need for more efficient utilization of spectrum resources by communication and radar systems. As the bandwidth of commercial communication systems increases , coexistence with various existing radar systems is anticipated, leading to the development of JCS concept[2]. JCS can provide integrated and collaborative gains for future systems[3]. On one hand, sharing spectrum and hardware resources can lead to high resource utilization efficiency. on the other hand, sensing function can assist communication in obtaining more accurate channel estimation models, which beneficial for beamforming and Spectrum resource management.

3GPP Release 16 has established standards for vehicle sidelink communication based on the 5G-NR PC5 air interface, enabling vehicles to communicate directly without the assistance of gNB[4], as illustrated in Figure 1. Sidelink is beneficial for reducing latency and improving communication. In the meanwhile, sidelink signals also can be used for near-field positioning, range sensing, and distance measurement[5], thereby complementing or enhancing positioning systems that may be limited by obstacles or other factors, such as network-based positioning or Global Navigation Satellite System(GNSS). Therefore, the V2X sidelink JCS system has significant development potential.

However, due to limited available bandwidth and without the assistance of base station, there is a conflicting requirement between radar and communication in spectrum resource utilization. Radar accuracy requires large bandwidth occupancy, which reducing available resources in the resource pool, increasing resource collision probability, thereby affecting the performance of communication. The issue of resource collision in sidelink scenario is related to resource allocation scheme. Therefore, a flexible and robust resource allocation scheme is crucial for mitigating resource pool conflicts.

Traditional sidelink resource allocation scheme are divided into dynamic allocation and sensing-based semi-persistent scheduling(SB-SPS). SB-SPS is widely used for sidelink resource allocation due to its better reliability and latency[6]. It allows vehicles to autonomously choose and reserve resources for a given reservation period. However, potential collisions may remain undetected due to the lack of coordination from base stations, resulting in consecutive packet collisions and deteriorating sidelink JCS performance. Numerous studies have modified and improved the sensing and reselection process of SB-SPS[7][8], but have struggled to resolve the issue of consecutive collisions. With the advancements in self-interference (SI) technology in recent years[9], simultaneous transmission and reception on the same frequency band with in-band full-duplex (FD) transceivers have become feasible, offering hope for the implementation of JCS systems in the sidelink. Additionally, the powerful sensing capabilities of full-duplex bring collision detection functionality, creating new opportunities for enhancing sidelink resource allocation scheme.

Inspired by the above, this paper focuses on high-positioning accuracy, low-latency and high-reliability in 5G NR-V2X sidelink JCS system. By studying the comprehensive impact of interference due to consecutive collisions, we propose a reinforcement learning-based collision mitigation resource allocation scheme(CCM-SPS). Specifically, this scheme employs JCS full-duplex collision detection and reinforcement learning to optimize traditional SB-SPS parameters, mitigating performance degradation from consecutive collisions. Furthermore, the impact of varying vehicle density and packet sizes on JCS performance in dynamic vehicular networks is discussed. Finally, the effectiveness of the proposed scheme in enhancing overall performance is validated using a V2X sidelink system-level simulator.

The main contributions of this work can be summarized as follows:

• In order to address the conflicting requirement between sensing accuracy and communication reliability for sidelink resources, a novel collision mitigation resource allocation scheme is proposed. The algorithm integrates the full-duplex detection capability of JCS with the resource sensing reservation process of the traditional SB-SPS scheme. This allows vehicles to dynamically optimize reservation times based on sensing channel information, effectively reducing consecutive packet collisions and enhancing the overall utilization efficiency of resources in the sidelink JCS system.
• The performance of the proposed CCM-SPS and traditional SB-SPS is theoretically analyzed in scenarios of variety vehicle density and packet size. Through reinforcement learning, comprehensive optimization of resource utilization for sensing and communication is achieved.
• Comprehensive evaluations are performed using the Cramér-Rao lower bounds(CRLB), packet reception rate (PRR) and update delay (UD). The novel scheme shows comparative advantages in positioning accuracy, latency, and reliability performance indicators over comparative scheme.

The remaining article is organized as follows. In Section 2, a review of recent literature is conducted, in Section 3, theoretical analysis performance on sidelink JCS system. In Section 4, analyzed the consecutive collision problem using traditional resource allocation scheme. Then, in Section 5, the specific implementation of the improved resource allocation scheme was introduced, including full-duplex collision detection and Q-learning based collisions mitigation scheme. The extensive results are presented in Section 6, which analyzes the performance indicators in various scenarios. In Section 7 presents the concluding remarks.

2. Related Works

In high dynamic V2X scenario, JCS systems can make network service adjustments more flexible and robust by simultaneously handling communication and sensing signals and dynamically coordinate[10]. Extensive research has focused on the JCS in vehicular networks, primarily concentrating on signal waveform design[11][12][13] and interference and resource management[14][15][16]. Regarding signal waveform design, reference[11] proposes a frequency modulated continuous wave (FMCW) radar to address the interference of communication signals, thereby achieving both radar and communication capabilities. Reference[12] proposed an auto-paired super-resolution ranging and velocity estimation method based on OFDM waveforms, and derived the Cramér-Rao lower bounds(CRLB) for ranging and velocity estimation. Reference[13] advocates using a novel orthogonal time frequency space (OTFS) modulation multi-carrier waveform to address OFDM’s sensitivity to Doppler frequency shifts. The joint radar parameter estimation and communication system using OTFS modulation achieves precise radar estimation accuracy. Though this theory is currently at the research stage. Extensive research on the design of signal waveforms provide theoretical model foundation for the feasibility of JCS systems. Regarding interference and resource management, reference[14] presents the hardware prototype design of a vehicular JCS system based on OFDM waveforms and proposes an integrated waveform smart time and frequency resource filling (STFRF) algorithm for flexible time-frequency resource sharing and utilization. Field test results confirm that the proposed algorithm demonstrates effective communication and sensing performance in the millimeter wave frequency band. Reference[15] addresses the optimization problem of minimizing the transmission power of base stations in JCS systems by proposing a dynamic subcarrier and power allocation algorithm that improves network stability while reducing transmit power. Reference[16] proposes a power allocation design scheme for road side unit (RSU) in JCS to minimize the sum of CRLB for multiple vehicles, subject to the total rate constraint of multiple vehicles, and significantly reduces estimation errors compared to the classical water-filling algorithm while improving communication rates. Through the introduction of the above literature, it can be found that existing research on interference and resource management in JCS primarily focuses on large bandwidth millimeter-wave bands or Vehicle-to-Infrastructure (V2I) scenario, which have certain limitations. The optimization of JCS system resource allocation is a challenge for V2V direct communication with limited resources and no base station assistance.

In Release 16, the 3rd Generation Partnership Project (3GPP) developed the NR-V2X standard to complement LTE-V2X. It defines a new air interface PC5, which allows for direct vehicle-to-vehicle communication to support various advanced use cases, covering four areas: platooning, extended sensing, remote driving, and autonomous driving[17]. Like other wireless access technologies in vehicular networks, NR-V2X also relies on OFDM at the physical (PHY) layer. In recent years, research has increasingly explored the sensing capabilities of NR-V2X sidelink communication signals. Reference[18] utilizes unused communication subcarriers of the 5G NR waveform as radar sensing subcarriers, aiming to minimize the CRLB of distance estimation by optimizing the amplitude of radar subcarriers, thereby improving sensing performance. Reference [19] derives the CRLB for distance/angle estimation accuracy based on sidelink OFDM signals, demonstrating the feasibility of sidelink JCS signals for near-field localization. References [5][20] derive the CRLB and mean squared error (MSE) for sensing location of sidelink signals and analyze in detail the impact of communication physical layer parameters on sensing performance under interference conditions. Reference[21] compares the radar sensing performance of sidelink resource allocation SPS algorithms versus random allocation algorithms under interference, indicating that the JCS resource allocation mechanism is also a key factor affecting sidelink sensing performance.

According to the 3GPP standard[22], NR-V2X sidelink typically employs a distributed resource reservation mechanism, known as the SB-SPS algorithm[23]. However, SB-SPS still faces consecutive resource collision caused by resource allocation conflicts among multiple vehicles in complex and dynamic scenarios[24]. Recently, there have been many achievements in improving the SPS algorithm to enhance the communication performance of vehicle networking [25][26][27]. Reference[25] improves the efficiency of resource utilization, enhances reliability, and reduces latency by designing vehicle reuse distance during the SPS reselection process. Reference[26] develops an adaptive sidelink open-loop power control (AS-OLPC) algorithm to dynamically adjust transmission power, thereby improving communication reliability in complex urban environments. Reference[27] analyzes the effectiveness of introducing a re-evaluation mechanism at the MAC layer of NR-V2X to avoid collisions caused by packet re-transmissions, though the overall improvement is modest and requires more refined allocation strategies. Further research incorporates methods such as reinforcement learning[28][29][30]. Specifically, Reference[28] uses Q-learning to optimize SPS algorithm parameters, including reservation probability (RP) and reselection counter (RC), enhancing packet reception rate (PRR) and reducing update delay (UD) in high-dynamics vehicular networks. Reference [29] proposes a deep reinforcement learning-based congestion control mechanism that optimizes channel busy rate (CBR) and age-of-information (AoI), showing significant improvements over traditional decentralized congestion control algorithms (DCC). Reference[30] treats each vehicle as an independent agent and employs a multi-agent deep reinforcement learning (MARL) resource allocation algorithm, enabling vehicles to learn to select resource blocks and transmission power for periodic packet broadcasting. This approach effectively reduces resource collision probability and improves data transmission reliability.

Additionally, with advancements in self-interference cancellation (SIC) technology for in-band full-duplex communication[31], some studies have begun using full-duplex antennas to enhance resource allocation mechanisms. Reference [32] utilizes an in-band full-duplex transceiver and its collision detection capabilities to trigger SPS resource reselection, improving communication performance and analyzing the relationship between collision detection threshold and vehicle density variations. Reference [33] also leverages collision detection capabilities brought by full-duplex communication to adjust the probability of maintaining the same subchannel for transmission.

However, most of the current resource allocation algorithm improvements only focus on improving communication performance, and there are few literature on joint optimization of sidelink JCS systems. Indeed, in practical sidelink scenarios with limited frequency spectrum, the radar sensing accuracy can impact communication reliability due to the conflict on bandwidth requirements[21]. Due to the lack of base station coordination, traditional SPS scheme may lead to serious consecutive packet collisions. To our knowledge, there currently a lack of extensive research of joint optimization for sensing and communication in sidelink JCS systems. Therefore, this paper leverages pertinent research and proposes the CCM-SPS scheme to address the issue of consecutive collisions in sidelink JCS system. This scheme utilizes full-duplex collision detection and Q-learning reinforcement learning methods, optimizing both the sensing range (CRLB) and ultra reliable low latency communications(URLLC) quality in vehicular JCS systems.

3. Theoretical Performance Analysis on Sidelink JCS System

Multiple vehicles simultaneously transmit data packets via full-duplex broadcast and use echoes signal to detect and locate passive targets in the surrounding environment. Obtaining sensing information such as target distance, relative speed, and signal-to-noise ratio(SNR). In this scenario, multiple vehicles send sidelink signals, such as cooperative awareness message(CAM) or collective perception message(CPM)[34]. It is initially assumed that all vehicles are equipped with in-band full-duplex transceivers with perfect SIC, enabling the use of echoes to detect channel quality[35].

Assume that during time slot $t\in \mathcal{T}$, vehicle $i\in {\mathcal{I}}_t$ generates a data packet and begins broadcasting the OFDM symbols $s_m\left(t\right)$. In this context, the symbol for the n-th subcarrier of the m-th symbol is denoted as $x_{n,m}$, with each symbol power being $P_{n,m}$. The symbol duration is defined as $T_{sym}=T+T_{cp}$, where $T_{cp}$ represents the cyclic prefix duration and $T=1/\Delta f$ , where $\Delta f$ is subcarrier space. The representation in the complex baseband is as follows:

$$s_m\left(t\right)=\sum _{n=0}^{N-1}\sqrt{P_{n,m}}{x}_{n,m}{e}^{\mathrm{j}2\pi n\Delta ft}rect\left({\displaystyle \frac{t-m{T}_{\mathrm{sym}}}{T_{\mathrm{sym}}}}\right)$$

(1)

The sidelink JCS data packet consists of N subcarriers and M OFDM symbols. Assume that all the symbols $x_{n,m}$ in the packet form an $N\times M$ matrix $\mathbb{X}=\left\{x_{n,m}\right\}\in {\mathbb{C}}^{N\times M}$, where each column represents an OFDM symbol and each row represents a subcarrier. Assume that the power of each symbol on each subcarrier is the same and normalized, i.e., $P_{n,m}=P_{avg}$. Therefore, for all N and M, the transmission power of each vehicle is given by $P_T=N·P_{avg}$.

The vehicle transmits the sidelink data packet to other vehicles and uses the same signal for radar sensing. We assume that the target vehicle is located at a distance d from the transmitting vehicle and is moving with a relative speed v. The transmitting vehicle simultaneously receives reflections of its own signal using single-antenna radar. The complex baseband representation of the received echo signal for the n-th subcarrier of the m-th symbol is given by:

$$y_{m,n}\approx \alpha e^{j2\pi m{T}_{\mathrm{sym}}\nu }{e}^{-j2\pi n\Delta f\tau }{x}_{m,n}$$

(2)

Among them, $\alpha$ is the channel coefficient. By receiving the echoes at the transmitting receiver, estimates of $\nu$ and $\tau$ are obtained as $\widehat{\nu }$ and $\widehat{\tau }$. The parameters $\nu$ and $\tau$ represent the Doppler shift and delay of the echo signal, respectively. The distance estimate $\widehat{d}$ between the transmitting vehicle and the target vehicle is calculated from $\widehat{\tau }$ as:

$$\widehat{d}={\displaystyle \frac{\widehat{\tau }}{2c}}$$

(3)

where c represents the speed of light. The carrier frequency is $f_x$. The relative speed v of the target vehicle with respect to the transmitting vehicle is estimated as $\widehat{v}$ from $\widehat{\nu }$ using:

$$\widehat{v}={\displaystyle \frac{\widehat{\nu }c}{2f_x}}$$

(4)

$P_R$ is the power received by the full-duplex radio related to the intended target, given by:

$$P_R=P_T{\displaystyle \frac{G^2{c}^2\sigma }{{\left(4\pi \right)}^3{f}_{\mathrm{c}}^2{d}^4}}$$

(5)

Among them, $P_T$ represents the transmit power, and G represents the transmit/receive antenna gain. The noise power is represented as $P_n=k_B{T}_{0}FW$, where $k_B$ is the Boltzmann constant, $T_0=290$ is the reference temperature, F is the noise figure of the full-duplex radar receiver, and the bandwidth of the sidelink transmission signal is $W=N\Delta f$.

In practice, during the transmission slot, interference occurs when other vehicles transmit using the same frequency resources, which affects radar sensing performance. The calculation formula is:

$$I_{t,i}=\sum _{k\in {\mathcal{I}}_t,k\ne i}{\eta }_{t,ki}{\displaystyle \frac{h_{t,ki}{P}_{\mathrm{T}}}{L\left(d_{t,ki}\right)}}$$

(6)

In the (6), $L\left(d_{t,ki}\right)$ represents the path loss at distance $d_{t,ki}$ between the interfering vehicle k and the transmitting vehicle i at time slot t, and the parameter $h_{t,ki}$ represents the large-scale fading at time slot t. Additionally, ${\eta }_{t,ki}$ is a coefficient that takes values of 0 or 1, indicating the absence or presence of interference in that time slot, respectively.

Therefore, considering Gaussian white noise and interference from other vehicles, the signal-to-interference-plus-noise ratio (SINR) of the reflected echo signal from a target located at a distance d from the transmitter is calculated as follows:

$${\mathsf{SINR}}_{t,i}={\displaystyle \frac{P_R}{P_{\mathtt{n}}+I_{t,i}}}$$

(7)

According to[36][37], the radar sensing performance of NR-V2X signals with OFDM waveforms is represented by the Cramér-Rao Lower Bound (CRLB) on distance estimation variances. The CRLB represents the best performance that can be achieved for unbiased estimation of these parameters. The CRLB for distance estimation for the transmitting/sensing vehicle i at time slot t is given by:

$${\mathsf{CRLB}}_{t,i}^{\left(\widehat{d}\right)}={\displaystyle \frac{3c^2}{{\mathsf{SINR}}_{t,i}8{\pi }^2\Delta f^{2}MN(N^2-1)}}$$

(8)

The CRLB clearly describes the most optimistic performance achievable and serves as a benchmark for characterizing sensing accuracy. Practically, the signal processing methods can’t achieve performance below the theoretical CRLB.

Further, based on the preceding analysis, the impact of sidelink bandwidth resources on sensing performance can be analyzed. According to (8), the CRLB for distance estimation decreases rapidly with the cube of the bandwidth. Therefore, increasing the subcarrier spacing ($\Delta f$) or the number of subcarriers (N) is advantageous for distance estimation.

Simultaneously, according to Shannon’s theorem, increasing bandwidth benefits the data rate in vehicular networks. However, in a multi-user distributed resource allocation system with limited resources, indiscriminately increasing N to enhance sensing and communication performance may exacerbate consecutive packet collisions within the existing sidelink resource allocation strategy. This interference significantly impacts the performance of the JCS system and reduces resource utilization efficiency[21]. The subsequent sections will provide a detailed analysis on optimizing allocation scheme to mitigate collision.

4. Consecutive Collision Problem Analysis of the Sidelink Resource Allocation

4.1. Principle of the SB-SPS Resource Allocation Scheme

NR-V2X supports Mode 2 sidelink communication and employs the sensing-based semi-persistent scheduling (SB-SPS) scheme[38]. Initially, the SB-SPS scheme is designed to support periodic safety messages, utilizing sensing windows and resource reservation mechanism to reduce end-to-end latency. The fundamental working principle of SB-SPS is illustrated in Figure 2, with the specific steps as follows:

In the sensing window, vehicles measure the reference signal received power (RSRP) of physical resource block (PRB), continuously generating a list of available resources $L_a$. This list includes time-frequency resources with RSRP values below the threshold $P_{th}$. If the number of resources in $L_a$ is less than $X\%$ of the total resources, $P_{th}$ is incrementally increased by 3 dB, and the process is repeated until the number of available resources exceeds $X\%$ of the total resources, and then forming a candidate resource pool. The $X\%$ threshold can be set to 20%, 35%, or 50% depending on the configuration and service priorities. Subsequently, during the resource selection window, a reselection counter(RC) is employed to manage the use of reserved resources, with resource reselection occurring only when the RC reaches 0. This process involves selecting a new resource with probability $(1-P_k)$ or continuing to use the previously reserved resource with probability $P_k$, where $P_k$ ranges from 0 to 0.8. Once a resource is selected, continuous transmission occurs in the same resource block, with the number of transmissions determined by the value of RC. RC is randomly chosen within a range between 5 and 15, and decremented by 1 after each transmission until RC reaches 0, at which point the next selection process is triggered. (9) demonstrates the $\Delta RC$ which is the RC decreasing step size of the SB-SPS:

$$\Delta RC=R{C}^{\prime }-RC=1$$

(9)

Due to the distributed resource allocation characteristics of sidelink, it is impossible to obtain complete channel state information, which means that it cannot be guaranteed that all vehicles select idle resources. In addition, there is partial overlap in the candidate resource pools between neighboring vehicles, resulting in packet collisions. This issue is exacerbated in dynamic scenarios with high density and large packet size services. Furthermore, the sidelink echo signals are also subject to interference from consecutive packet collisions, which degrades the sensing performance.

4.2. Markov Chain Model of SB-SPS

In this section, a Markov chain analytical model[39] is presented, as shown in Figure 3. At any slot t, the corresponding $RC\left(t\right)\in [0,15]$. If $RC\left(t\right)=0$, it is randomly re-initialized s.t. $RC(t+1)\in [5,15]$. Thus the probability that

$$Pr\{RC(t+1)=i|RC\left(t\right)=0\}={\displaystyle \frac{1}{11}}.$$

(10)

Denote ${\pi }_i$ as the probability that $RC\left(t\right)=i,0\le i\le 15$. According to Figure 3, ${\pi }_i$ satisfies:

$${\pi }_i=\left\{\begin{array}{cc}{\pi }_0\hfill & \mathrm{for}0\le i\le 4,\hfill \\ {\displaystyle \frac{1}{11}}{\pi }_0+{\pi }_{i+1}\hfill & \mathrm{for}5\le i\le 14,\hfill \\ {\displaystyle \frac{1}{11}}{\pi }_0\hfill & \mathrm{for}i=15.\hfill \end{array}\right.$$

(11)

Using the normalization condition ${\sum }_{i=0}^{15}{\pi }_i=1$, and solving (10), we obtain ${\pi }_0={\displaystyle \frac{1}{11}}$.

Since access collisions are caused by resource reselection, we first define the probability of a reselection resource collision. A collision will occurs when multiple vehicles reselect the same resources within overlapping selection windows. Assume that at time t, vehicle UE0 is in the state $\mathrm{RC}=0$, and it performs a reselection during the selection window $[t,t+\mathrm{RRI}]$. During this time, other UEs also engage in reselection. UEs transition to $\mathrm{RC}=0$ with probability ${\pi }_0$ and reselect with probability $1-P_k$, moving to a new state. Since the RC states of each UE are independent, if n out of $N_{\mathrm{UE}}$ UEs are involved in reselection, the probability that other vehicles also trigger reselection when UE0 triggers reselection is given by:

$$\begin{array}{cc}& P_s\left(n\right)=Pr\{nR{C}_{UEs}=0,\mathrm{Reselect}|R{C}_{UE0}=0,\mathrm{Reselect}\}\hfill \\ & =\left(\begin{array}{c}{N}_{UE}\\ n\end{array}\right){\left({\pi }_0(1-P_k)\right)}^n{\left(1-{\pi }_0(1-P_k)\right)}^{N_{UE}-n}\hfill \end{array}$$

(12)

where n represents the number of vehicles simultaneously reselecting within UE0’s reselection window. Access collision may occur if at least one other vehicle selects the same available PRB as UE0.

We define the collision involving n UEs reselecting within the overlapping selection window as an n-fold collision, given by the following formula:

$$\begin{array}{cc}& P_r\left(n\right)=Pr\left\{n-\mathrm{fold}\mathrm{Collision}\right|nR{C}_{UEs}=0,\mathrm{Reselect}\}\hfill \\ & =1-{\left(1-{\displaystyle \frac{1}{\overline{N_a}}}\right)}^n\hfill \end{array}$$

(13)

where $\overline{N_a}$ represents the average number of available PRBs within the selection window. This number is influenced by the vehicle density and the packet size. higher vehicle density and larger packet sizes result in fewer available resources.

Thus, when UE0 performs reselection within the selection window, the access collision probability can be obtained as follows:

$$\begin{array}{cc}& P_c=Pr\left\{\mathrm{Collision}\right|R{C}_{UE0}=0,\mathrm{Reselect}\}\hfill \\ & =\sum _{n=1}^{N_{UE}}{P}_r\left(n\right)P_s\left(n\right)\hfill \end{array}$$

(14)

In SB-SPS, the same resources would be continuously collision after an access collision. The collision will last for at least $\min [R{C}_{\mathrm{UE}0},R{C}_{\mathrm{UE}1}]$ times between two UE0 and UE1. Therefore, it is useful to adjust $\Delta RC$ that is the RC decreasing step size to mitigate consecutive collisions. By increasing the $\Delta RC$ can reduce consecutive collisions. However, an overly aggressive increase in $\Delta RC$ will increase the number of vehicles entering selection window simultaneously, thereby increasing the probability of access collision, according to (14). So, it is necessary to optimize the $\Delta RC$.

5. Q-learning based CCM-SPS Resource Alloaction Scheme proposed for JCS Sidelink

Our proposed CCM-SPS scheme to improve the sensing and resource reservation process. Details will be elaborated below.

5.1. Collision Detection Mechanism

The full-duplex (FD) transceiver calculates the echo power as a condition for collision detection. The total received power of the FD receiver can be expressed as:

$$P_r^{FD}=P_R+P_n+I_{t,i}$$

(15)

Where $P_R$ represents the power of reflected echo signal. Assuming only the closest to the transmission vehicle is considered. Since the CAM signal contains distance informatio. $P_R$can be calculated through path loss using the distance of CAM messages. $P_n$ is the Gaussian white noise power. $I_{t,i}$ represents the interference of other transmitting vehicles to VUE i in the same time slot t according to (6).

By calculating the difference between the received power $P_r^{FD}$ and the echo power $P_R$ from the nearest vehicle, while also accounting for the noise power $P_n$, the interference power strength $I_{t,i}$ can be effectively estimated.

Then the detection rules for collisions in the JCS sidelink are as given follows:

$$\left\{\begin{array}{cc}{P}_r^{FD}-P_R-P_n>P_{\Delta },\hfill & COL=1\hfill \\ P_r^{FD}-P_R-P_n<P_{\Delta },\hfill & COL=0\hfill \end{array}\right.$$

(16)

The threshold is used to determine whether the interference impact exceeds the critical power. If $COL=0$, indicates no resource collision; if $COL=1$ indicates a resource collision.

5.2. Q-learning based CCM-SPS Scheme for JCS Sidelink

After detecting resource collision, a consecutive collision elimination mechanism based on Q-learning is proposed to mitigate consecutive resource collisions. Vehicles interact with the environment in real time and intelligently determine the optimal actions given the current state.

5.2.1. Vehicular Agent based on the Reinforce Learning Model

In a typical reinforcement learning framework[40], as illustrated in Figure 4, the agent achieves its learning objectives through an iterative process of receiving rewards from the environment. This process allows the agent to learn and improve its policy based on the rewards it receives from its interactions with the environment.

In the proposed sidelink JCS system, each vehicles updates its current state after transmitting data packets and detecting collisions. The current state includes the collision status and the current RC value. For vehicle i, the state $s_i\in S$ can be described as follows:

$$s_i=(CO{L}_i,R{C}_i)$$

(17)

Where ${\mathrm{COL}}_i$ indicates whether the vehicle i experiences a resource collision when transmitting a data packet. ${\mathrm{RC}}_i$ represents the current value of the reselection counter, with a range of [0, 15]. Therefore, the state set consists of a discrete state space.

The traditional SB-SPS scheme reduces the RC by 1 after each packet transmission. To regulate the decrease efficiency of RC, we propose a set of a discrete action space that represent the selection of actions based on acquired state information. This enables resource collision vehicles to adjust RC decreasing step size($\Delta RC$).

$$A=\{a_i\in \Delta RC\}$$

(18)

Where $\Delta RC$ represents the value of the RC that the vehicle i to reduce after each transmitting. The vehicle will dynamically select an action in the action space of [0,15] based on the observed state. When the $RC-\Delta RC<0$, the vehicle enters the reselection process and reselects a new resource.

After a vehicle executes an action, its state will transition, and it will learn the instantaneous reward R from the environment.

$$R=\left\{\begin{array}{cc}(exp(-N_{col})-1)/lo{g}_{10}(RC+1),\hfill & \mathrm{COL}=1\hfill \\ 1,\hfill & \mathrm{COL}=0\hfill \end{array}\right.$$

(19)

Where $N_{col}$ represents the times of a consecutive resource collision for the vehicle. When the collision does not occur, the reward is 1. When a collision occurs, the greater $N_{col}$ is, the greater the penalty is, ensuring that the collision state can be removed in as few times as possible. In addition, the current RC value is introduced as a constraint. The smaller the RC, the greater the penalty. When different RC vehicles collision in the same resources, the probability of the vehicle with small RC entering the reselection stage increases, while the vehicle with large RC can continue to transmit using the current resources. This mechanism can effectively reduce the number of vehicles in the reselection stage simultaneously, thus helping to maintain the stability of the system.

In Q-learning, Q value $Q(s,a)$ was calculated and updated using reward model to evaluate the state-action mapping policy under the state action pair $(s,a)$. The Q value is updated by the Bellman equation, as shown below:

$$\begin{array}{cc}\hfill Q(s,a)& \leftarrow Q(s,a)+\alpha [R\hfill \\ & +\eta \underset{s\in S}{max}Q(s^{\prime },a)-Q(s,a)]\hfill \end{array}$$

(20)

When vehicles optimize the search for the optimal action, they need to balance the exploitation and exploration of learned knowledge to ensure that each action has a possibility of being selected. This paper adopts an $ϵ$-greedy strategy to balance the exploitation-exploration process. $ϵ$ starts at 1 and decreases to 0 as the training time increases.

$$a=\left\{\begin{array}{cc}arg\underset{a\in A}{max}Q(s,a),\hfill & \mathrm{with}\mathrm{prob}.1-\epsilon \hfill \\ \mathrm{random},\hfill & \mathrm{with}\mathrm{prob}.\epsilon \hfill \end{array}\right.$$

(21)

5.2.2. Algorithm Flow and Pseudo-code

Based on previous analysis, we proposed CCM-SPS scheme to reduce the times of consecutive collisions. Specifically, during transmission, the full-duplex (FD) echo detection capability is used to sense the channel state. If a collision is detected, an $\Delta RC$ is selected based on the current state. This means that when a collision is detected, the scheme decrease the times of consecutive collision and restricts the number of vehicles entering the reselection process simultaneously, as shown in Figure 5.

The pseudocode of its algorithm is shown below:

Algorithm 1 Pseudo-code of the proposed CCM-SPS

Input: Vehicle density,Packets occupy bandwidth Output: $Qtable$ 1: Initialize the parameters such as learning rate $\alpha$ and discount factor $\eta$.   2: Initialize the vehicles’ states, actions, and $Qtable$   3: loop   4:    Begin the new packet transmission   5:    Observe transmitted vehicle $s_i$    6:    if collision occurred, $COL=1$ then    7:      the number of consecutive resource collisions      $N_{col}=N_{col}+1$   8:      Vehicle i obtain the negative reward:      $(exp(-N_{col})-1)/log(RC+1)$ 9:    else   10:      the number of consecutive resource collisions      $N_{col}=0$ 11:      Vehicle i obtain the positive reward 1   12:    end if   13:    Vehicle update the Q(s,a)   14:    Vehicle update the probability $ϵ$ according to simulation time   15:    if exploration then   16:      Vehicle randomly select an action $\alpha$   17:    else   18:      Vehicle select the optimal action ${\alpha }_{opt}$ with $Q_{max}$   19:    end if 20: end loop

6. JCS Sidelink Performance Evaluation using CCM-SPS

In this section, key performance metrics such as CRLB, PRR, and UD were evaluated under varying vehicle density and packet size scenarios. And comprehensive experimental discussions were conducted.

6.1. Simulation Setup

The main settings are reported in Table 1 and discussed hereafter.

The performance of NR-V2X radar sensing is characterized by using a system level simulator, which is an event driven simulator for NR-V2X sidelink network. In particular, vehicles perform SINR evaluation for collision detection and CRLB calculation after each transmission.

Table 1. SIMULATION PARAMETERS.

Parameter	Symbol	Value
Scenario
Road layout	- -	Highway,3+3 lanes
Density	- -	50,150,250 vehicles/km
Average speed	- -	70 km/h
STD of vehicle speed	- -	7 km/h
Target RCS	$\sigma$	10 dBsm
Power and propagation
Channel model(interference)	- -	WINNER+, B1
Available channel bandwidth	$W_{ch}$	40 MHz
Transmitted power	$P_T$	23 dBm
Antenna gain	G	3 dBm
Noise figure	F	6 dB
Center frequency	$f_c$	5.9 GHz
Shadowing	- -	Variance 3 dB,decorr.dist. 25 m
Physical layer
SCS	$\Delta f$	15 kHz
MCS	- -	5(QPSK,$R_c=0.3$)
Sbuchannel size	- -	10 PRBs
Access layer
Keep probability	$P_{keep}$	0.8
Initial reselection counter	$RC$	$[5,15]$
RSRP sensing threshold	- -	-126 dBm
Data traffic
Packet generation interval	- -	100 ms
Packet size	- -	350,1000 bytes

Table 1. SIMULATION PARAMETERS.

Parameter	Symbol	Value
Scenario
Road layout	- -	Highway,3+3 lanes
Density	- -	50,150,250 vehicles/km
Average speed	- -	70 km/h
STD of vehicle speed	- -	7 km/h
Target RCS	$\sigma$	10 dBsm
Power and propagation
Channel model(interference)	- -	WINNER+, B1
Available channel bandwidth	$W_{ch}$	40 MHz
Transmitted power	$P_T$	23 dBm
Antenna gain	G	3 dBm
Noise figure	F	6 dB
Center frequency	$f_c$	5.9 GHz
Shadowing	- -	Variance 3 dB,decorr.dist. 25 m
Physical layer
SCS	$\Delta f$	15 kHz
MCS	- -	5(QPSK,$R_c=0.3$)
Sbuchannel size	- -	10 PRBs
Access layer
Keep probability	$P_{keep}$	0.8
Initial reselection counter	$RC$	$[5,15]$
RSRP sensing threshold	- -	-126 dBm
Data traffic
Packet generation interval	- -	100 ms
Packet size	- -	350,1000 bytes

Scenario. We consider a highway scenario with 3 lanes per direction and a variable number of vehicles per kilometer (variable vehicle density). Each vehicle, randomly-deployed at the beginning of the simulation, moves with average speed of 70 km/h and a standard deviation of 7 km/h. The radar cross-section of the vehicle is 10dBsm.

Power settings and channel model. Vehicles are equipped with TX/RX antennas with a gain of 3 dBi, transmitting a fixed power ($P_T$ = 23 dBm). A receiver with a noise figure (F) of 6 dB is considered. The channel between vehicles, which determines the interference level through path loss characterization, follows the WINNER+ scenario B1. In this scenario, the channel experiences correlated lognormally distributed shadowing, with a standard deviation of 3 dB and a decorrelation distance of 25 m, as suggested in [41] and commonly adopted. The path loss related to the backscattered channel between the sensing vehicle and the radar target according to (5), as we are focusing on detecting the closest target in a line-of-sight (LOS) highway scenario. We assume ideal full-duplex radios. A carrier frequency $f_c$ = 5.9 GHz is considered with fixed channel bandwidth $W_{c}h$ to accommodate sidelink signals.

Physical layer and Data traffic. We assume all the users in the scenario adopting the same setup in terms of MCS, SCS, and packet size. Packets are generated every 100 ms; two packet sizes are considered, 350 and 1000 bytes, to simulate typical packets that are used in V2X communication services, e.g. for CAM and CPM. The packet size, under fixed MCS and SCS configurations, will affect the bandwidth occupied by vehicle transmission packets, as shown in Table 2.

Table 2. RESULTING NUMBER OF SUBCARRIERS, TRANSMISSION BANDWIDTH AND TTI FOR THE CONFIGURATIONS CONSIDERED IN SIMULATIONS.

Packet	SCS	MCS	${\mathit{N}}_{\mathit{P}\mathit{R}\mathit{B}}^{\mathit{m}\mathit{a}\mathit{x}}$	${\mathit{N}}_{\mathit{s}\mathit{u}\mathit{b}}$	${\mathit{N}}_{\mathit{P}\mathit{R}\mathit{B}}$	W[MHz]
350	15	5	216	4	40	7.2
1000	15	5	216	10	100	18

Table 2. RESULTING NUMBER OF SUBCARRIERS, TRANSMISSION BANDWIDTH AND TTI FOR THE CONFIGURATIONS CONSIDERED IN SIMULATIONS.

Packet	SCS	MCS	${\mathit{N}}_{\mathit{P}\mathit{R}\mathit{B}}^{\mathit{m}\mathit{a}\mathit{x}}$	${\mathit{N}}_{\mathit{s}\mathit{u}\mathit{b}}$	${\mathit{N}}_{\mathit{P}\mathit{R}\mathit{B}}$	W[MHz]
350	15	5	216	4	40	7.2
1000	15	5	216	10	100	18

6.2. JCS Sidelike Performances with Dynamic Vehcile Density

Firstly, the JCS performances of the proposed CCM-SPS has been evaluated, compared with a benchmark scheme in case of various vehicle densities. The benchmark scheme, FD-enhanced as proposed in [32]. It detects resource collisions using full-duplex during transmitting, and then uses the aggressive resource reselection by setting RC to 0 for all vehicles that access collision occurs. It can quickly break off consecutive collisions to somehow improve communication performance. The propose CCM-SPS shceme employ Q-learning to optimize resource reselection process in addition to FD detection in resource reservation process.

Figure 6 illustrates the empirical CDF of the root CRLB for range using SB-SPS, FD-enhanced and CCM-SPS in various density scenarios. As the vehicle density increases, the sensing performance of root CRLB range decreases. And the FD-enhanced significantly improves performance at medium and low vehicle densities, but the improvement is not as obvious at high vehicle densities. However, the proposed CCM-SPS not only further enhances performance in medium to low density but also effectively improves performance in high-density scenarios. Figure 7 presents 95% range root CRLB for different algorithms at varying densities. The bar graph provides a more visual representation showing the superior performance of the proposed CCM-SPS across all density scenarios compared to the other two schemes.

Similar results also appear in discussions on communication performance. Figure 8 illustrates PRR over distance using SB-SPS, FD-enhanced and CCM-SPS in different density scenarios. As the vehicle density increases, the PRR of different schemes all show a downward trend. In mid and low density scenarios, the FD-enhanced scheme shows some improvements in PRR within a 200m communication range. However, in high-density scenarios, the improvements are not significant. By examining the maximum distance where PRR exceeds 0.95 under three different scheme at varying densities, as shown in Figure 9. the advantages of the CCM-SPS scheme become more obviously. Compared to the FD-enhanced scheme, the proposed algorithm can further enhance communication metrics effectively across different density scenarios, especially in high vehicle density. The maximum communication ranges in low, mid, and high density scenarios are increased by 22.2%, 30%, and 50%, respectively. This analysis indicates that the CCM-SPS scheme demonstrates robustness and effectiveness in various traffic density scenarios.

In short, as vehicle density rises, available resources in the pool decrease, increasing the probability of access collisions and resulting in consecutive resource collisions. It leading to JCS performance degradation. The FD-enhanced employs full-duplex to detect collisions and to achieve early termination of consecutive collisions. But an overly aggressive reselection mechanism leads to an excessive number of vehicles entering the reselection stage in high-density scenarios. According to (13), an increase in the number of vehicles in the reselection process within the same time slot raises the probability of new access collisions. The advantages of the FD-enhanced scheme over traditional SB-SPS are more pronounced at mid and low densities, but diminish at high densities. However, the CCM-SPS scheme outperforms FD-enhanced in a variety of density scenarios. It can be demonstrated that the Q-learning based scheme significantly enhances performance, particularly in high-density environments. This is due to the reward function defined in the proposed Q-learning algorithm, vehicles that not occur resource collisions will receive rewards and strive to maintain the current state as much as possible, ensuring the stability of the resource pool. Vehicles that experienced resource collisions are penalized based on the current RC and the times of consecutive collisions. That encouraging vehicles to learn the best strategy to minimize consecutive collisions, and avoid new collisions.

6.3. JCS Sidelike Performances with Dynamic Packet Sizes

6.3.1. Conflicting Impacts of Packet Sizes on JCS Performance Metrics

Secondly, in this section, we investigate the conflicting impacts of packet sizes on JCS performance metrics in terms of root CRLB range and PRR in different vehicle density scenarios by using traditional SB-SPS scheme. Then we evaluate optimization performance results of Q-learning based CCM-SPS scheme, including sensing and communication metrics in different vehicle density scenarios.

Figure 10 illustrates the empirical CDF of root CRLB range estimation with different packet sizes and vehicle densities. With given pack size, the empirical CDF of root CRLB Range decrease rapidly with increase of vehicle density. As the packet size increases, the CDF curve of root CRLB Range shift towards left with a small value of CRLB. This results indicate the enhanced sensing accuracy performance.

Figure 11 illustrates the communication reliability metric PRR vs. distance under different packet sizes and vehicle densities. With a given packet size, the communication reliability PRR decrease with the transmission distance. As the packet size or the vehicle density increases, the PRR curves decline fast.

Based on above discussion results of both Figure 10 and Figure 11 of JCS system with the traditional SB-SPS scheme, it can be seen that with increasing vehicle density, both sensing and communication performances decreases. This is because the worse channel quality of the sidelink in a high-density vehicle network yields more consecutive collisions over the shared resource pool with traditional SB-SPS. Moreover, it also can be seen that with increasing packet size, the sensing accuracy increases with smaller root CRLB range value while the communication reliability declines with smaller PRR value. In case of big packet size, the require bandwidth for transmission increase by allocate more subcarrier. The echo radio with a large subcarrier number N can give a small the range CRLB value, according to (8). At the same time, there is a high probability that different vehicles occupy the same spectrum resource, resulting in the serious consecutive collisions due to the resource competition over the sidelink, which would deny a successful access and give a small PRR.

In short, the traditional sidelink resource allocation scheme is less effective for sidelink JCS system to support various services in highly dense dynamic vehicular networks. In order to overcome the conflicting requirement on the spectrum resource allocation between sensing and communication, this paper has proposed a novel CCM-SPS scheme to realize an effective sidelink resource allocation for enhancing JCS performances. Next, in the following section, performance JCS will be evaluated.

6.3.2. optimization performance evaluation of Q-learning based CCM-SPS

The proposed CCM-SPS can optimize JCS performance metrics by introducing Q-learning method, in order to control the repetitions times of the reserved resources through adjusting RC decreasing step size and suppress the consecutive collisions probability.

Figure 12 illustrate the CCM-SPS’s range sensing performance evaluation on empirical CDF of root CRLB with different pack sizes in case of density = 50 150 250 veh/km. The performance traditional SB-SPS with the same configuration are also given as a comparison. As for the CDF curve of root CRLB Range, CCM-SPS’s performance curve increases faster than traditional SB-SPS at a given packet size and vehicle density, giving a high probability of small CRLB value. Meanwhile, with the increase of packet size, the large packet size, the more CRLB. As the vehicle density increase, CCM-SPS has the advantage of smaller CRLB value than SB-SPS can be maintained.

On the other hand, Figure 13 shows PRR vs. distance performance of CCM-SPS with different pack sizes in different density. Figure 14 indicates CCM-SPS’s communication performance evaluation on empirical CDF of updata delay with different pack sizes in different density. The performance traditional SB-SPS with the same configuration are also given as a comparison. From both Figure 13 and Figure 14 communication performances result.

It can been seen that, as a given vehicle density, CCM-SPS can obtain a higher PRR than SB-SPS. With the packet size increase, PRR of CCM-SPS decreases slower than SB-SPS. As the vehicle density becomes large, CCM-SPS can hold a relative high PRR and SB-SPS lose PRR even in the low density scenario in contrast. Similarly, with a given vehicle density, CCM-SPS can achieve obviously deduction on update delay compared with SB-SPS. As vehicle density become dense, update delay increase of both scheme, but CCM-SPS can keep high probability of small update delay value. Therefore, the proposed CCM-SPS can achieve the optimal communicate quality such as high reliability and low latency better than the traditional scheme.

In view of the above discussion results, it is significant that CCM-SPS can fulfill comprehensive performance enhancements on both sensing and communication without cost of each other. CCM-SPS makes resource reservation feasible for sidelink JCS access after FD detecting the dynamic SINR from the echo signal. According to reward function (19) related to RC and $N_{col}$, the JCS vehicle agent can learn from the dynamic network environment and feedback a corresponding reward in order to optimize $\Delta RC$ selection action in the reservation process through Q-learning approach. As a result, the effective resource allocation simultaneously for both sensing and communication can be realized by using CCM-SPS. So it is promising for CCM-SPS to be used in JCS systems to realize accurate sensing, high reliability and low latency applications.

7. Conclusions

This paper proposes a resource allocation scheme in sidelink JCS system, named consecutive collision mitigation semi-persistent scheduling (CCM-SPS). By employing the collision detection referring to the echo power threshold and Q-learning to train the RC decreasing step size, this scheme can effectively suppress the consecutive collision probability. Compared with traditional SB-SPS and FD-enhanced scheme, CCM-SPS can achieve superior both sensing and communication performance even in the high-density vehicle scenarios. Furthermore, CCM-SPS can support services with large packet sizes to achieve accurate sensing with less cost of communication reliability with the increasing of distance. It is particularly meaningful of CCM-SPS in the perspective of enabling sidelinks to support sensing and communication collaboration in 6G network. In future work, there are interesting studies such as practical full-duplex impacts from interference and cross-layer optimization.