Deep Reinforcement Learning for Optimizing Restricted Access Window in IEEE 802.11ah MAC Layer

1. Introduction

With the rapid development of Internet of Things (IoT) applications and technologies, IoT has emerged as a pivotal enabler bridging the physical and digital realms. IoT has been widely used in industry, agriculture, healthcare and other fields. Statistics show that IoT connected devices are expected to exceed 30 billion units by 2025, more than doubling from 13.8 billion in 2021 [1]. With expanding scope of applications, IoT has its own set of requirements: very low power, longer range connections, and support for a greater number of client devices per access point (AP) [2]. The fulfillment of these requirements relies on the selection of wireless communication technologies.

To meet the key requirements of IoT applications, the Wi-Fi Alliance has introduced Wi-Fi HaLow technology [3]. The Wi-Fi HaLow technology is based on the IEEE 802.11ah standard [4], operating in the unlicensed sub-1 GHz radio frequency spectrum band and utilizing narrower channels. IEEE 802.11ah is built upon the IEEE 802.11 standards with modifications for IoT applications. The physical (PHY) layer of IEEE 802.11ah is designed for long-range communication. At the medium access control (MAC) layer, novel channel access control mechanisms are introduced to facilitate access for a large number (up to 8191) of stations (STAs) and support low power consumption. Leveraging the novel features at the PHY and MAC layers, IEEE 802.11ah offers up to 100 times longer range compared to other IoT technologies, with a data rate ranging from approximately 150 kbps to a maximum of around 86.7 Mbps [4]. As shown in Figure 1, the IEEE 802.11ah-based Wi-Fi HaLow technology provides a well-balanced combination of data rate, coverage range, and energy efficiency, outperforming low-power IoT technologies such as LoRa, NB-IoT, and Zigbee [3]. Wi-Fi HaLow also features easier deployment and integration into IP networks compared to other technologies, with scalability similar to LoRa. Therefore, Wi-Fi HaLow is well-suited to meet the key requirements of IoT applications.

At the MAC layer, the restricted access window (RAW) mechanism is introduced to manage the significant number of STAs accessing the network [4]. The idea of RAW is to divide the channel time into one or more access windows, where only some of the STAs can access the channel in the designated access windows, while the others are restricted from random access. As shown in Figure 2, for STAs with certain traffic patterns, the AP divides them into one or more RAW groups during a traffic indication map (TIM) beacon interval. On the arrival of each RAW, STAs assigned to the current RAW have the right to access the channel for data transmission, while the other STAs remain dormant and cache non-urgent data until the arrival of their corresponding RAW. To further alleviate contention, each RAW is subdivided into multiple time slots with equal duration. STAs are uniformly distributed among these slots by default. During each slot, only STAs assigned to the current slot are permitted to contend for data transmission, ensuring that STAs restricted in different slots do not conflict with each other.

The operation of RAW mainly consists of two parts. One is the division of STAs into different RAW groups. The other is the configuration of RAW-related parameters, including the number of RAW groups, the number of slots in each RAW, the duration of each RAW, and the number of STAs in each RAW group. However, details about RAW parameters setting and RAW grouping are not specified in the IEEE 802.11ah standard. This allows researchers the flexibility to customize the RAW configuration to meet the specific requirements of different application scenarios. Moreover, the performance of RAW can be validated in the NS-3 simulator. In [5], the authors constructed simulation environments for IEEE 802.11ah sensor networks in the NS-3 simulator that closely resembles real-world network conditions. Through simulations, the detailed analysis of the impacts of RAW parameters (i.e., number of RAW groups, RAW group duration, and station division) on network throughput, transmission delay, and energy efficiency have been conducted in the literature. The experiments in [6] also revealed that network performance largely depends on the RAW parameters setting.

Based on this observation, some studies have focused on finding the optimal RAW parameters to improve network performance. Researchers have conducted complicated mathematical models and proposed heuristic methods to determine the optimal RAW parameters or grouping scheme [7,8,9]. However, most of the analytical models fail to consider the complexities and dynamic changes of network conditions, leading to discrepancies between the results derived by analytical models and those obtained from the NS-3 simulator. Moreover, heuristic methods for optimizing RAW parameters are often constrained by specific assumptions, such as fixed network topologies and known traffic patterns. The applicability of these methods in various scenarios requires further validation. Therefore, this paper aims to propose a flexible model-free learning method for finding the optimal RAW parameters, which is scalable, robust, light-weight, and capable of generalizing across different scenarios.

Due to the high efficiency and strong generalization capabilities, artificial intelligence (AI) methods have found broad applications in wireless networks in recent years. There is a growing number of studies employing AI methods to solve RAW parameters optimization and grouping problems. Researchers in [10] used neural networks to decide the optimal number of RAW groups and the number of slots in each RAW for given network conditions. Moreover, machine learning (ML) methods such as K-means have been used to solve grouping problems [11]. It is noteworthy that deep reinforcement learning (DRL) integrates deep learning (DL) and reinforcement learning (RL) by using deep neural networks (DNNs) to approximate value functions or optimal policies, thereby enabling the handling of high-dimensional and complex state and action spaces. DRL’s strong performance in dealing with complex and dynamic environments endows it with powerful generalization capability, making it widely applied in wireless networks for solving parameterized optimization problems such as resource allocation and scheduling [12,13]. Therefore, it is feasible to employ the DRL approach to solve RAW parameters and network performance optimization problems.

In this paper, we present a DRL method based on the proximal policy optimization (PPO) algorithm for optimizing RAW parameters in the IEEE 802.11ah-based IoT network. To the best of our knowledge, there are limited prior studies on RAW mechanism optimization using DRL-based approaches. Our study primarily focus on designing an effective DRL algorithm to find the optimal RAW parameters to enhance network throughput, and validating the feasibility of the optimized RAW parameters configuration in the NS-3 simulator. Our study can serve as a reference for applying DRL to the RAW mechanism and further extensions for optimizing other mechanisms of IEEE 802.11ah. We summarize our contributions as follows:

• Elaboration of the RAW mechanism and RAW-based channel access in the IEEE 802.11ah-based IoT network are presented to provide a more comprehensive understanding of RAW. Furthermore, mathematical analysis is presented to show significant impact of RAW parameters on network performance.
• A PPO-based DRL algorithm is designed to find the optimal RAW parameters to improve network throughput, which interacts with the network environments in the NS-3 simulator.
• Network environments with periodic and random traffic are considered in the NS-3 simulator to validate the feasibility and generalization capability of the PPO-based DRL algorithm, and the performance of the proposed algorithm is evaluated through NS-3.

The remainder of this paper is organized as follows. Prior studies on RAW-based network performance optimization and related works using AI-based methods for RAW mechanism are presented in Section 2. In Section 3, the network model considered in this paper and the operation of RAW are elaborated, throughput modeling with respect to RAW parameters is presented, and the RAW parameters optimization problem is established. The problem is reformulated as a Markov decision process (MDP), and a PPO-baed DRL algorithm for RAW parameters optimization is proposed in Section 4. In Section 5, the performance of the proposed DRL algorithm is evaluated in simulation environments built in the NS-3 simulator. Conclusions are drawn, and future studies are discussed in Section 6.

2. Related Work

2.1. Analytical Modeling for RAW Mechanism

To investigate the impact of the RAW mechanism on network performance, researchers have developed several evaluation models for the RAW-based channel access process. Typically, researchers introduce characteristics of RAW into the analytical model of the distributed coordination function (DCF) of IEEE 802.11 standards [14]. Given the known number of STAs in the network, researchers analyzed the transmission and collision probabilities in a single RAW slot. The analysis is then extended to one or multiple RAWs to derive formulas for calculating network performance metrics such as throughput, delay, and energy consumption [7,15,16]. These analytical models are validated by comparing results with those obtained from the NS-3 simulator. However, they require a series of assumptions, including saturated network traffic, ideal channel conditions, and packet loss solely caused by collisions. To obtain more accurate models, researchers further took unsaturated traffic, heterogeneous networks, signal capture, and other network conditions into account [6,17,18].

Moreover, researches investigated the impact of different RAW parameters on the network performance based on analytical models or simulation results. Authors in [19] pointed out that dividing more RAWs in a beacon interval period can reduce collision probability as the total number of competing STAs in each RAW group decreases. However, this also leads to increased delay, as larger RAW segmentations increase the probability of packet buffering. Similarly, authors in [18] stated that the more slots divided in each RAW, the fewer number of STAs competing for channel access in a single slot, thereby reducing the probability of collisions. However, the time overhead increases due to the non-cross-slot-boundary setting. Authors in [5] emphasized that a longer RAW duration generally results in better throughput. However, excessively long RAW durations perform worse in terms of latency. Moreover, the duration of a RAW should be determined based on the traffic load in each RAW group. The critical impact of RAW duration on the network performance was further discussed in [6].

2.2. Optimization in RAW Mechanism

Given the critical impact of RAW parameters on network performance, an important issue of optimizing RAW mechanism is the optimization of RAW parameters. RAW parameters include the number of RAW groups, number of slots in each RAW, duration of each RAW (which can be calculated given the slot count and slot duration), and the number of STAs in each RAW group (which can be calculated given the number of RAW groups and STAs in the network). It has been validated that the optimization of RAW parameters depends on various network variables, such as number of STAs, traffic load, and traffic patterns [5]. Most of the studies are network performance optimization-oriented, in which authors formulate RAW parameters optimization problems and obtain one or more optimal RAW parameters using various optimization methods. To jointly maximize uplink energy efficiency and delay, authors in [7] proposed an energy-delay aware window control algorithm based on the gradient descent method, enabling adaptive adjustment of slot count and slot duration according to the number of STAs in each RAW group. Similarly, the authors in [20] proposed an group-size-adaptive algorithm to determine the duration of each RAW. To cope with dynamic changes in the network size and heterogeneous traffic conditions in sensor networks with uplink traffic, authors in [21] proposed TAROA, which can adaptively adjust RAW parameters according to the current (or estimated) traffic conditions and assign STAs to different RAW groups based on the estimated transmission frequency. TAROA has been further refined in [22]. Oriented towards delay-sensitive emergency alarm sensor networks and closed-loop communication scenarios, authors in [23] proposed an RAW parameters selecting algorithm to minimize channel time-sharing consumption. Additionally, in [8], authors formulated the optimal RAW scheduling problem as an integer nonlinear programming problem with the objective of minimizing channel time at key STAs, and designed a heuristic algorithm to find the optimal RAW configurations.

Moreover, some studies focus on RAW grouping, which allocates STAs to different RAW groups based on various characteristics of STAs. According to the priority level of the STAs, authors in [24] proposed a QoS-aware priority grouping and scheduling algorithm. Considering traffic characteristics (e.g., traffic demand, mult-irate) of the STAs in heterogeneous networks, authors in [25] proposed MoROA that employs mathematical methods to solve the grouping problem and to determine the optimal RAW configurations. To achieve fairness in inter-group throughput and channel utilization, authors in [9] and [26] proposed heuristic grouping algorithms. Furthermore, in [27] and [28], authors introduced grouping strategies based on greedy algorithms and genetic algorithms, respectively.

2.3. AI-Based Methods for RAW Mechanism

It is noteworthy that in recent years, there has been a growing number of studies employing AI methods to solve RAW parameter optimization and grouping problems. Authors in [29] proposed a surrogate model for RAW performance in realistic IoT scenarios by integrating ML methods such as support vector machine and artificial neural networks (ANNs). This model accurately predicts network performance for given RAW configurations in heterogeneous networks. The predicted values can serve as inputs for real-time RAW parameters optimization algorithms, thereby enhancing algorithm accuracy. In [10], authors used ANNs to find the optimal number of RAW groups given the network size, data rate, and RAW duration. Using ML methods such as K-means, the authors implemented traffic classification and grouping schemes that can dynamically adapt to various network conditions (e.g., received signal strength, multiple rates, traffic load, and traffic arrival interval) [11,30,31,32]. In the recent study [33], the authors employed recurrent neural network based on gated recurrent units to estimate the optimal number of RAW slots, enhancing the performance in dense IEEE 802.11ah IoT network. To the best of our knowledge, there are limited prior studies using DRL methods for RAW mechanism optimization.

3. RAW Mechanism in Wireless IoT Networks

In this paper, we consider uplink data transmissions in a wireless IoT network employing the RAW mechanism. As shown in Figure 3, the network consists of one center-located AP and N randomly distributed STAs within a coverage range of several hundred meters. STAs transmit sensory data to the AP using a specific channel access control protocol. The network traffic includes periodically generated data, as well as randomly generated data following a certain probability distribution. Since the IEEE 802.11ah standard is an ideal choice for low-power IoT networks, we employ the IEEE 802.11ah-based RAW mechanism for multiple STAs access. We further describe the RAW mechanism operating in the IoT network.

3.1. Operation of the RAW Mechanism

In Section 1, we briefly introduce the idea of RAW. In this section, we elaborate on the RAW parameter set involved in RAW configuration and the channel access process based on RAW in a beacon interval. We aim to explain how key RAW parameters influence network performance at the mechanism principle level.

3.1.1. Structure of the RAW Parameter Set

The IEEE 802.11ah standard defines an information element field in the beacon frame for group-based restricted channel access, known as the RAW parameter set (RPS) [4]. RPS primarily consists of one or more RAW assignment subfields. Each RAW assignment subfield contains necessary RAW control subfields, RAW slot definition subfields, and RAW grouping subfields, for performing restricted channel access to one or multiple STAs in a RAW. According to specific requirements, elements such as RAW start time, channel indication, and periodic operation parameters subfields are conditionally present. The RAW slot definition subfield further defines the slot duration, slot count, and access restrictions between slots. As beacon frames are broadcast by the AP, STAs can learn from the related subfield of the RPS element which RAW group they belong to, as well as the number of RAW groups in a beacon interval, the number of slots in each RAW, and the duration of a single slot in each RAW. The specific rules for calculation are described as follows.

Slot duration and slot count. The formula for calculating the duration of a single slot in a RAW is as follows [4]:

$$T_{slot}=500us+C\times 120us.$$

(1)

Let the length of slot duration count field be y. According to the IEEE 802.11ah standard, when $y=11$ bits, $C=2^{11}-1=2047$, the maximum duration of a slot is $T_{slot}^{max}=246.14$ ms, and the maximum number of slots in a RAW is $K_{max}=2^{14-y}=7$. When $y=8$ bits, $C=2^8-1=255$, $T_{slot}^{max}=31.1$ ms, $K_{max}=2^{14-y}=63$. The selection of y depends on the number of STAs in each RAW. Apparently, the duration of a RAW can be calculated as $T_{RAW}=K·T_{slot}$.

Slot assignment. A mapping method for allocating STAs into the corresponding slots in a RAW is defined in the IEEE 802.11ah standard [4]. It is implemented by defining a mapping function:

$$i=f\left(x\right)=(x+N_{offset})modK,$$

(2)

where x is the assosiation ID (AID) of the STA in a RAW group, $N_{offset}$ is the allocation offset, which means that the first STA in the group will be allocated to the $N_{offset}-th$ time slot, and K is the number of slots in a RAW. We provide an illustration of slot allocation in a RAW as shown in Figure 3. If $N_{offset}=1,K=1$, the STA with AID-1 is allocated to $slo{t}_2$. The mapping function ensures a uniform distribution of STAs across slots.

Cross slot boundary. The IEEE 802.11ah standard defines restrictions on channel access across slot boundaries. STAs can access the channel either in a cross-slot-boundary way or in a non-cross-slot-boundary (NCSB) way [4]. To alleviate the hidden nodes problem and facilitate performance analysis, it is generally advisable to employs the non-cross-slot-boundary mechanism [16]. Therefore, the holding time is defined to be $T_H\ge T_{TXOP}$, where $T_{TXOP}$ is the time required for one successful data transmission, and its expectation can be obtained through statistical analysis. With this constraint, it can be ensured that the last data transmission in the current slot has been completed by the end of slot. If the time remaining in the current slot is not sufficient for one data transmission, the STAs cache their data and wait for the arrival of the next slot to which they belong.

3.1.2. RAW-Based Channel Access and Data Transmission

The channel access and data transmission process of STAs in the IEEE 802.11ah network with RAW mechanism can be summarized as follows and shown in Figure 3.

• The STAs listen to the beacon frames broadcast by the AP, request association and authentication, and receive their AID. The AP periodically broadcasts beacon frames carrying the RPS element and informs STAs of information including their RAW group, the slot count in a RAW, and the slot duration. STAs are then assigned to different slots based on the mapping function (2).
• The STAs contend for channel access following the enhanced distributed channel access (EDCA) mechanism when their slot arrives: STAs perform carrier listening for a distributed inter-frame spacing (DIFS) time before initiating channel access. Once the channel is sensed idle, STAs start decreasing their backoff counter, and initiate channel access when their backoff counter reaches zero. If $ST{A}_a$’s backoff counter decreases to zero before $ST{A}_b$’s, $ST{A}_a$ initiates channel access, while $ST{A}_b$ suspends its backoff counter until the channel is sensed idle again.
• If the backoff counters of two or more STAs in the network decrease to zero simultaneously, these STAs attempt to access the channel at the same time, which may result in collisions. Upon encountering a collision, the STAs increase and reset their backoff counter until they reach the maximum retry limit, at which point packet loss occurs.
• STAs that successfully access the channel will transmit their data after waiting for a short inter-frame spacing (SIFS) time. A received acknowledgment (ACK) frame from the AP indicates the completion of data transmission. The time taken for one data transmission is denoted as $T_{TXOP}$.

The operation of the RAW mechanism elaborated above can provide a preliminary explanation at the mechanism level for the significant impact of RAW parameters on network performance: Firstly, an excessive number of STAs contending for channel access in a slot increases the probability of collisions and packet loss. Secondly, inadequate slot duration limits the amount of data STAs can transmit, thereby reducing overall throughput. Lastly, due to the NCSB mechanism, an excessive number of slots leads to frequent slot boundary switching, resulting in increased holding time overhead and buffered data. In the next subsection, we will analyze the impact of RAW parameters on network throughput at the mathematical analysis level.

3.2. Performance Modeling and RAW Parameters Optimization

Assuming that the number of RAW groups is denoted as $N_{RAW}$, the number of slots in each RAW group is represented by $k_i$, and the duration of a slot in each RAW group is denoted as $t_i$, where $i\in \left[1,N_{RAW}\right]$. Thus, the set of number of STAs in each RAW group, the set of number of slots in each RAW, and the set of slot durations for RAW groups are represented as ${\mathbf{N}}_{STA}=\{n_1,...,n_i,...,n_{N_{RAW}}\}$, ${\mathbf{K}}_{RAW}=\{k_1,...,k_i,...,k_{N_{RAW}}\}$, and ${\mathbf{T}}_{RAW}=\{t_1,...,t_i,...,t_{N_{RAW}}\}$, respectively.

The correlation between RAW parameters and network throughput can be derived based on the analytical model proposed in [14]. Given that the STAs are uniformly distributed among slots in a RAW, the number of STAs in each slot can be approximated as $x_i=\frac{n_i}{k_i}$, and the intensity of contention in each slot is considered to be the same. Consequently, for the STAs in each slot of the i-th RAW, the probability of STAs suspending their backoff counter is defined as $p_{f,i}({\tau }_i,x_i,t_i)$, indicating that the suspending probability is related to the transmission probability ${\tau }_i$, the number of STAs in each slot $x_i$, and the slot duration $t_i$. The collision probability is denoted as $p_{c,i}$.

The backoff process of a STA’s backoff counter can be analyzed using a two-dimensional Markov chain [14]. Each state during the backoff process can be represented as a probability, and the steady-state probability of each state can be further determined. According to the normalization formula, a closed-form expression for the steady-state probability of the backoff counter decreasing to zero can be obtained as $b_{i,0}(p_f,p_c,C{W}_{min},m)$, indicating that the steady-state probability at state-0 is dependent on the suspending probability $p_{f,i}$, the collision probability $p_c$, the given minimum size of the contention window $C{W}_{min}$ and retry limit m. Subsequently, the transmission probability can be computed as

$${\tau }_i=\frac{1-{\left(p_{c,i}\right)}^{m+1}}{1-p_{c,i}}{b}_{i,0}.$$

(3)

The collision probability is given by $p_{c,i}=1-{(1-{\tau }_i)}^{x_i-1}$, and the probability that at least one STA transmits data in a slot is denoted as $P_{tr,i}=1-{(1-{\tau }_i)}^{x_i}$. Furthermore, the successful transmission probability can be represented as

$$P_{suc,i}=\frac{x_i{\tau }_i{(1-{\tau }_i)}^{x_i-1}}{P_{tr,i}}.$$

(4)

The normalized slot throughput can be calculated as

$$u_i=\frac{P_{tr,i}{P}_{suc,i}E\left(D\right)}{(1-P_{tr,i})\sigma +P_{suc,i}{T}_{suc}+p_{c,i}{T}_c},$$

(5)

where $E\left(D\right)$ represents the average payload size of a data frame, $\sigma$ is the time of a mini-slot in the contention window. The time for a successful data transmission and the time spent due to collision is respectively denoted as $T_{suc}$ and $T_c$, which are calculated in [14]. The effective time for data transmissions in a slot is $t_i^{\prime }=t_i-T_H$. Finally, the normalized throughput of the network can be denoted by

$$U=\sum _{i=1}^{N_{RAW}}\frac{u_i{k}_i{t}_i^{\prime }}{T_{BI}},$$

(6)

where the duration of the beacon interval $T_{BI}$ is dependent on the total duration of RAWs in one beacon interval.

According to (6), network throughput is related to the successful transmission probability, which in turn depends on the collision probability and the transmission probability. These probabilities are influenced by the number of STAs in a slot and the slot duration. Moreover, the number of RAW groups and the number of slots in a RAW jointly determine the number of STAs in a slot. Intuitively, the increasing number of RAW groups and slot divisions reduce the number of STAs per slot, thereby decreasing the collision probability. Increasing the slot duration, on the other hand, allows more time for data transmission in a slot, thereby reducing data buffering. Therefore, increasing the number of RAW groups, dividing more slots in a RAW and extending the slot duration can greatly enhance network throughput. However, excessive RAW divisions may cause more STAs to remain idle, leading to data buffering. Similarly, an excessively long slot duration may result in wasted time in networks with low traffic loads. There is a trade-off in adjusting the RAW parameters. Hence, by jointly optimizing the number of RAW groups $N_{RAW}$, RAW slot counts $k_i\in {\mathbf{K}}_{RAW}$, and slot durations $t_i\in {\mathbf{T}}_{RAW}$ with $i\in \left[1,N_{RAW}\right]$, we can formulate the network throughput maximization problem as follows:

$$\begin{array}{cc}\hfill \underset{N_{RAW},{\mathbf{K}}_{RAW},{\mathbf{T}}_{RAW}}{max}& U\hfill \\ \hfill s.t.& \left(1\right),\left(3\right),\left(4\right),\left(5\right),\mathrm{and}\left(6\right)\hfill \\ \hfill & \sum _i{k}_i{t}_i\le T_{BI}.\hfill \end{array}$$

(7)

The existing studies prefer to construct complicated analytical models of RAW, and further propose optimization methods to find the optimal RAW parameters to improve network throughput. However, solving RAW parameters optimization problems based on analytical models may lead to a high level of computational complexity or even impracticality in dynamic networks. On the one hand, these analytical models require a series of assumptions, including saturated network traffic, ideal channel conditions, and packet loss solely caused by collisions. Moreover, the analytical models do not comprehensively consider details about the RAW mechanism and channel conditions. Although some studies have refined the analytical models and taken more complex network conditions into account, this has made the analysis process more cumbersome. On the other hand, due to the mathematical or heuristic methods often involve complex rules and have not been validated in different network scenarios, their generalization ability in complex and dynamic network conditions needs to be improved.

To investigate practical network states, the IEEE 802.11ah network simulation environment is developed based on a widely used network simulator called NS-3 [5]. NS-3 is used to bulid simulation environments that closely resemble real-world network environments. The partial mechanisms of the PHY and MAC layers including the RAW mechanism are also implemented. While analytical results serve as references for optimizing RAW parameters, the simulation environment implemented by NS-3 undoubtedly provides more accurate results and can serve as a benchmark for validating these analytical results. Additionally, with the capability of handling complex and dynamic environments, DRL-based methods are well-suited for addressing RAW parameters optimization problems and demonstrate strong generalizability across various scenarios.

To determine the optimal RAW parameters in complex network environments that closely mimic real-world scenarios, we construct network environments using the NS-3 simulator, and employ the DRL-based method to decide on the RAW parameters to enhance network throughput. The specific methodology will be elaborated in the following section.

4. DRL for RAW Parameters Optimization

In this section, we propose a learning framework for optimizing RAW parameters. As depicted in the Figure 4, we set up network simulation environments in NS-3 and execute agent training in the DRL environment. The PPO algorithm [34] is employed as the specific DRL implementation algorithm in the DRL-guided NS-3 simulation framework. During training, the DRL agent receives network observations from the NS-3 simulation environment, serving as inputs to the DNNs. Each learned action (i.e., RAW parameters) are then applied as the configuration parameters for the RAW mechanism in NS-3, and a new simulation is executed to obtain an updated reward (i.e., network throughput). The DRL agent continues to receive observations from the network environment for a new training episode. Interactions between the DRL agent and the NS-3 environment continue until the DRL agent learns the optimal RAW parameters and achieves enhanced network throughput. To utilize PPO for optimizing RAW parameters to maximize network throughput, we first reformulate problem (7) as an MDP.

4.1. MDP Reformulation

Given the network conditions, we aim to optimize the RAW parameters (i.e., the number of RAW groups, the number of slots in each RAW and the slot duration in each RAW) to reduce contentions among STAs and consequently improve network throughput. To facilitate the problem formulation, the following assumptions are made for the IoT network:

• The AP collects information about the network (e.g., network size N and traffic arrival of the STAs) through management frames. Based on received packets from the STAs, network performance such as throughput and packet loss ratio can be statistically determined.
• To alleviate hidden nodes issues and collisions, STAs obey the NCSB mechanism when accessing the channel among slots.

In RL, the interaction between the agent and the environment is typically modeled as an MDP, which can be represented by a tuple $(\mathbf{S},\mathbf{A},P,r,\gamma )$, where $\mathbf{S}$ represents the state space, $\mathbf{A}$ represents the action space, the transition probability function $P\left(s^{\prime }\right|s,a)$ represents the probability of transitioning from state s to state $s^{\prime }$ when action a is taken, the reward function $r(s,a)$ represents the reward obtained after taking action a in state s, and $\gamma \in [0,1]$ is a constant discount factor. Specifically, the definitions of state, actions, reward, and observations are given as follows:

• State: The state at the current time step is defined as the throughput obtained from the current simulation statistics, denoted as $s_t=U_t$. During the simulation, the AP collects the number of packets received and the payload size of each packet at the current time step to calculate the network throughput at the end of the current step.
• Action: The actions in the MDP are defined as the RAW parameters, including the number of RAW groups, the number of slots in each RAW group, and the slot duration in each RAW group. Thus, the action at step t is denoted as $a_t=(N_{RAW},{\mathbf{K}}_{RAW},{\mathbf{T}}_{RAW})$.
• Reward: According to the optimization objective, the reward is defined as the throughput obtained at each time step, represented as $r_t=U_t$.
• Observation: The observation set is defined as the network information observable by the AP, including network size N, the set of traffic loads $\mathbf{D}$, and the set of traffic intervals $\mathbf{I}$, which can be represented as $o_t=(N,\mathbf{D},\mathbf{I})$.

In the following subsection, we elaborate on the PPO algorithm for RAW parameters optimization.

4.2. PPO for Optimizing RAW Parameters

Given a policy approximator ${\pi }_{\theta }\left(a\right|s)$ with parameters $\theta$, policy-based policy gradient (PG) algorithms find the optimal $\theta$ to maximize the reward or value function. For a given input state $s_t$, the policy network directly outputs either the action or the probability associated with the action. It then selects the appropriate action based on the probability, allowing the output action to be a continuous value. The expected value function in PG algorithms can be represented in terms of the policy parameters as

$$\begin{array}{c}\hfill J\left(\theta \right)=\sum _s{d}^{{\pi }_{\theta }}\left(s\right)V^{{\pi }_{\theta }}\left(s\right)=\sum _s{d}^{{\pi }_{\theta }}\left(s\right)\sum _a{\pi }_{\theta }(s,a)Q^{{\pi }_{\theta }}(s,a),\end{array}$$

(8)

where $d^{{\pi }_{\theta }}$ is the stationary distribution of the Markov chain for ${\pi }_{\theta }$, and $Q^{{\pi }_{\theta }}(s,a)$ denotes the Q-value of the state-action pair $(s,a)$ following the policy ${\pi }_{\theta }$. The goal of PG is to find parameters $\theta$ that maximize $J\left(\theta \right)$ by ascending he gradient of the policy. The evaluation of the policy gradient ${\nabla }_{\theta }J\left(\theta \right)$ can be simplified as [36]

$$\begin{array}{c}\hfill {\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{\pi }[Q^{\pi }(s,a){\nabla }_{\theta }ln{\pi }_{\theta }\left(a\right|s))]={\mathbb{E}}_{\pi }[\sum _{t=1}^T{Q}^{\pi }(s_t,a_t){\nabla }_{\theta }ln{\pi }_{\theta }\left(a_t\right|s_t))],\end{array}$$

(9)

where the expectation is taken over all possible state-action pairs following the same policy ${\pi }_{\theta }$. The policy gradient ${\nabla }_{\theta }J\left(\theta \right)$ can be evaluated by sampling historical decision-making trajectories.

For each episode, all the $(s,a,r,s^{\prime })$ tuples acquired by the agent can be collectively represented as a state-action trajectory resulting from the agent’s interaction with the environment over the current episode, which is denoted as $\tau =(s_0,a_0,r_1,s_1,...,s_{T-1},a_{T-1},r_{T-1},s_T)\sim ({\pi }_{\theta },P\left(s_{t+1}\right|s_t,a_t))$. Let $G_t={\sum }_{k=t}^{T}r(s_k,a_k)$ be the reward for a trajectory $\tau$, and estimate the Q-value $Q^{\pi }(s_t,a_t)$ in (9) by $G_t$. Therefore,the policy gradient in each time step can be approximated by randomly sampling $G_t{\nabla }_{\theta }ln{\pi }_{\theta }\left(a_t\right|s_t))$, and the policy parameters can be updated as $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J\left(\theta \right)$ , where $\alpha$ denotes the step size for gradient update. To reduce prediction variability and improve learning efficiency, the value function $V_{\pi }\left(s\right)$ can be used as the baseline, and the advantage function $A_{\pi }(s,a)\triangleq Q^{\pi }(s,a)-V_{\pi }\left(s\right)$ is further introduced to replace $G_t$.

To address high-dimensional state and action spaces while stabilizing the learning process, actor-critic (AC)-based DRL algorithms introduce a DNN with weight parameters $\omega$ to approximate the Q value. AC algorithms update both the policy network and the Q-value network. Specifically, at each learning step t, the actor updates the policy network by updating the policy parameters $\theta \leftarrow \theta +{\alpha }_{\theta }{Q}_{\omega }(s,a){\nabla }_{\theta }ln{\pi }_{\theta }\left(a\right|s))$, while the critic updates the Q network by minimizing a loss function and updates the parameters $\omega \leftarrow \omega +{\alpha }_{\omega }{\delta }_t{\nabla }_{\omega }{Q}_{\omega }(s,a)$ by gradient ascent, where ${\delta }_t=r_t+\gamma Q_{\omega }(s^{\prime },a^{\prime })-Q_{\omega }(s,a)$ denotes the TD error. To further stabilize the training process, the deep deterministic gradient policy (DDPG) algorithm [36] utilizes two DNNs with different parameters, i.e., the online Q-network $Q_{\omega }(s,a)$ and the target Q-network $Q_{{\omega }^{\prime }}(s,a)$. The TD error is rewritten as ${\delta }_t=r_t+\gamma Q_{{\omega }^{\prime }}(s^{\prime },a^{\prime })-Q_{\omega }(s,a)$.

To facilitate the agent’s utilization of past experiences and improve sample efficiency, PG can be transformed into off-policy learning through the utilization of importance sampling [37]. Sample collection can be conducted under a behavior policy ${\pi }_o(s,a)$ distinct from the target policy ${\pi }_{\theta }\left(a\right|s)$. To mitigate the effects of improper step size in policy optimization on training stability, the off-policy trust region policy optimization (TRPO) algorithm imposes an additional constraint on the gradient update [37], ensuring that the old and new policies do not diverge significantly. Let ${\rho }_{theta}=\frac{{\pi }_{\theta }(s,a)}{{\pi }_o(s,a)}$ denote the probability ratio of the divergence between the old and new policies. TRPO maximizes the objective by applying conservative policy iteration without limiting the probability ratio to an appropriate range. This could lead to an excessively large policy update. Intuitively, a smaller deviation between the behavior policy and the target policy is better. Hence, the PPO algorithm [34] modifies the objective by constraining ${\rho }_{\theta }$ in a region $[1-ϵ,1+ϵ]$, and penalizing changes to the policy that move ${\rho }_{\theta }$ away from 1. The objective in $PP{O}_{CLIP}$ is

$$\begin{array}{cc}\hfill \underset{\theta }{max}& L^{CLIP}\left(\theta \right)=\tilde{J}\left(\theta \right)={\mathbb{E}}_{{\pi }_o}[min\{{\rho }_{\theta }{\widehat{A}}_{{\pi }_o},clip({\rho }_{\theta },1-ϵ,1+ϵ){\widehat{A}}_{{\pi }_o}\}]\hfill \\ \hfill s.t.& D_{KL}({\pi }_o,{\pi }_{\theta })\le {\delta }_{KL},\hfill \end{array}$$

(10)

where $D_{KL}(P_1,P_2)\triangleq {\int }_{\infty }^{\infty }{P}_1\left(x\right)log(P_1\left(x\right)/P_2\left(x\right))dx$ denotes a distance measure in terms of the Kullback-Leibler (KL) divergence between two different probability distributions. The advantage function $A_{{\pi }_o}$ in the objective of problem (10) is the approximation of the actual advantage $A_{{\pi }_{\theta }}$ corresponding to the target policy ${\pi }_{\theta }$. PPO constrains the parameters search within a region by introducing the inequality constraint in problem (10), which ensures that the KL convergence between ${\pi }_{\theta }$ and ${\pi }_o$ is bounded by ${\delta }_{KL}$. The clip function returns ${\rho }_{\theta }\in [1-ϵ,1+ϵ]$, and the hyper-parameter $ϵ=0.2$ by default.

During the training of the DNNs, PPO employs fixed-length(e.g. T time steps) trajectory segments. A truncated advantage estimation is computed to replace the advantage function in problem (10) as

$${\widehat{A}}_t^{{\pi }_o}={\delta }_t+\gamma {\delta }_{t+1}+\cdots +{\gamma }^{T-t+1}{\delta }_{T-1},$$

(11)

where ${\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)$. The loss function of PPO is

$$L_t\left(\theta \right)={\mathbb{E}}_t[L_t^{CLIP}\left(\theta \right)-c_1{L}_t^{VF}\left(\theta \right)+c_{2}S\left[{\pi }_{\theta }\right]\left(s_t\right)],$$

(12)

where $L^{VF}=V_{\theta }\left(s_t\right)-V_t^{target}$ is the mean squared error loss, $c_1,c_2$ are coefficients, S is an entropy bonus.

The PPO algorithm for optimizing RAW parameters in networks built in NS-3 is summarized in Algorithm 1. PPO utilizes two DNNs to approximate the policy networks. In each learning episode, the PPO agent runs the old/behavior policy ${\pi }_{{\theta }_o}$ (i.e., RAW parameters), observes network throughput obtained from the NS-3 network simulations environment for T time steps, and stores T transition tuples $(s_t,a_t,r_t,s_{t+1}),t\in T$ in the experience replay buffer. Then, it samples mini-batches of transition tuples from the replay buffer and computes advantage estimates ${\widehat{A}}_1^{{\pi }_o},\cdots ,{\widehat{A}}_T^{{\pi }_o}$. Subsequently, weight parameters of the target policy network ${\pi }_{\theta }$ are updated by using mini-batches randomly sampled from the replay buffer through importance sampling and by optimizing the surrogate loss in (12). Weight parameters of the behavior policy network are updated by ${\theta }_o\leftarrow \theta$. Limiting the probability ratio of the two policies ${\rho }_{\theta }\in [1-ϵ,1+ϵ]$ ensures that the probability distribution of the output actions from the two policy networks remains similar.

The uniform grouping scheme is verified to perform better in homogeneous networks [38]. Considering the networks with periodic and random traffic in this paper, we employ the uniform grouping scheme, where STAs are evenly distributed in each RAW. Consequently, the slot duration and number of slots in each RAW group are considered to be equal. As a result, the actions of the MDP can be further simplified to the number of RAW groups, the number of slots in one RAW group, and the slot duration in one RAW group.

5. DRL-guided NS-3 Simulation

In this section, we investigate the performance of the proposed PPO-based DRL algorithm for RAW parameters optimization in networks with periodic or random traffic, which are set up in the NS-3 simulator. We firstly demonstrate the learning performance of the PPO algorithm on finding optimal RAW parameters and improving network throughput. Then, we investigate the adaptive capability of RAW parameters under dynamic network conditions such as traffic load and network size. Finally, we compare the performance of the PPO-based slot division scheme with the equal-slot-division scheme (i.e., one STA per slot) and no-slot-division scheme (i.e., only one ’slot’ in a RAW).

5.1. Simulation Setup

We set up the training environment for DRL on the Linux operating system. Specifically, we set up the DRL agent in a Python environment based on the PyTorch framework, and set up the network topology in the NS-3 simulator as depicted in Figure 3. Network conditions and simulation results are input into the PPO agent as environment states. The RAW parameters are configured based on the actions learned by the PPO agent and used for subsequent simulations in the NS-3 simulator. Throughout the training process, the PPO agent interacts numerous times with the simulated network environment set up in the NS-3 simulator.

The parameters for our network topology and the PPO agent are shown in Table 1. For analytical and simulation design purposes, we define the time step t as each fixed-duration simulation iteration performed in NS-3, where each episode consists of only one step. When performing simulations, we collect statistical information regarding network performance after every simulation iteration with a duration of 10 seconds. Note that the number of slots $K_{RAW}$ is in the range of $[1,63]$ according to the restriction in (1). When $K_{RAW}\in [1,7]$, the maximum slot duration is 246.14 ms, and when $K_{RAW}\in [8,63]$, the maximum slot duration is 31.1 ms. This can serve as a constraint for the agent during learning.

5.2. Learning Performance in Periodic Traffic Networks

We first consider the RAW parameters optimization in networks with periodic traffic, where all STAs are assigned to one RAW group. The action of the MDP is $a_t=(K_{N_{RAW=1}},T_{N_{RAW=1}})$. In each iteration during the training process, the PPO agent observes throughput $s_t$ and other information $o_t$ from the wireless environment and employs policy $\pi$ to determine the RAW parameters setting $a_t$ for the next time step. The effectiveness of the RAW parameters is evaluated at the subsequent time step with the reward obtained from NS-3, and the RAW parameters for the following time step are determined accordingly. The PPO agent is trained through numerous interactions with network simulation environments bulit in NS-3.

5.2.1. Convergence to the Optimal RAW Parameters

We first validate the convergence performance of the PPO algorithm on a basic network topology. In the periodic traffic network, the traffic interval of all STAs is fixed (e.g., 0.1 ms). We train the PPO agent though numerous interactions with the network environment built in NS-3. The convergence performance of the PPO algorithm is shown in Figure 5, and Figure 6 demonstrates the convergence process of the PPO agent interacting with the network simulation environment in the NS-3 simulator. As the training iteration proceeds, the PPO agent learned better actions, leading to significantly increased normalized rewards obtained from interacting with the NS-3 simulation environment. After 10,000 training episodes, the reward stabilized at its maximum value. This indicates that the PPO agent has learned the optimal RAW parameters by the end of training and achieved the optimized network throughput in the periodic traffic network. We also observe the improvement of network throughput with the NS-3 simulator during the PPO agent’s training process. It can be seen in Figure 6 that the network throughput is ascending when the training process proceeds. Compared to the network throughput obtained with default settings ($K_{RAW}=1,TBI=100ms$), the network throughput with the optimal RAW parameters obtained by PPO agent is improved by about 70%. It is evident that employing DRL method for optimizing RAW parameters is feasible, and the RAW parameters derived from learning lead to a significant enhancement in network throughput compared to default settings.

Additionally, we depict the convergence performance of the slot count within a RAW and the slot duration with different number of STAs in the network. To provide a more straightforward demonstration, we calculate the approximate duration of a beacon interval as $T_{BI}=N_{RAW}·K·T_{slot}$ and use beacon interval dynamics to represent variations in slot duration in the following subsections. As shown in Figure 7, both parameters converge to stable values for different network sizes, further validating the algorithm’s convergence. As the network size is relatively small, the number of slots is similar when the number of STAs in the network is 40, 50 and 60 respectively. The duration of the beacon interval increases by about 40% when the number of STAs in the network increases from 40 to 60, indicating that the slot duration is adaptively adjusting to the network size with DRL.

Figure 7. Convergence performance of $K_{RAW}$.

Figure 7. Convergence performance of $K_{RAW}$.

Figure 8. Convergence performance of $T_{BI}$ ($w.r.t.T_{slot}$).

Figure 8. Convergence performance of $T_{BI}$ ($w.r.t.T_{slot}$).

5.2.2. Throughput Performance with Different Traffic Loads

In this section, we analyze the adaptive adjustment of RAW parameters obtained through PPO method with different network loads. We assume homogeneous traffic load 0.05 Mbps for each STA. Therefore, the traffic load in the network increases as the number of STAs in network increases. We observe the adaptive adjustments of RAW parameters and changes in network throughput as the number of STAs increases from 30 (10) to 100.

As shown in Figure 9, both the slot count and slot duration increase with the growing number of STAs and traffic load in network. The number of slots in a RAW increases stepwise with the network size and traffic load. Specifically, the slot count remains constant when the number of STAs is between 50-70 and 80-90. This is attributed to the fact that dividing fewer slots in a RAW significantly reduces contentions when the network size is small. Overall, the RAW mechanism ensures that the number of STAs in each RAW slot is not excessive. When the number of STAs in the network is less than 50, the slot duration increases significantly from $10ms$ to about $50ms$, approximately 4 times longer, with the increasing network size and traffic load.

This trend is consistent with the changes in network throughput depicted in Figure 10. When the number of STAs in the network is less than 40, the network throughput remains unsaturated with few STAs and low traffic load in the network. As the number of STAs increases from 10 to 40, along with the ascending network traffic load, the network throughput subsequently increases from $0.076Mbps$ to $0.248Mbps$, approximately 4 times larger. At a certain point, with number of STAs = 40, the network traffic load reaches its maximum capacity, leading to saturated network throughput under current network conditions. As the number of STAs in the network continues to increase from 40 to 90, contentions intensify, leading to a higher probability of transmission collisions. Meanwhile, when the network traffic load exceeds its capacity, constrained by the data transmission rate and the duration of a single simulation, the AP cannot handle all the traffic from STAs, resulting in a slight decrease by about 7% in network throughput. This trend is consistent with the variation of throughput with the number of STAs in IEEE 802.11 networks [14].

5.3. Learning Performance in Random Traffic Networks

To further validate the generalization ability of the proposed DRL algorithm, in this section, we modify the network conditions. While in the previous subsection, packets are transmitted at identical intervals, we now adjust the packet transmission intervals for each STA in the network. As depicted in Figure 11, the packet transmission intervals of each STA are modeled by a Poisson distribution, introducing randomness into the transmission process. Given that all STAs have the same data transmission rate, the network traffic can be considered as random. Moreover, we significantly increase the network size to emulate real-world network scales. The parameter settings are shown at Table 2. We scaled up the number of STAs in the network to exceed 150, and increase the training iterations of the PPO agent.

5.3.1. Convergence to the Optimal RAW Parameters

We first validate the convergence performance of the PPO algorithm in the new network conditions. In the random traffic network implemented in NS-3, all the STAs transmit packets at random intervals following a Poisson distribution. Additionally, the network size is larger than that in the periodic traffic network, necessitating the division of STAs into more RAW groups. Therefore, the RAW parameters to be learned include RAW group count, slot count in one RAW and slot duration in one RAW.

As shown in Figure 12, the normalized reward obtained by the PPO agent from interacting with the NS-3 simulation environment increases significantly as the training iterations progress, stabilizing at its maximum value after 17,000 training episodes. This indicates that the PPO agent can still learn the optimal RAW parameters and achieve optimized network throughput in the new network environment, i.e., the random traffic network. We also observe that as the network conditions become more complex, such as an increase in network size and random traffic arrivals, the PPO agent requires more interactions with the network simulation environment set up in NS-3. It needs to learn the optimal action selection strategy over twice as many training iterations in the random traffic network compared to the periodic traffic network.

As depicted in Figure 13, Figure 14 and Figure 15, when the number of STAs in the netwotk is 150, both the number of RAW groups and the slot duration significantly increase compared to those in a small network size, while the number of slots remains small. This implies that the PPO agent tends to divide more RAW groups rather than more slots at this network size. The convergence performance to the optimal RAW parameters obtained by the PPO agent further demonstrates the generalization capability of the proposed DRL algorithm in a complex network environment with random traffic.

5.3.2. Throughput Performance with Different Network Sizes

In this subsection, we validate and analyze the adaptive adjustment of RAW parameters obtained using the PPO algorithm under different network sizes, which is reflected by changes in the number of STAs. We increase the number of STAs from 150 to 300, a sufficiently large number to achieve saturated or oversaturated traffic load, which is suitable for validating the adjustment capability of RAW parameters and network throughput of the proposed DRL framework. We observe the adaptive changes in RAW parameters learned by the PPO agent and the network throughput obtained from the NS-3 simulation environment as the number of STAs increases. As shown in Figure 16, given that traffic load reaches saturation in large-scale networks, with the number of STAs in the network increasing from 150 to 300, the network throughput decreases by about 13%. It is evident that the increasing network size leads to intensified contention and collisions, thereby resulting in a significant decline in network throughput.

As shown in Figure 17, we observe that when the number of STAs ranges from 150 to 200, the PPO agent tends to divide STAs into roughly 3 times more RAW groups. However, when the number of STAs increases to 250-300, the agent leans towards dividing more slots (from 1 to 5) in each RAW group. We analyze that within a certain range of network sizes, simply dividing RAW groups is capable of handling the current traffic load and mitigating contention. However, as the network size grows, it becomes necessary to both divide RAW groups and more slots within each RAW group. The adaptive adjustment strategy learned by the PPO agent reduces the contention among STAs per slot, thus ensuring network throughput. Additionally, as the network size increases, the agent prefers to shorten the slot duration, consequently reducing the beacon interval duration by about 10%. We analyze that shortening the beacons broadcasting period allows the AP to schedule STAs more frequently for uplink data transmissions, thereby maintaining network throughput in intensified network conditions. We also notice that as the number of STAs increases from 60 to 150, the BI duration obtained by DRL increases from $60ms$ to $100ms$, approximately by 66%, and the slot count increases significantly by about 3 times compared to the network size in the periodic traffic network. It is evident that as the network size scales up, contentions between STAs in the network intensify. To alleviate collisions and ensure network throughput, the PPO agent tends to dividing more slots, leading to an overall increase in the BI duration.

5.4. Throughput Comparison of Different Slot Division Schemes

To further demonstrate the improvement in network performance achieved by the PPO-based algorithm, we compare the network throughput obtained from the PPO-based slot division scheme with two baseline slot division schemes. In the no-slot division scheme, all STAs contend for channel access in the same RAW group without slot division. Conversely, in the equal-slot division scheme, each STA is allocated one slot in every RAW group, ensuring non-contention-based access where only one STA can access the channel in a slot.

The overall BI durations are the same among different slot division schemes, as determined by the BI duration learned by the PPO agent. In this case, the slot durations vary among different schemes due to different slot division methods. As depicted in Figure 18, in generally speaking, the throughput performance obtained from the NS-3 simulation environment with RAW slot division scheme learned by the PPO agent significantly outperform the two basic schemes as the number of RAW slots and slot duration adaptively adjust according to the network size. In the worst case where the number of STAs in the network is 300, the network throughput obtained from the PPO-based slot division scheme is still improved by about 80% and 1.3 times, respectively, compared to the two slot division schemes.

It can be observed that as the network size increases, contentions and collisions among STAs in the network intensify, leading to a decrease in network throughput for all three slot division schemes. However, when the number of STAs in the network is between 150 and 250, the decrease in network throughput obtained from the PPO-based slot division scheme in the NS3 simulation environment is much smaller than that of the other two schemes. This indicates that the learning-based RAW slot division scheme can maintain better network throughput than the basic slot division schemes in deteriorating network conditions. It can be verified that in the DRL-guided NS-3 simulation framework, the PPO agent can learn the optimal RAW parameters and improve the network throughput in situations of intense contention. We also observe that as the network size increases, the network performance obtained by the scheme that allocates one slot to each STA is significantly better than that of the scheme that does not divide slots within a RAW. This further validates the necessity of using the RAW grouping mechanism in large-scale networks and its improvement on network performance.

6. Conclusion

In this paper, we have proposed a PPO-based DRL algorithm for optimizing RAW parameters in the IEEE 802.11ah-based IoT network. Necessary analysis is first provided to emphasize the significant impact of RAW parameters on network throughput, and the RAW parameters optimization problem is formulated. A DRL framework interacting with the NS-3 simulator is then proposed, in which the optimization problem is reformulated as an MDP, and a PPO algorithm for RAW parameters optimization is proposed. In network environments with periodic and random traffic built in the NS-3 simulator, the performance of the proposed DRL algorithm is evaluated. Simulation results show that the PPO-based DRL scheme can obtain optimal RAW parameters under different network conditions, and achieve significantly improved network throughput compared to that of the baseline slot division schemes.

For future studies, we aim to explore the applications of DRL methods in more intricate network scenarios and optimization problems. For networks with time-varying network size and heterogeneous traffic, apart from optimizing RAW parameters, we will delve into the RAW grouping problems. The RAW grouping problems involve categorizing STAs into different RAW groups, which require optimizing grouping strategies. In this paper, we have demonstrated the capability of the DRL approach to effectively determine the optimal RAW parameters across different network environments. Consequently, the DRL approach can be extended to address the challenge of finding the optimal RAW grouping strategy. Furthermore, the dynamic nature of the network necessitates requires algorithms to perform in real-time, prompting us to further refine our algorithm.