Joint Scheduling and Resource Allocation in Heterogeneous Queuing Systems with Bursty Traffic: A Constrained Soft Actor-Critic Approach

1. Introduction

Joint scheduling and resource allocation in heterogeneous queuing systems—where multiple queues with distinct priorities and queuing delay requirements contend for a single output link—is critical for providing differentiated quality of service (QoS). It underpins a wide range of communication and network systems, ranging from the output-port scheduling of routers and switches under heterogeneous QoS classes, to per-class service sharing within a 5G network slice at the base station [1], and to resource allocation at edge-gateway nodes in industrial internet of things (IoT) deployments where alarm, control, and best-effort traffic coexist [2].

However, maintaining the queuing delay requirement of high-priority traffic while maximizing system throughput utility presents a fundamental challenge in these systems. This tension lies at the core of any joint scheduling and resource allocation problem. Specifically, biasing transmission service toward the high-priority queue reduces its waiting time but increases that of others, whereas throughput-oriented scheduling keeps the link highly utilized but risks allowing some high-priority packets to exceed their maximum allowable delay. This inherent difficulty is further compounded by the stochastic nature of both arrivals and link capacity. Arrivals are rarely stationary—high-priority traffic may alternate between normal and burst modes, while low-priority traffic can persistently consume a large share of the bandwidth. Meanwhile, the usable link capacity is often not fixed but fluctuates dynamically with the operating environment, further perturbing delay evolution.

Under the combined effect of these stochastic factors—bursty arrivals and dynamic link capacity—conventional heuristic scheduling rules exhibit intrinsic limitations. That is, fixed-weight and backlog-driven policies cannot react to abrupt arrival-mode transitions or to head-of-line packets nearing their deadlines; and deadline-driven rules lack an explicit trade-off in the joint “throughput + delay + drop” setting. Additionally, unconstrained learning policies that encode delay requirements into a scalar reward may accept queuing delay violations in exchange for higher aggregate rewards. Consequently, queuing delay depends not only on current queue length, but also on link capacity fluctuations and the history of past scheduling decisions, making it particularly difficult to control and predict as a key performance indicator [3,4]. These observations motivate treating the queuing delay violation budget as a standalone constraint, separate from the throughput-maximization objective, rather than as an ordinary cost that high rewards can offset.

To this end, this paper proposes a joint scheduling and resource allocation approach for heterogeneous queuing systems based on constrained soft actor-critic (CSAC). In this approach, the scheduler takes actions generated by a neural-network agent that dynamically selects queues and allocates the link capacity among active queues, according to real-time queue states, recent violation feedback, and link-capacity history. The main contributions of this paper are enumerated as follows.

• CSAC-based Network Architecture for Heterogeneous Queuing: We model the high-priority queue’s delay violation as an independent episodic constraint, explicitly decoupled from the throughput objective, such that the binary “violation-or-not” semantics are embedded in the constraint term rather than in hand-tuned reward weights. Building upon this formulation, we develop a CSAC-based solver that integrates this constrained objective into the actor-critic learning process, ensuring stable and reliable policy optimization under stringent queuing delay constraints.
• Continuous-to-Discrete Action Mapping under Per-Time-Slot Link Capacity Constraints: We design a two-stage mapping that transforms the continuous policy logits into integer transmission quotas that are non-negative, do not exceed each queue’s backlog, and sum to the effective capacity available to backlogged queues, so they can be executed directly without post-processing.
• Comprehensive Experimental Validation under Heterogeneous Traffic: We systematically compare the proposed approach against an unconstrained soft actor-critic (SAC) baseline and simplified heuristic baselines under heterogeneous traffic conditions. The experimental scenarios include one queue following a Markov-modulated Poisson process (MMPP) with the highest priority and a stringent queuing delay requirement, as well as two queues following independent Poisson arrivals with lower priorities and best-effort queuing delay requirements. Results verify its advantages in throughput, queuing delay, and overflow packet count behavior.

2. Related Works

Multiple queues with distinct priorities and queuing delay requirements contend for a shared output link. This constitutes a fundamental problem in queue scheduling and transmission resource allocation. Throughput, overflow packet count, and queuing delay are the primary performance metrics for this problem [4]. Existing approaches broadly fall into two categories: conventional non-learning (heuristic) methods and learning-based methods.

2.1. Non-Learning Methods

Non-learning methods do not rely on training data. Instead, they adjust resource allocation at runtime through fixed logic based on predefined rules or analytical models. The deficit round robin (DRR) serves queues through deficit counters and quantum-based transmission at low complexity [5]. MaxWeight allocates resource by backlog-weighted rates and keeps the system stable under suitable conditions [6]. The generalized $c\mu$ rule ties a dynamic priority to residual waiting time and is asymptotically optimal in heavy traffic [7]. The earliest-deadline-first (EDF) orders transmission by deadline and underpins schedulability analysis in hard real-time settings [8]. The fixed-rule and backlog-driven scheduling ideas have also been used for inter-slice capacity allocation in 5G network slicing [1]. These rules perform well under regular conditions, but each depends on fixed weights or model-specific assumptions. Analytical work based on queuing theory and network calculus can provide worst-case delay bounds for such rules [3,9,10]. Tabatabaee and Le Boudec derive dedicated network-calculus delay bounds for DRR [11]. Hu et al. and Yang et al. further study delay-bound analysis and scheduling design for diverse time-sensitive networking (TSN) scenarios [12,13]. However, these analytical results often rely on known arrival envelopes and transmission-rate assumptions. In practical bursty multi-queue systems, traffic arrivals may switch rapidly between different burst modes, and the usable link capacity may fluctuate with the environment. Such dynamics make it difficult to maintain fixed arrival and transmission assumptions during transient bursts. Therefore, adaptive data-driven methods are needed to adjust the transmission process according to the observed system state.

2.2. Learning-Based Methods

Learning-based methods learn a state-to-decision mapping from data through interaction with the environment, without static assumptions on the arrival or transmission processes [14,15]. This learning paradigm has been applied across many scheduling settings. The QoS-aware deep reinforcement learning agent (QADRA) scheduler optimizes per-flow QoS satisfaction in 5G New Radio (NR) [16]. Lu et al. propose an intelligent scheduler to improve the reliability of delay-sensitive flows in edge-enabled industrial IoT [2]. Forero et al. integrate SAC with weighted fair queuing (WFQ) and controlled delay (CoDel) for queue management in undersea acoustic networks [17]. The soft actor-critic-based flow scheduling optimization (SAC-FSO) algorithm of Wang et al. maps decision variables and detects violations at runtime for multi-link scheduling [18]. Mawlood and Mahmood combine SAC with elastic weight consolidation for continuous-action multi-queue scheduling [19]. These learning-based schedulers improve adaptability under dynamic traffic and transmission conditions. However, most of them still fold throughput, delay, and packet drop into a single scalar reward with hand-tuned weights. During burst periods, the instantaneous cost of a delay violation may be masked by later high-throughput returns. As a result, the learned policy may trade delay violations for higher rewards, so some high-priority packets can still exceed their delay bound.

Constrained reinforcement learning (CRL) can separate delay requirements from the reward and handle them as explicit constraints. For example, the worst-case soft actor-critic (WCSAC) characterizes the worst-case constraint tail through conditional value at risk (CVaR) [20]. CVaR-constrained policy optimization further extends this line of work [21]. The constrained reinforcement learning based resource allocation (CLARA) framework introduces constraints into network slicing through a projection layer [22]. The recurrent softmax delayed deep double deterministic policy gradient (RSD4) introduces constraints into delay-sensitive scheduling through Lagrangian duality [23]. Alvi et al. apply constrained SAC to edge computing under instantaneous latency constraints [24]. These works demonstrate the usefulness of explicit constraints in safety- or latency-critical decision-making. However, most existing CRL studies still focus on a single end-to-end latency constraint for one user or task class. This single-constraint focus differs from the bursty multi-queue scheduling and resource allocation problem considered in this paper.

In our setting, multiple queues share a fluctuating output link, and the high-priority queue has a stringent queuing delay violation requirement. The scheduler must allocate limited resource among competing queues while explicitly controlling the queuing delay violation of the high-priority queue. The above scalar-reward deep reinforcement learning (DRL) schedulers and existing single-constraint CRL methods are therefore not directly suited to this setting. Unlike existing scalar-reward and single-constraint approaches, our proposed approach leverages constrained SAC to support adaptive multi-queue scheduling and resource allocation under fluctuating link capacity, with explicit control of the high-priority queue’s delay violation budget.

3. System Model

We consider a heterogeneous queuing system with D queues sharing a common output link, as illustrated in Figure 1. Queue 0 follows an MMPP, whose state switches between a “normal” mode and a “burst” mode, each with a distinct arrival rate. The other queues follow independent Poisson arrival processes. Each queue has a finite buffer capacity B, measured in packets, and overflow packets are dropped.

The timeline is divided into time slots of duration $T_{\mathrm{ep}}$, with the tth slot spanning from time $t{T}_{\mathrm{ep}}$ to time $(t+1)T_{\mathrm{ep}}$ for $t=0,1,2,\dots$. In time slot t, the link capacity $K_t$ is defined as the maximum number of packets that can be transmitted on the output link. We assume that $K_t$ is drawn independently from the discrete uniform distribution on $\{K_{min},K_{min}+1,\dots ,K_{max}\}$, and is independent of the arrival processes. Within each time slot, when at least one queue is nonempty, the link capacity is allocated to backlogged queues according to scheduling weights determined by the agent. The joint scheduling and resource allocation objective is to maximize the system throughput utility—defined as the weighted sum of per-queue throughputs with priority-dependent weights—while respecting each queue’s delay requirement, where higher-priority queues are subject to tighter delay bounds. Queue i has higher priority than queue $(i+1)$, $i=0,1,\dots ,D-2$. For queue 0 carrying bursty traffic, the average queuing delay ${\delta }_{0,t}$ of packets departing in time slot t (considering only non-empty slots) is subject to a long-term violation constraint. Specifically, the queuing delay violation rate is defined as the fraction of non-empty time slots within a given observation window where ${\delta }_{0,t}$ exceeds the maximum allowable delay ${\Delta }_0$, and this fraction must remain below the tolerance ${\epsilon }_0$ (e.g., ${\epsilon }_0=0.01$). For queue i ($i>0$) carrying best-effort traffic, the average queuing delay ${\delta }_{i,t}$ of packets departing in time slot t (considering only non-empty slots) is expected to remain within the maximum allowable value ${\Delta }_i$; nevertheless, violations are tolerable under resource scarcity, provided that lower-priority queues are prevented from experiencing starvation. ${\delta }_{i,t}$ is defined by Equation (1).

$${\delta }_{i,t}=\left\{\begin{array}{cc}{\displaystyle {\displaystyle \frac{1}{|{\mathcal{D}}_{i,t}|}}\sum _{p\in {\mathcal{D}}_{i,t}}{\delta }_p,}\hfill & |{\mathcal{D}}_{i,t}|>0,\hfill \\ 0,\hfill & |{\mathcal{D}}_{i,t}|=0.\hfill \end{array}\right.$$

(1)

${\mathcal{D}}_{i,t}$ denotes the set of packets that have departed from queue i in time slot t. ${\delta }_p$ denotes the queuing time of packet p in ${\mathcal{D}}_{i,t}$.

Let $Q_{i,t}$ be the number of packets in queue i at the start of time slot t (i.e., at time $t{T}_{\mathrm{ep}}$). The events within time slot t occur in the following order.

1.

The agent observes system state ${\mathbf{s}}_t$, selects a continuous action ${\mathbf{a}}_t$ based on ${\mathbf{s}}_t$ and past actions, and maps ${\mathbf{a}}_t$ to integer transmission quotas $n_{i,t}$ for queue i, ${\sum }_{i=0}^{D-1}{n}_{i,t}\le K_t$, $0\le n_{i,t}\le Q_{i,t}$.

2.

The $n_{i,t}$ HoL packets in queue i are transmitted in first-in, first-out (FIFO) order.

3.

$A_{i,t}$ new packets arrive at queue i, and the length of queue i evolves as

$$Q_{i,t+1}=min\left\{Q_{i,t}-n_{i,t}+A_{i,t},B\right\},$$

(2)

where the min operator accounts for the dropping of overflow packets.

4.

The reward $r_t$ is obtained, along with the queuing delay violation cost $c_t$ of the highest-priority queue 0.

5.

The system state is updated for time slot $(t+1)$, denoted ${\mathbf{s}}_{t+1}$.

4. The Proposed Constrained Soft Actor-Critic Approach

This section presents the proposed approach for joint scheduling and resource allocation in heterogeneous queuing systems based on CSAC (abbreviated as CSAC-HQS), in six parts: basic architecture, state space, action space, reward and queuing delay violation cost, parameter update, and approach execution.

4.1. Basic Architecture

The CSAC-HQS approach comprises three main functional modules: the heterogeneous queuing system (as the environment), the actor-critic agent (consisting of an actor network, twin reward critics with their target networks, and a constraint critic with its target network), and an experience replay buffer. These modules interact with each other (see Figure 2) to perform experience collection and parameter update.

4.1.1. Experience Collection

In each time slot t, the actor ${\pi }_{\varphi }$ observes the system state ${\mathbf{s}}_t$ from the heterogeneous queuing system and outputs a continuous action ${\mathbf{a}}_t$. Through the masked-softmax mapping rule, ${\mathbf{a}}_t$ is converted into per-queue transmission weight, according to which the queuing system allocates executable integer transmission quotas among the backlogged queues. After execution, the queuing system returns the transition tuple $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,c_t,{\mathbf{s}}_{t+1},{\mathrm{done}}_t)$, which is stored as one experience record in the replay buffer. Here, ${\mathrm{done}}_t$ indicates whether the current state is a terminal state. The meanings of the remaining symbols are given in Section 3. This interaction cycle of observing, mapping, executing, and storing repeats at every time slot, so that experience is continuously collected to support the subsequent parameter update.

4.1.2. Parameter Update

Parameter updates are driven by mini-batches of transitions sampled from the replay buffer. At each update step, the twin reward critics $Q_{{\theta }_1}$, $Q_{{\theta }_2}$ are updated by minimizing the Bellman residual with respect to the reward target. These critics estimate the expected cumulative reward, which jointly accounts for the system throughput utility and the queuing delay violations of queues 1 to $(D-1)$. Meanwhile, the constraint critic $Q_{\psi }^{\mathrm{C}}$ is updated by minimizing the Bellman residual with respect to the constraint target. This critic estimates the expected queuing delay violation for queue 0 and enforces its stringent delay constraint. The actor ${\pi }_{\varphi }$ integrates the evaluations from both the reward critics and the constraint critic, and is updated by minimizing the policy loss, where the constraint critic’s contribution is weighted by the Lagrange multiplier $\lambda$. $\lambda$ is not a fixed hyperparameter but is dynamically adjusted and bounded at the end of each episode based on the total violation cost of queue 0 accumulated over the current episode, ensuring that the constraint penalty intensifies as violations increase while maintaining training stability.

Remark 1.

The interaction between the actor and the heterogeneous queuing system is modeled as a Markov Decision Process (MDP) with fixed-length episodes, each of which lasts for H time slots. At the end of every episode, the queuing system is reset. However, this reset is implemented as a time truncation rather than a true MDP termination. The bootstrap term of the parameter update process is weighted by $(1-\mathrm{done})$, where $\mathrm{done}$ is the terminal indicator of the underlying MDP. Since our MDP has no natural absorbing state, $\mathrm{done}$ remains 0 at the episode truncation boundary, so the bootstrap target is not forced to zero at the end of each episode.

4.2. State Space

In time slot t, the system state ${\mathbf{s}}_t$ is defined as

$${\mathbf{s}}_t=[{\mathbf{b}}_t,{\mathbf{v}}_t,{\mathbf{h}}_t].$$

(3)

The three components of ${\mathbf{s}}_t$ capture the queuing dynamics over the last ${\tau }_W$ time slots, providing the policy with sufficient temporal context to anticipate burst transients and capacity fluctuations. ${\mathbf{b}}_t$ is the queue occupancy history, ${\mathbf{v}}_t$ is the past violation feedback, and ${\mathbf{h}}_t$ is the link capacity history.

4.2.1. Queue Occupancy History

For each queue $i\in \{0,1,\dots ,D-1\}$ in time slot t, the normalized backlog over the last ${\tau }_W$ time slots (including the current slot) is aggregated as

$${\mathbf{b}}_t^{\left(i\right)}=\left[{\displaystyle \frac{Q_{i,t-{\tau }_W+1}}{B}},{\displaystyle \frac{Q_{i,t-{\tau }_W+2}}{B}},\dots ,{\displaystyle \frac{Q_{i,t}}{B}}\right]\in {[0,1]}^{{\tau }_W}.$$

(4)

The vectors ${\mathbf{b}}_t^{\left(i\right)}$ are concatenated as ${\mathbf{b}}_t=[{\mathbf{b}}_t^{\left(0\right)},{\mathbf{b}}_t^{\left(1\right)},\dots ,{\mathbf{b}}_t^{(D-1)}]\in {[0,1]}^{D·{\tau }_W}$. ${\mathbf{b}}_t$ allows the policy to identify which queues are accumulating backlog.

4.2.2. Past Violation Feedback

For each queue $i\in \{0,1,\dots ,D-1\}$ in time slot t, the pieces of past violation feedback are aggregated as

$${\mathbf{v}}_t^{\left(i\right)}=\left[{\widehat{c}}_{i,t-{\tau }_W},{\widehat{c}}_{i,t-{\tau }_W+1},\dots ,{\widehat{c}}_{i,t-1}\right]\in {\{0,1\}}^{{\tau }_W}.$$

(5)

${\widehat{c}}_{i,t^{\prime }}$ denotes the queuing delay violation cost of queue i in time slot $t^{\prime }$, and it is defined as

$${\widehat{c}}_{i,t^{\prime }}\triangleq \mathbb{1}\{{\overline{\delta }}_{i,t^{\prime }}>{\Delta }_i\},$$

(6)

where $t^{\prime }\in \{t-{\tau }_W,t-{\tau }_W+1,\dots ,t-1\}$.

$${\overline{\delta }}_{i,t}=\left\{\begin{array}{cc}{\displaystyle {\displaystyle \frac{1}{|{\mathcal{D}}_{i,t}|}}\sum _{p\in {\mathcal{D}}_{i,t}}{\delta }_p,}\hfill & |{\mathcal{D}}_{i,t}|>0,\hfill \\ W_{i,t}^{\mathrm{end}},\hfill & |{\mathcal{D}}_{i,t}|=0,Q_{i,t+1}>0,\hfill \\ 0,\hfill & |{\mathcal{D}}_{i,t}|=0,Q_{i,t+1}=0.\hfill \end{array}\right.$$

(7)

$W_{i,t}^{\mathrm{end}}$ is the HoL packet’s queuing time in queue i at the end of time slot t, given no departure occurs in that slot and $Q_{i,t+1}$ packets remain buffered. This prevents the scheduler from being able to “hide” the accumulated wait of a starved queue by simply not serving it. The vectors ${\mathbf{v}}_t^{\left(i\right)}$ are concatenated as ${\mathbf{v}}_t=[{\mathbf{v}}_t^{\left(0\right)},{\mathbf{v}}_t^{\left(1\right)},\dots ,{\mathbf{v}}_t^{(D-1)}]\in {\{0,1\}}^{D·{\tau }_W}$. ${\mathbf{v}}_t$ allows the policy to recognize that a queue has recently crossed its delay requirement, providing the policy with explicit feedback on past violation events independent of the current backlog level.

4.2.3. Link Capacity History

The link capacities over the last ${\tau }_W$ time slots (including the current slot) are aggregated as:

$${\mathbf{h}}_t=\left[{\displaystyle \frac{K_{t-{\tau }_W+1}}{K_{max}}},{\displaystyle \frac{K_{t-{\tau }_W+2}}{K_{max}}},\dots ,{\displaystyle \frac{K_t}{K_{max}}}\right]\in {[0,1]}^{{\tau }_W}.$$

(8)

$K_{t^{\prime }}$ denotes the link capacity in time slot $t^{\prime }$, $t^{\prime }\in \{t-{\tau }_W+1,t-{\tau }_W+2,\dots ,t\}$. $K_{max}$ is the upper bound of the link capacity defined in Section 3. ${\mathbf{h}}_t$ captures the recent fluctuation of the shared output link and helps the policy adapt to capacity variations.

4.3. Action Space

A continuous action space is designed to implement fine-grained, adaptive link capacity allocation in two stages.

Stage 1: From continuous logits to normalized allocation weights. In each time slot t, the actor network outputs a vector ${\mathbf{u}}_t\in {\mathbb{R}}^D$, using the reparameterization trick

$${\mathbf{u}}_t={\mu }_{\varphi }\left({\mathbf{s}}_t\right)+{\sigma }_{\varphi }\left({\mathbf{s}}_t\right)\odot {ϵ}_t,{ϵ}_t\sim \mathcal{N}(\mathbf{0},\mathbf{I}),$$

(9)

where ⊙ denotes the Hadamard (element-wise) product, and $\mathbf{0}$, $\mathbf{I}$ are the zero vector and the identity matrix of appropriate dimensions. ${\mathbf{u}}_t$ is squashed to yield the action vector ${\mathbf{a}}_t=tanh\left({\mathbf{u}}_t\right)\in {(-1,1)}^D$, composed of continuous actions. The action vector ${\mathbf{a}}_t$ is then sharpened by the sharpening factor $\beta >1$ to form the softmax logit

$${\mathbf{z}}_t=\beta {\mathbf{a}}_t\in {(-\beta ,\beta )}^D.$$

(10)

Then ${\mathbf{z}}_t$ is normalized to an allocation weight vector ${\mathbf{w}}_t=(w_{0,t},w_{1,t},\dots ,w_{D-1,t})$ on the simplex through a masked softmax:

$$w_{i,t}={\displaystyle \frac{m_{i,t}exp\left(z_{i,t}\right)}{{\sum }_{j=0}^{D-1}{m}_{j,t}exp\left(z_{j,t}\right)}},i\in \{0,1,\dots ,D-1\},$$

(11)

where $z_{i,t}$ is the ith component of ${\mathbf{z}}_t$, and $m_{i,t}=\mathbb{1}\{Q_{i,t}>0\}$ is an empty-queue mask. If all queues are empty, $w_{i,t}=0$ for any i, namely, no transmission occurs in this time slot. The sharpening factor $\beta$ enlarges the effective ratio between the largest and smallest weights and therefore allows the scheduler to rapidly prioritize the highest-priority queue 0 during transient overload.

Stage 2: From normalized allocation weights to integer transmission quotas. In each time slot t, given the allocated capacity ${\overline{K}}_t=min\left\{K_t,{\sum }_{i=0}^{D-1}{Q}_{i,t}\right\}$, the allocation weight vector ${\mathbf{w}}_t$ is converted into an integer transmission quota vector ${\mathbf{n}}_t=(n_{0,t},n_{1,t},\dots ,n_{D-1,t})$ as follows.

1.

Compute the ideal allocation ${\tilde{n}}_{i,t}=w_{i,t}{\overline{K}}_t$ for $i=0,1,\dots ,D-1$.

2.

Floor to $n_{i,t}^{\left(0\right)}=\lfloor {\tilde{n}}_{i,t}\rfloor$ for $i=0,1,\dots ,D-1$.

3.

The residual capacity ${\rho }_t={\overline{K}}_t-{\sum }_i{n}_{i,t}^{\left(0\right)}$ is distributed to ${\rho }_t$ queues with the largest remainders $\left({\tilde{n}}_{i,t}-n_{i,t}^{\left(0\right)}\right)$, each receiving one extra unit. For queues with the same remainder, allocation is performed in order of priority from highest to lowest.

The above basic steps do not yet account for the per-queue backlog upper bound. If any queue i would receive more than its backlog (i.e., $n_{i,t}^{\left(0\right)}>Q_{i,t}$), the surplus is handled recursively: (a) clip the quota of that queue to $n_{i,t}\leftarrow Q_{i,t}$ and reclaim the surplus into the residual capacity; (b) redistribute the residual capacity, using the same largest-remainder rule, among the remaining nonempty queues that have not yet reached their backlog limit; (c) repeat until every queue’s quota does not exceed its backlog or until the residual capacity is zero. The transmission quota ultimately obtained by queue i is $n_{i,t}$. The quotas satisfy

$$0\le n_{i,t}\le Q_{i,t},{\displaystyle \sum _{i=0}^{D-1}}{n}_{i,t}={\overline{K}}_t.$$

(12)

So the masked-softmax mapping always produces executable integer transmission quotas. If ${\sum }_{i=0}^{D-1}{Q}_{i,t}\le K_t$, the allocation result of the above process is $n_{i,t}=Q_{i,t}$.

4.4. Reward and Queuing Delay Violation Cost

4.4.1. Throughput-Oriented Reward

The reward encourages useful transmission across all queues, penalizes queuing delay violations for queues 1 to $D-1$, and discourages packet drops in all queues. At time slot t, the reward is defined as

$$r_t={\alpha }_{\mathrm{scale}}{\displaystyle \sum _{i=0}^{D-1}}{\kappa }_i\left[C_1{n}_{i,t}-C_2{\chi }_{i}min\{v_{i,t},k_0{\Delta }_i\}-C_3{\xi }_{i,t}\right].$$

(13)

${\alpha }_{\mathrm{scale}}$ is a reward-scaling coefficient. ${\kappa }_i$ is the priority weight of queue i. $C_1$, $C_2$, $C_3$, and $k_0$ are throughput utility coefficient, queuing delay violation penalty coefficient, drop penalty coefficient, and queuing delay violation penalty clip multiplier. ${\chi }_i$ is the queuing delay violation penalty indicator of queue i, where ${\chi }_0=0$ and ${\chi }_i=1$ for $i=1,2,\dots ,D-1$. $v_{i,t}$ denotes delay violation magnitude, and is calculated as

$$v_{i,t}=max\{0,{\overline{\delta }}_{i,t}-{\Delta }_i\}.$$

(14)

${\xi }_{i,t}$ is the number of packets that are dropped, and it is calculated as

$${\xi }_{i,t}=max\{0,Q_{i,t}-n_{i,t}+A_{i,t}-B\}.$$

(15)

4.4.2. Queuing Delay Violation Cost for Highest-Priority Queue 0

The highest-priority queue 0 carries bursty traffic with stringent delay requirements. To enforce its delay constraint, we treat its queuing delay violation cost ${\widehat{c}}_{0,t}$ in time slot t as a constraint signal. For brevity, ${\widehat{c}}_{0,t}$ is abbreviated as $c_t$. The policy is then required to satisfy $J_C^{\mathrm{ep}}={\sum }_{t=0}^{H-1}{c}_t\le d_c$, where $d_c$ is a predefined violation budget.

4.5. Parameter Update

At each update step, a mini-batch of transitions $(\mathbf{s},\mathbf{a},r,c,{\mathbf{s}}^{\prime },\mathrm{done})$ is sampled from the replay buffer $\mathcal{B}$. The per-step updates are performed as follows.

Reward Critic Update. Each reward critic minimizes the squared Bellman residual,

$${\mathcal{L}}_Q\left({\theta }_i\right)={\mathbb{E}}_{(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime })\sim \mathcal{B}}\left[{\left(Q_{{\theta }_i}(\mathbf{s},\mathbf{a})-y^R\right)}^2\right],i\in \{1,2\},$$

(16)

with target value

$$y^R=r+\gamma (1-\mathrm{done})\left[\underset{i=1,2}{min}{Q}_{{\overline{\theta }}_i}({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime })-\alpha log{\pi }_{\varphi }({\mathbf{a}}^{\prime }\mid {\mathbf{s}}^{\prime })\right].$$

(17)

Here, $\gamma \in (0,1)$ is the discount factor. ${\mathbf{a}}^{\prime }\sim {\pi }_{\varphi }(·\mid {\mathbf{s}}^{\prime })$ is the next action sampled from the current policy. $\alpha$ is the entropy temperature.

Constraint Critic Update. The constraint critic minimizes the squared Bellman residual,

$${\mathcal{L}}_{Q^C}\left(\psi \right)={\mathbb{E}}_{(\mathbf{s},\mathbf{a},c,{\mathbf{s}}^{\prime })\sim \mathcal{B}}\left[{\left(Q_{\psi }^{\mathrm{C}}(\mathbf{s},\mathbf{a})-y^C\right)}^2\right],$$

(18)

with target value

$$y^C=c+\gamma (1-\mathrm{done})Q_{\overline{\psi }}^{\mathrm{C}}({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime }).$$

(19)

Here, ${\mathbf{a}}^{\prime }$ is the same next action sampled from ${\pi }_{\varphi }(·\mid {\mathbf{s}}^{\prime })$ used in Equation (17). Crucially, we deliberately omit the entropy bonus in the constraint target, ensuring that $Q_{\psi }^{\mathrm{C}}$ remains a pure estimate of the long-term cumulative violation risk, rather than a maximum-entropy value.

Actor Update. The actor is updated by minimizing the Lagrangian SAC loss,

$${\mathcal{L}}_{\pi }\left(\varphi \right)={\mathbb{E}}_{\mathbf{s}\sim \mathcal{B},\tilde{\mathbf{a}}\sim {\pi }_{\varphi }}\left[\alpha log{\pi }_{\varphi }\left(\tilde{\mathbf{a}}\right|\mathbf{s})+\lambda Q_{\psi }^{\mathrm{C}}(\mathbf{s},\tilde{\mathbf{a}})-\underset{i=1,2}{min}{Q}_{{\theta }_i}(\mathbf{s},\tilde{\mathbf{a}})\right].$$

(20)

The first term encourages exploration. The second—weighted by the Lagrange multiplier $\lambda$—steers the policy away from actions predicted to increase the queuing delay violation cost of the highest-priority queue 0. The third pushes the policy toward high-reward actions. $\tilde{\mathbf{a}}$ is the output of the actor network ${\pi }_{\varphi }$ with input $\mathbf{s}$, i.e., $\tilde{\mathbf{a}}\sim {\pi }_{\varphi }(·\mid \mathbf{s})$.

The Lagrange multiplier $\lambda$ is updated as Equation (21). To prevent unbounded growth and improve stability, $\lambda$ is projected onto $[0,{\lambda }_{max}]$ and updated separately at episode boundaries.

$$\lambda \leftarrow {\Pi }_{[0,{\lambda }_{max}]}\left(\lambda +{\alpha }_{\lambda }\left(J_C^{\mathrm{ep}}-d_c\right)\right),$$

(21)

where ${\Pi }_{[0,{\lambda }_{max}]}(·)$ is defined as Equation (22), ${\alpha }_{\lambda }$ is the multiplier learning rate, and ${\lambda }_{max}$ is the upper bound of the multiplier.

$$\begin{array}{c}\hfill {\Pi }_{[0,{\lambda }_{max}]}\left(x\right)=\left\{\begin{array}{cc}0,\hfill & x<0,\hfill \\ x,\hfill & 0\le x\le {\lambda }_{max},\hfill \\ {\lambda }_{max},\hfill & x>{\lambda }_{max}.\hfill \end{array}\right.\end{array}$$

(22)

Entropy Temperature Update. The entropy temperature $\alpha$ is auto-tuned to track a target entropy $\overline{\mathcal{H}}=-dim\left(\mathcal{A}\right)$, as shown in Equation (23), where $dim\left(\mathcal{A}\right)$ denotes the dimension of the action space $\mathcal{A}$ and equals D.

$$\alpha \leftarrow \alpha -{\alpha }_{\alpha }{\nabla }_{\alpha }{\mathbb{E}}_{\mathbf{s}\sim \mathcal{B},\tilde{\mathbf{a}}\sim {\pi }_{\varphi }}\left[-\alpha \left(log{\pi }_{\varphi }\left(\tilde{\mathbf{a}}\right|\mathbf{s})+\overline{\mathcal{H}}\right)\right],$$

(23)

where ${\alpha }_{\alpha }$ is the temperature learning rate. This auto-tuning eliminates the need to manually search for an entropy temperature that suits the action space dimensionality.

Target Network Update. At each gradient step, the target critic parameters track the online parameters via Polyak averaging:

$$\begin{array}{cc}\hfill & {\overline{\theta }}_i\leftarrow \tau {\theta }_i+(1-\tau ){\overline{\theta }}_i,i\in \{1,2\},\hfill \\ \hfill & \overline{\psi }\leftarrow \tau \psi +(1-\tau )\overline{\psi },\hfill \end{array}$$

(24)

where $\tau \in (0,1)$ is a soft-update coefficient. This mechanism keeps bootstrap targets stable while still tracking the learning progress of the online networks.

Remark 2.

The actor, critics (including the constraint critic), and entropy temperature are updated at every gradient step on the inner timescale, whereas λ is updated only at episode boundaries (i.e., every H time slots). This two-timescale separation aligns the constraint signal that drives λ with the form in which $d_c$ is declared (a per-episode budget).

4.6. Approach Execution

The execution process of CSAC-HQS is formally summarized in Algorithm 1.

Algorithm 1 CSAC-HQS

1: Initialization: actor ${\pi }_{\varphi }$, reward critics $Q_{{\theta }_1},Q_{{\theta }_2}$, constraint critic $Q_{\psi }^{\mathrm{C}}$; target networks ${\overline{\theta }}_1\leftarrow {\theta }_1$, ${\overline{\theta }}_2\leftarrow {\theta }_2$, $\overline{\psi }\leftarrow \psi$; replay buffer $\mathcal{B}\leftarrow \varnothing$; Lagrange multiplier $\lambda \leftarrow 0$; entropy temperature $\alpha \leftarrow 0.2$; step counter $k\leftarrow 0$ 2: Parameters: episode length H, update interval G, mini-batch size N, soft-update coefficient $\tau$, discount factor $\gamma$, per-episode queuing delay violation budget $d_c$, sharpening factor $\beta$, warm-up steps $T_{\mathrm{warm}}$, Lagrange multiplier learning rate ${\alpha }_{\lambda }$, Lagrange multiplier upper bound ${\lambda }_{max}$, temperature learning rate ${\alpha }_{\alpha }$, target entropy $\overline{\mathcal{H}}=-dim\left(\mathcal{A}\right)$ 3: for each episode do 4:    Reset environment to obtain ${\mathbf{s}}_0$; set $J_C^{\mathrm{ep}}\leftarrow 0$ 5:    for each time slot $t=0,1,\dots ,H-1$ within the current episode do 6:      Observe current state ${\mathbf{s}}_t=[{\mathbf{b}}_t,{\mathbf{v}}_t,{\mathbf{h}}_t]$ 7:      Sample ${\mathbf{u}}_t={\mu }_{\varphi }\left({\mathbf{s}}_t\right)+{\sigma }_{\varphi }\left({\mathbf{s}}_t\right)\odot {ϵ}_t$, ${ϵ}_t\sim \mathcal{N}(\mathbf{0},\mathbf{I})$; ${\mathbf{a}}_t=tanh\left({\mathbf{u}}_t\right)$; ${\mathbf{z}}_t=\beta {\mathbf{a}}_t$ 8:      Apply masked softmax to ${\mathbf{z}}_t$ to obtain allocation weight vector ${\mathbf{w}}_t$ as Stage 1 in Section 4.3 9:      The weight vector ${\mathbf{w}}_t$ is converted into integer transmission quota vector ${\mathbf{n}}_t$ as Stage 2 in Section 4.3 10:      Execute transmission according to ${\mathbf{n}}_t$; observe $r_t$, $c_t$, ${\mathbf{s}}_{t+1}$, ${\mathrm{done}}_t$ 11:      Store $({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,c_t,{\mathbf{s}}_{t+1},{\mathrm{done}}_t)$ in $\mathcal{B}$ 12:      $J_C^{\mathrm{ep}}\leftarrow J_C^{\mathrm{ep}}+c_t$; $k\leftarrow k+1$ 13:      if $\left|\mathcal{B}\right|\ge N$; and; $k\ge T_{\mathrm{warm}}$; and; $kmodG=0$ then 14:         Sample a random mini-batch of N transitions from $\mathcal{B}$ 15:         Update reward critics $Q_{{\theta }_1},Q_{{\theta }_2}$ by minimizing ${\mathcal{L}}_Q\left({\theta }_i\right)$ as Equations (16) and (17) 16:         Update constraint critic $Q_{\psi }^{\mathrm{C}}$ by minimizing ${\mathcal{L}}_{Q^C}\left(\psi \right)$ as Equations (18) and (19) 17:         Update actor ${\pi }_{\varphi }$ by minimizing ${\mathcal{L}}_{\pi }\left(\varphi \right)$ as Equation (20) 18:         Update entropy temperature $\alpha$ as Equation (23) 19:         Soft-update targets as Equation (24) 20:      end if 21:    end for 22:    Update the Lagrange multiplier $\lambda$ as Equation (21) 23: end for

5. Evaluation

This section evaluates the proposed CSAC-HQS via experiments, using the parameters summarized in Table 1 and the following baseline methods.

1) Static: As a simple heuristic baseline, we employ a round-robin-like policy with fixed proportional weights. The raw weights for queues 0, 1, and 2 are set to 1,2,3, respectively, which are normalized by their sum to obtain the transmission shares $(1/6,2/6,3/6)$. To make these shares physically realizable, we apply the integer quota mapping described in Section 4.3, which translates the shares into per-slot transmission counts while strictly adhering to the capacity and backlog constraints.

2) Dynamic: As another simplified heuristic baseline, this policy allocates resources proportionally to instantaneous queue lengths, i.e., $w_i\propto Q_{i,t}$. Specifically, the raw weights are normalized by their sum to obtain the effective transmission shares $w_i=Q_{i,t}/{\sum }_j{Q}_{j,t}$. The integer quota mapping from Section 4.3 is subsequently applied to translate these shares into feasible per-slot transmission counts.

3) SAC: As a learning-based baseline without explicit delay constraints, we adopt the standard SAC [25] with the same state space, action space, network architecture, and training settings as CSAC-HQS, but omit the constraint critic and the Lagrangian multiplier (i.e., $\lambda =0$). For action generation, it employs the basic masked-softmax mapping, corresponding to a fixed sharpening factor $\beta =1$. Since no explicit constraint is enforced, the queuing delay violation penalties of all queues are incorporated into the reward as a linearly scalarized combination of throughput (positive reward) and packet-loss (negative penalty terms), with hand-tuned coefficients. This formulation aligns with recent SAC-based multi-queue schedulers, e.g., the WFQ continual-DRL framework of Mawlood and Mahmood [19]. We summarize the key distinctions between SAC and CSAC-HQS in Table 2.

All experiments are implemented in Python using a Gymnasium-style environment interface and PyTorch [26]. For non-learning baselines (Static and Dynamic)—which are deterministic—each is evaluated over 100 test episodes, and we report the average performance across these episodes. The performance per episode is quantified by the following metrics: (i) the queuing delay violation rate for queue 0 (which carries bursty traffic with a stringent delay constraint), ${\displaystyle \frac{1}{H}}{\sum }_{t=0}^{H-1}\mathbb{1}\{{\delta }_{0,t}>{\Delta }_0\}$; (ii) the average queuing delay for each best-effort queue (queues $i=1,2$), ${\displaystyle \frac{1}{H}}{\sum }_{t=0}^{H-1}{\delta }_{i,t}$; (iii) the throughput (packets per time slot) for each queue; and (iv) the per-episode total overflow packet count for each queue. For each learning-based method—including SAC, CSAC-HQS ($\beta =1$), and CSAC-HQS ($\beta =5$)—we perform 5 independent training runs with different random seeds, yielding 5 trained policies. For each of these policies, we compute four average performance metrics over 100 test episodes following the same procedure described above. Then we report the mean and standard deviation for each of these four average metrics across the 5 policies.

5.1. Queuing Delay Violation Rate for Queue 0

For queue 0, the maximum allowable queuing delay violation rate is set to 1.0%, corresponding to a per-episode delay budget of $d_c=2$ time slots over an episode length of $H=200$ (i.e., $d_c/H$). As shown in Figure 3, the violation rates for the Static and Dynamic baselines are 18.38% and 34.28%, respectively, both far exceeding this budget line. The Dynamic method exhibits a higher violation rate than the Static method. This is because the Dynamic method allocates resource in proportion to instantaneous backlog and therefore reacts to accumulated queue length rather than the imminent deadline risk of packets; this strategy frequently causes queue 0 to exceed ${\Delta }_0$. Among the three learning-based methods, CSAC-HQS ($\beta =1$) and CSAC-HQS ($\beta =5$) achieve violation rates of $0.91\%\pm 1.30\%$ and $0.05\%\pm 0.09\%$, respectively, while the unconstrained SAC reaches $7.87\%\pm 5.98\%$. In terms of mean values, both CSAC-HQS variants fall below the 1.0% budget line. However, when requiring that the upper bound of the standard deviation also remains under the threshold, only CSAC-HQS ($\beta =5$) qualifies (i.e., mean + std < 1.0%). This underscores the effectiveness of action sharpening in not only enforcing the delay constraint but also stabilizing performance across different random seeds.

5.2. Average Queuing Delay per Best-Effort Queue

As shown in Figure 4, for the Static method, the average queuing delay of queue 1 exceeds the 20 ms threshold, while queue 2 falls below the 30 ms threshold. Dynamic, SAC, and the two CSAC-HQS variants maintain the average queuing delays of queue 1 and 2 within their respective thresholds. Although the Dynamic method achieves the lowest average queuing delays on both queues, this comes at the cost of a higher queuing delay violation rate for queue 0, see Figure 3. Among the learning-based methods, the unconstrained SAC yields the highest average queuing delay on both best-effort queues ($12.4\pm 3.1$ ms on queue 1 and $18.9\pm 4.1$ ms on queue 2); the two CSAC-HQS variants attain lower average queuing delays than SAC, with CSAC-HQS ($\beta =5$) achieving the lowest. Overall, these results demonstrate that the standalone constraint mechanism, together with action sharpening, effectively enforces the delay requirement of queue 0 without starving the best-effort queues.

5.3. Throughput and Overflow Packet Count per Queue

As shown in Figure 5, all learning-based methods deliver nearly identical throughputs for queues 0–2. For instance, CSAC-HQS ($\beta =5$) achieves throughputs of $37\pm 0$, $148\pm 0$, and $196\pm 0$ packets/time slot on queues 0, 1, and 2, respectively, which are comparable to the best throughput performance observed among the baselines; SAC and CSAC-HQS ($\beta =1$) attain closely matching throughputs, with $192\pm 5$ and $195\pm 1$ packets/time slot on queue 2, respectively.

The overflow packet count results further differentiate these methods. Under its fixed transmission rate, the Static method keeps queue 1’s buffer near capacity and incurs a severe overflow loss, with 15.61 packets/time slot dropped on queue 1. The learning-based methods differ mainly on queue 2. The unconstrained SAC incurs an overflow packet count of $4.31\pm 4.16$ packets/time slot on queue 2; CSAC-HQS ($\beta =1$) reduces this to $1.98\pm 0.54$, and CSAC-HQS ($\beta =5$) further lowers it to $1.59\pm 0.18$. Although the Dynamic method achieves an even lower overflow count on queue 2 (0.48 packets/time slot), this comes at the cost of the worst violation rate for queue 0 (cf. Figure 3). Moreover, SAC’s standard deviation on queue 2 ($\pm 4.16$) is substantially larger than those of the two constrained variants ($\pm 0.54$ and $\pm 0.18$), further confirming the enhanced cross-seed stability provided by the standalone constraint mechanism and action sharpening.

6. Conclusions

This paper proposes a CSAC approach for joint scheduling and resource allocation in a heterogeneous queuing system, aiming to maximize throughput utility subject to per-queue delay constraints. The system consists of multiple priority queues sharing a single output link: one high-priority queue carries bursty traffic with a stringent delay constraint, while the remaining lower-priority queues are best-effort with distinct delay requirements. Compared with conventional heuristics and an unconstrained SAC baseline, the proposed approach effectively satisfies the delay constraint of the highest-priority queue, while achieving comparable or superior performance in terms of throughput, lower-priority queuing delays, and overflow packet count. These improvements stem from two key components: (i) decoupling the delay constraint from the reward, which we treat as a standalone violation budget, and (ii) a two-stage mapping mechanism that transforms continuous policy logits into integer transmission quotas. Together, these components ensure stringent QoS guarantees without sacrificing best-effort performance.

In future work, we plan to extend our approach to more complex scenarios involving multiple priority queues, each carrying bursty traffic with distinct delay constraints, and to further investigate joint scheduling, resource allocation, and admission control in such settings.