Joint Scheduling and Resource Allocation in Heterogeneous Queuing Systems with Bursty Traffic: A Constrained Soft Actor-Critic Approach

Joint scheduling and resource allocation are investigated in heterogeneous queuing systems—where multiple queues have distinct priorities and share a single output link—with the aim of maximizing throughput utility under per-queue delay constraints. The primary challenge arises from stochastic packet arrivals and the time-varying capacity of the shared link. To address this challenge, conventional heuristic methods and unconstrained learning-based methods exhibit inherent limitations, particularly when handling bursty traffic with stringent delay requirements. The former lack real-time reactivity to system dynamics; the latter, which encode delay constraints into the reward, may sacrifice deadline compliance for higher cumulative rewards. To this end, this paper proposes a constrained soft actor-critic (CSAC) approach with two key designs. First, we decouple the delay constraint of the queue with a stringent delay requirement from the reward, treating it as a standalone violation budget. Second, we introduce a two-stage mapping mechanism that transforms continuous policy logits into integer transmission quotas. We compare the proposed approach with an unconstrained soft actor-critic (SAC) baseline and several heuristic baselines under heterogeneous traffic conditions. The experimental setup comprises one Markov-modulated Poisson process (MMPP) highest-priority queue with a maximum allowable queuing delay violation rate of 1%, along with two independent Poisson process lower-priority queues with distinct priorities, each with a best-effort queuing delay requirement. Results demonstrate that the proposed approach reduces the highest-priority queue’s delay violation rate to 0.05%±0.09%, compared to 7.87%±5.98% for unconstrained SAC and 18.38% and 34.28% for the two heuristic baselines, while maintaining comparable throughput, lower-priority delays, and overflow packet counts.