Meta-MARL-ESA: Toward Zero Shot Generalization for Energy Efficient Sleep Scheduling in Dynamic UAV Assisted 6G HetNets

1. Introduction

The sixth generation (6G) of wireless networks envisions a seamless convergence of terrestrial and non terrestrial infrastructures, integrating satellites, high altitude platforms, and unmanned aerial vehicles (UAVs) into a unified connectivity fabric known as Non Terrestrial Networks (NTNs) [1]. This paradigm promises ubiquitous coverage, extreme reliability, and on demand capacity scaling ambitions that standardization bodies such as 3GPP are actively translating into concrete technical frameworks [2]. Among the enabling technologies, UAVs operating as aerial base stations stand out for their deployment agility: they can be rapidly dispatched to disaster zones, temporarily boost capacity at crowded events, or fill coverage holes in remote or underserved regions [3].

Yet this agility comes at a steep price. Rotary wing UAVs consume substantial propulsion power, typically 300–500 watts, during sustained hovering, which often dominates their total energy budget and severely limits mission endurance [4]. As 6G networks aspire to support massive connectivity with minimal carbon footprint, energy efficiency (EE) has emerged as a first class design objective, alongside spectral efficiency (SE) and latency.

1.1. Related Work

A proven strategy for improving EE in dense HetNets is dynamic sleep scheduling: simply deactivating network nodes during periods of low traffic demand [5]. While effective in static deployments, this approach becomes considerably more challenging in UAV assisted environments where topologies change frequently—UAVs reposition, user demand fluctuates, and ad hoc deployments form and dissolve in response to operational needs. The problem is inherently multi-agent, requiring decentralized coordination under partial observability, making it a natural fit for Multi-Agent Reinforcement Learning (MARL). Reinforcement Learning (RL), and particularly MARL, has shown remarkable success in learning complex coordination policies for network control problems. Prior work has explored various approaches to energy efficient network management.

1.2. Gap Analysis

Despite the advances summarized in Table 1, existing methods either lack generalization capability entirely or address only isolated aspects of the problem (e.g cannot adapt to traffic variations), leaving a clear gap for the framework proposed in this work (Generalizes to task distribution). No existing work simultaneously addresses the three core challenges of UAV-assisted sleep scheduling: (i) multi-agent coordination under partial observability, (ii) topology-agnostic policy learning, and (iii) rapid adaptation to dynamic network changes. Conventional MARL approaches (e.g., MARL-ESA) achieve (i) but fail at (ii) and (iii). GNN-based methods partially address (ii) but lack meta-learning for adaptation. Meta-learning approaches address (iii) but are typically single-agent. Our work uniquely bridges these gaps through the synergistic combination of GNNs and MAML within a Dec-POMDP framework.

1.3. Contribution

The work presented in this paper bridges this gap through a two stage contribution: As a first contribution, we develop MARL-ESA, a MARL framework for energy efficient sleep scheduling in static UAV assisted HetNets. MARL-ESA employs centralized training with decentralized execution (CTDE) and achieves substantial energy savings while maintaining quality of service (QoS). However, its policies are intrinsically tied to the specific network topology on which they were trained. Each new configuration—different UAV positions, number of nodes, or traffic patterns—demands complete retraining from scratch, requiring over 1500 episodes and several hours of simulation time. In dynamic 6G NTN environments, this retraining bottleneck renders conventional MARL approaches impractical. To overcome this limitation, we propose Meta-MARL-ESA, a significant evolution that injects generalization capability at its core. Our framework shifts the learning objective from optimizing a single topology to maximizing expected performance across a distribution of topological tasks $\tau \sim p\left(\mathcal{T}\right)$. By synergistically combining a permutation invariant Graph Neural Network (GNN) policy with Model Agnostic Meta Learning (MAML), Meta-MARL-ESA learns invariant control principles that transfer across network configurations without retraining. The resulting metapolicy enables two modes of operation: (i) zero shot deployment, where it performs competently on unseen topologies immediately; and (ii) few shot fine tuning, where it achieves near optimal performance after only a handful of gradient steps using limited online data.

Extensive simulations validate our approach. Meta-MARL-ESA achieves 12× faster convergence on unseen tasks compared to training from scratch, preserves 82% of optimal performance in zero shot deployment, and improves energy efficiency by 21.5% over converged baselines on out of distribution scenarios. Moreover, during dynamic UAV repositioning events, the framework recovers to 98% of optimal performance within 8 decision epochs, demonstrating its practical viability for real time 6G NTN operation. The remainder of this paper is organized as follows. Section 2 describes the proposed framework. Section 3 presents the dynamic system model and generalized problem formulation. Section 4 details the MARL-ESA framework for static scenarios. Section 5 introduces the Meta-MARL-ESA framework for generalization. Section 6 provides comprehensive simulation results and analysis. Finally, Section 7 concludes the paper and discusses future directions.

2. Proposed Framework: Integrating MARL, GNNs, and Meta-Learning

The proposed Meta-MARL-ESA framework builds upon three foundational pillars: MARL for decentralized coordination, GNNs for topology aware representation, and Meta-Learning for cross task generalization. This section reviews each pillar, culminating in the justification for their synergistic integration.

2.1. MARL for Network Energy Efficiency

MARL has emerged as a powerful paradigm for optimizing resource allocation in wireless networks, particularly in scenarios requiring decentralized coordination under partial observability. Early approaches relied on independent Q-learning or fully centralized critics to coordinate cell sleep modes in terrestrial HetNets. The CTDE paradigm, exemplified by Multi-Agent Deep Deterministic Policy Gradient (MADDPG) [16], became the de facto standard for managing the non stationarity inherent in multi-agent environments. CTDE enables agents to learn coordinated policies during training while executing autonomously during deployment—a property essential for practical network operation (Figure 1).

In UAV assisted networks, MARL has been applied to trajectory optimization, resource allocation, and user association [8,18]. However, these approaches share a critical limitation: they assume a fixed network topology during training and deployment. The policies they produce are inherently specialized to the specific number and arrangement of nodes, traffic patterns, and environmental conditions encountered during training. When the topology changes—as it inevitably does in UAV assisted networks—these policies degrade significantly, and retraining from scratch becomes necessary.

Our baseline, MARL-ESA, addresses this limitation by introducing a tailored framework for sleep scheduling that explicitly accounts for UAV specific energy characteristics, including hovering power and mobility constraints. MARL-ESA achieves substantial energy savings in static scenarios, but it remains fundamentally tied to the topology on which it was trained. This realization leads us to a critical question: how can we train a policy that generalizes across topologies, rather than memorizing a single one?

2.2. Generalization in RL and Meta-Learning

The challenge of generalization—the ability of an RL agent to perform well in environments not encountered during training—is one of the most persistent obstacles in RL. Common strategies include training across diverse environments with domain randomization [19] or leveraging recurrent policies to encode environmental history. While these approaches improve robustness, they do not guarantee performance on truly unseen configurations.

Meta Reinforcement Learning (Meta-RL) offers a more principled solution. Instead of learning a policy for a single task, Meta RL learns how to learn, it optimizes for rapid adaptation across a distribution of tasks. Algorithms such as MAML [20], Reptile [21], and context based methods [22] aim to identify parameter initializations that are sensitive to task variations, such that a few gradient steps suffice to produce a specialized policy. This capability is precisely what our problem demands: we need a policy that can adapt quickly to new network configurations without requiring thousands of retraining episodes.

Recent work has demonstrated the potential of Meta RL in wireless communications. For instance, [23] applied Meta RL to enhance reliability in dynamic vehicular networks, and [17] proposed a few shot meta offline RL algorithm for resilient UAV trajectory planning. However, these frameworks are limited to single agent scenarios or trajectory optimization and do not address the multi-agent coordination required for network wide sleep scheduling. This gap motivates our integration of Meta RL with MARL.

2.3. Graph Neural Networks for Topology-Aware Representation

To enable generalization across topologies, the policy must process network configurations of arbitrary size and structure. Traditional Multi Layer Perceptrons (MLPs) assume fixed input dimensions, making them ill suited for dynamic networks. GNNs overcome this limitation through their inherent permutation invariance and ability to handle relational data. In a wireless network, nodes (base stations, UAVs) and edges (communication links) form a natural graph, making GNNs an ideal architectural choice.

Recent works have successfully applied GNNs to power allocation [24], link scheduling [25], and resource slicing [26], demonstrating that GNNs can generalize to larger networks than those seen during training. A comprehensive survey [27] highlights the scalability and generalization properties of GNNs in radio resource management.

2.4. Synthesis: Why MARL + GNN + Meta-Learning?

Our work is positioned at the intersection of these three advancing fields. The justification for their integration follows a clear logical chain:

1.

MARL provides the framework for decentralized multi-agent coordination, essential for sleep scheduling across multiple nodes.

2.

GNNs provide the architectural foundation for processing arbitrary network topologies, enabling permutation invariant representations that naturally generalize across different node counts and connectivity patterns.

3.

Meta Learning provides the algorithmic mechanism for rapid adaptation, enabling the policy to quickly adjust to new configurations without retraining from scratch.

Together, these components form Meta-MARL-ESA: a framework that learns how to schedule sleep across a distribution of tasks $\tau \sim p\left(\mathcal{T}\right)$, rather than memorizing a single topology. To the best of our knowledge, this is the first framework to combine GNNs with Meta RL specifically for the problem of cross topology generalization in dynamic UAV HetNets, creating a policy that is both architecturally and algorithmically geared for transfer.

3. System Model and Problem Formulation

Throughout this paper, we employ a consistent set of abbreviations and mathematical notations, summarized in Table 2 for quick reference. This nomenclature underpins the description of the dynamic UAV assisted HetNet topology, energy consumption model, and the subsequent Dec-POMDP formulation presented in the following subsections.

3.1. Dynamic UAV Assisted HetNet Topology Model

Let us define the network model. We consider a standard 6G HetNet comprising two distinct classes of infrastructure nodes: terrestrial base stations (TBSs) $\mathcal{B}=\{b_0,b_1,\dots ,b_B\}$, and rotary wing UAVs operating as aerial base stations (UABSs) $\mathcal{U}=\{u_1,u_2,\dots ,u_U\}$. The TBSs consist of one macro cell $b_0$, which provides ubiquitous coverage, and several small cells that, together with the UABSs, offer localized capacity boosting. The complete set of network nodes is denoted by $\mathcal{N}=\mathcal{B}\cup \mathcal{U}$, with cardinality $N=\left|\mathcal{N}\right|$. Each node $i\in \mathcal{N}$ acts as an independent agent in our multi-agent framework.

Unlike static models in prior work, we explicitly model the network’s dynamic and heterogeneous nature. We define a network task $\tau$ as a specific instance of this system, characterized by:

1.

Spatial Layout: The 2D/3D coordinates of all nodes, ${\mathbf{L}}_{\tau }=\{{\mathbf{l}}_i\in {\mathbb{R}}^3\mid i\in \mathcal{N}\}$.

2.

Traffic Demand Matrix: A time varying matrix ${\mathbf{D}}_{\tau }\left(t\right)$ specifying the requested data rate between each user equipment (UE) cluster and its potential serving nodes.

In our framework, the network is not fixed but rather drawn from a distribution $p\left(\mathcal{T}\right)$, where $\mathcal{T}$ denotes the set of all possible network configurations encountered in practice: different UAV positions, different traffic loads, different numbers of active nodes. Each configuration is a "task" $\tau$, and we want our policy to work well no matter which $\tau$ in practical deployments. A task $\tau$ is considered "unseen" during meta training if $\tau \notin {\mathcal{T}}_{\mathrm{train}}$, where ${\mathcal{T}}_{\mathrm{train}}$ is the finite set of tasks used for meta learning (Figure 2).

Each node $i\in \mathcal{N}$ is characterized by its type ${\tau }_i\in \{\mathrm{macro},\mathrm{small}\text{-}\mathrm{cell},\mathrm{UAV}\}$, which determines its energy consumption parameters. The node features ${\mathbf{f}}_i\left(t\right)$ include a one-hot encoding of the agent type, enabling the shared GNN policy to learn type-specific strategies.

The neighborhood $\mathcal{N}(i,t)$ of node i at time t is defined as the set of nodes within communication range $R_{\mathrm{comm}}=500$ m:

$$\mathcal{N}(i,t)=\{j\in \mathcal{N}\setminus \left\{i\right\}\mid \parallel {\mathbf{l}}_i\left(t\right)-{\mathbf{l}}_j\left(t\right){\parallel }_2\le R_{\mathrm{comm}}\}.$$

(1)

The observation vector ${\mathbf{f}}_i\left(t\right)$ for agent i has dimension 8 and comprises:

• (1) Traffic load (number of active UEs / capacity): $[0,1]$
• (2) Remaining battery energy (for UAVs): $[0,1]$
• (3) Current power consumption: scaled $[0,1]$
• (4) Current sleep/active status: $a_i(t-1)\in \{0,1\}$
• (5-7) three-bit one-hot encoding of node type ( macro, small-cell, UAV)
• (8) Average SINR to associated UEs: scaled $[0,1]$

The edge features ${\mathbf{e}}_{i,j}\left(t\right)$ have dimension 3:

• (1) Distance between nodes i and j: scaled $[0,1]$
• (2) Channel gain $h_{i,j}\left(t\right)$: scaled $[0,1]$
• (3) Relative velocity (for UAVs): scaled $[-1,1]$

Neighbor discovery is performed using periodic beacon signals, and nodes with distance $>R_{\mathrm{comm}}=500$ m are excluded. The maximum number of neighbors is capped at 10; if more neighbors exist, the closest 10 are selected.

3.2. Communication and User Association Model

The received signal power at UE cluster k from node i at time t is:

$$P_{i,k}^{\mathrm{rec}}\left(t\right)=P_i^{\left(\mathrm{tx}\right)}\left(t\right)·h_{i,k}\left(t\right),$$

(2)

where $h_{i,k}\left(t\right)$ is the channel gain including path loss and small-scale fading.

The Signal-to-Interference-plus-Noise Ratio (SINR) at UE k from node i is:

$${\gamma }_{i,k}\left(t\right)={\displaystyle \frac{P_i^{\left(\mathrm{tx}\right)}\left(t\right)·h_{i,k}\left(t\right)}{N_0+{\sum }_{j\ne i}{P}_j^{\left(\mathrm{tx}\right)}\left(t\right)·h_{j,k}\left(t\right)}},$$

(3)

where $N_0$ is the noise power and the summation represents co-channel interference from other active nodes.

Each UE associates with the active node offering the highest SINR:

$$\mathrm{serving}(k,t)=arg\underset{i\in \mathcal{N},a_i\left(t\right)=1}{max}{\gamma }_{i,k}\left(t\right).$$

(4)

If the serving node enters sleep mode ($a_i\left(t\right)$ changes from 1 to 0), the UE is handed over to the next best active node, potentially with a temporary performance degradation captured by the QoS violation indicator.

3.3. Channel and Energy Consumption Model

The channel gain $h_{i,k}\left(t\right)$ between node i and UE cluster k at time t is modeled as

$$h_{i,k}\left(t\right)=P{L}_{i,k}\left(t\right)·{\zeta }_{i,k},$$

(5)

where $P{L}_{i,k}$ is the path loss and ${\zeta }_{i,k}$ models small scale fading. For UABS to ground links (U2G), we employ the probabilistic Line of Sight (LoS) / Non Line of Sight (NLoS) model from the UMi (Urban Micro) scenario [28,29], with path loss given by:

$${\mathrm{PL}}_{i,k}^{\mathrm{U}2\mathrm{G}}\left(t\right)\left[\mathrm{dB}\right]=20{log}_{10}\left({\displaystyle \frac{4\pi f_c{d}_{i,k}\left(t\right)}{c}}\right)+{\eta }_{\mathrm{LoS}/\mathrm{NLoS}}\left(t\right),$$

(6)

where $f_c$ is the carrier frequency, c is the speed of light, $d_{i,k}$ is the 3D distance, and ${\eta }_{\mathrm{LoS}/\mathrm{NLoS}}$ is the environment dependent excess loss term. Terrestrial links follow standard 3GPP models [28].

In our model, the sleep action $a_i\left(t\right)=0$ for a UAV does not eliminate hovering power entirely, as the UAV must remain airborne. However, in a sleep state, the UAV reduces its propulsion power to a minimum hovering level ($P_i^{\left(\mathrm{hover},\min \right)}\approx 50$ W) by entering a low-power loitering mode. This contrasts with active hovering ($P_i^{\left(\mathrm{hover},\mathrm{active}\right)}\approx 300$ W), which includes additional thrust for maneuvering and maintaining precise positioning. The energy savings reported in our framework arise primarily from the ability to put terrestrial small cells into deep sleep (saving 400+ W each) and from reducing UAV power from active to minimal hovering. For the macro cell, which provides ubiquitous coverage, sleep is not permitted. Detailed power consumption figures are provided in Table 3.

The total power consumption of node i at time t is a critical metric. We adopt a comprehensive model from the GreenTouch project [30] and prior work on UAV energy [32]:

$$P_i^{\left(\mathrm{total}\right)}\left(t\right)=P_i^{\left(\mathrm{sleep}\right)}\left(t\right)·(1-a_i\left(t\right))+a_i\left(t\right)\left[P_i^{\left(\mathrm{circ}\right)}\left(t\right)+{\Delta }_i\left(t\right)·P_i^{\left(\mathrm{tx}\right)}\left(t\right)\right]+{\delta }_i·P_i^{\left(\mathrm{mob}\right)}\left(t\right),$$

(7)

where:

• $a_i\left(t\right)\in \{0,1\}$ is the binary sleep/active scheduling action for node i.
• $P_i^{\left(\mathrm{sleep}\right)}$ is the constant power in sleep mode.
• $P_i^{\left(\mathrm{circ}\right)}$ is the circuit power in active mode (processing, cooling).
• ${\Delta }_i$ is the slope of the load dependent power consumption.
• The transmit power to serve the associated users $\mathcal{K}$, with SINR ${\gamma }_{i,k}\left(t\right)$, data rate $r_{i,k}\left(t\right)$ bandwidth W, and power amplifier efficiency ${\kappa }_i$.

  $$P_i^{\left(\mathrm{tx}\right)}\left(t\right)=\sum _{k\in {\mathcal{K}}_i}{\displaystyle \frac{r_{i,k}\left(t\right)}{{\kappa }_i·W·{log}_2(1+{\gamma }_{i,k}\left(t\right))}}$$

  (8)
• The mobility power term $P_i^{\left(\mathrm{mob}\right)}\left(t\right)$ is non zero only for UABSs (${\delta }_i=1$ for $i\in \mathcal{U}$, else 0). For a rotary wing UAV, it is primarily the hovering power [32]:

  $$P_i^{\left(\mathrm{hover}\right)}\left(t\right)={\displaystyle \frac{\sqrt{{\left(m_{i}g\right)}^3}}{2\pi R_p^2{N}_r\rho \left(t\right)}}·{\displaystyle \frac{1}{\eta }},$$

  (9)

  where $m_i$ is mass, g is gravity, $R_p$ is rotor radius, $N_r$ is number of rotors, $\rho$ is air density, and $\eta$ is overall powertrain efficiency.

3.4. Problem Statement: Sleep Scheduling Optimization

The energy-efficient sleep scheduling problem can be formally stated as:

$$\underset{{\left\{a_i\left(t\right)\right\}}_{i\in \mathcal{N}}}{min}{\displaystyle \sum _{t=0}^T}\sum _{i\in \mathcal{N}}{P}_i^{\left(\mathrm{total}\right)}\left(t\right)$$

(10)

subject to:

$$\begin{array}{cc}\hfill r_{k,t}& \ge r_{min},\forall k\in \mathcal{K}\left(\mathrm{QoS}/\mathrm{rate}\mathrm{requirement}\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \sum _{i\in \mathcal{N}}{\mathbb{I}}_{\{a_i\left(t\right)=1\}}& \ge C_{min},\left(\mathrm{coverage}\mathrm{constraint}\right)\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill \sum _{k\in {\mathcal{K}}_i\left(t\right)}{r}_k\left(t\right)& \le C_i\left(t\right),\forall i\in \mathcal{N}\left(\mathrm{node}\mathrm{capacity}\right)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill a_i\left(t\right)& \in \{0,1\},\forall i\in \mathcal{N}\left(\mathrm{binary}\mathrm{decision}\right)\hfill \end{array}$$

(14)

where $r_{k,t}$ is the achievable rate of UE k, $r_{min}$ is the minimum acceptable rate, $C_{min}$ is the minimum number of active nodes required for coverage, and $C_i\left(t\right)$ is the capacity of node i. This optimization is intractable for large dynamic networks, motivating our RL-based approach.

3.5. Problem Formulation as a Dec-POMDP

The multi-agent sleep scheduling problem is formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). For the static scenario addressed by MARL-ESA, the tuple is defined as $\langle \mathcal{I},\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\Omega ,\mathcal{O},\gamma \rangle$:

• Agents: Each network node (TBS or UABS) is an independent agent, $\mathcal{I}=\mathcal{N}$.
• State Space $\mathcal{S}$: The global state at time t is $s_t=(\mathbf{L},\mathbf{D}\left(t\right),\mathbf{H}\left(t\right),{\mathbf{a}}_{t-1},\mathbf{P}(t-1))$, encompassing the fixed layout, current demand, channel matrix, previous joint action, and power consumption vector.
• Action Space $\mathcal{A}$: Each agent i selects a binary sleep/active action, $a_i\left(t\right)\in \{0,1\}$. The joint action is ${\mathbf{a}}_t\in {\{0,1\}}^N$.
• Observation Space $\Omega$: Due to decentralization and practical constraints, each agent i receives a partial observation $o_i\left(t\right)\in \Omega$:

  $$o_i\left(t\right)=\left\{{\mathbf{f}}_i\left(t\right),\{{\mathbf{f}}_j\left(t\right),{\mathbf{e}}_{i,j}\left(t\right)\mid \forall j\in \mathcal{N}(i,t)\}\right\},$$

  (15)

  where ${\mathbf{f}}_i$ are node features (e.g., traffic load, remaining energy) and ${\mathbf{e}}_{i,j}\left(t\right)$ denotes the edge features characterizing the wireless link between i and j (e.g., channel gain, distance, path loss).
• Observation Function $\mathcal{O}$: $o_i\left(t\right)=\mathcal{O}(s_t,i)$ maps the global state to the agent’s local perspective.
• State Transition Probability $\mathcal{P}$: $s_{t+1}\sim \mathcal{P}\left(s_{t+1}\right|s_t,{\mathbf{a}}_t)$.
• Reward Function $\mathcal{R}$: The global reward $r{d}_t^{\left(\tau \right)}=\mathcal{R}(s_t,{\mathbf{a}}_t)$ balances network performance and EE:

  $$r{d}_t^{\left(\tau \right)}={\omega }_1·\sum _{k\in {\mathcal{K}}^{\left(\tau \right)}}U\left(r_{k,t}^{\left(\tau \right)}\right)-{\omega }_2·\sum _{i\in {\mathcal{N}}^{\left(\tau \right)}}{P}_i^{\left(\mathrm{total}\right)}\left(t\right)-{\omega }_3·{\mathbb{I}}_{\left\{\mathrm{QoS}\mathrm{Violation}\right\}},$$

  (16)

  where $U(·)$ is a concave utility function of the achieved UE rate $r_{k,t}$, and $\mathbb{I}$ is an indicator for QoS violations (e.g., rate below a threshold $r_{min}$).
• Discount Factor: $\gamma \in (0,1]$.

For the dynamic scenario addressed by Meta-MARL-ESA, we extend this formulation to account for the task distribution $p\left(\mathcal{T}\right)$, yielding the tuple $\langle \mathcal{I},p\left(\mathcal{T}\right),\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\Omega ,\mathcal{O},\gamma \rangle$. The objective becomes:

$${\theta }^{*}=arg\underset{\theta }{max}{\mathbb{E}}_{\tau \sim p\left(\mathcal{T}\right)}\left[{\mathbb{E}}_{{\pi }_{{\theta }^{\prime }\left(\tau \right)}}\left[{\displaystyle \sum _{t=0}^{\infty }}{\gamma }^t{r}_t^{\left(\tau \right)}\right]\right],$$

(17)

where ${\theta }^{\prime }\left(\tau \right)=\mathrm{Adapt}(\theta ,\tau ,{\mathcal{D}}_{\tau }^{\mathrm{tr}})$ represents the parameters after adaptation on task $\tau$. For zero shot generalization, ${\theta }^{\prime }=\theta$.

4. MARL-ESA: Multi-Agent RL for Static UAV Assisted HetNets

Let’s start simple. Imagine a network that doesn’t change: fixed tower positions and predictable traffic and no surprises. That’s where MARL-ESA shines. It uses a clever trick called CTDE: during training, a central brain coordinates everything; during execution, each node acts on its own. This way, we get the best of both worlds—coordinated learning without central bottlenecks.

4.1. Architecture Overview

The MARL-ESA framework consists of three key components:

1.

Decentralized Actor Networks: Each agent i maintains a policy network ${\pi }_{{\theta }_i}\left(a_i\right|o_i)$ that maps local observations to action probabilities. These networks are implemented as multi layer perceptrons (MLPs) with two hidden layers of 256 neurons each.

2.

Centralized Critic Network: A central value function $V_{\varphi }({\mathbf{o}}_t,{\mathbf{a}}_{t-1})$ estimates the state action value using global information during training. This critic receives observations from all agents and the previous joint action, addressing the non stationarity inherent in multi-agent environments.

3.

Experience Replay Buffer: Transitions $({\mathbf{o}}_t,{\mathbf{a}}_t,r_t,{\mathbf{o}}_{t+1})$ are stored in a shared replay buffer for off policy learning.

4.2. Training Algorithm

We employ a multi-agent variant of Proximal Policy Optimization (PPO) [33] for stable training. The centralized critic computes advantages:

$$A_t^{\left(\tau \right)}={\displaystyle \sum _{k=t}^{T-1}}{\left(\gamma \lambda \right)}^{k-t}{\chi }_k^{\left(\tau \right)},{\chi }_k^{\left(\tau \right)}=r_k^{\left(\tau \right)}+\gamma V^{\left(\tau \right)}\left(s_{k+1}\right)-V^{\left(\tau \right)}\left(s_k\right),$$

(18)

where $s_T$ is the terminal state with $V\left(s_T\right)=0$, T is the total episode length, t is the current time step, and $\lambda$ is the Generalized Advantage Estimation (GAE) parameter. The actor is updated using the clipped surrogate objective:

$${\mathcal{L}}_{\mathrm{actor}}(\theta ,t)={\mathbb{E}}_t\left[min\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|o_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|o_t)}}{A}_t,\mathrm{clip}\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|o_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|o_t)}},1-ϵ,1+ϵ\right)A_t\right)\right]$$

(19)

The critic is updated by minimizing the mean squared error:

$${\mathcal{L}}_{\mathrm{critic}}\left(\varphi \right)={\mathbb{E}}_t\left[{\left(V_{\varphi }\left({\mathbf{o}}_t\right)-{\widehat{R}}_t\right)}^2\right],$$

(20)

where ${\widehat{R}}_t={\sum }_{l=0}^{T-t-1}{\gamma }^l{r}_{t+l}$ is the discounted return.

4.3. Energy Specific Reward Shaping

To guide the learning process toward energy efficient policies, we incorporate domain knowledge through reward shaping. The reward function in Eq. (16) is augmented with an EE bonus:

$$r{d}_t^{\mathrm{shaped}}=r_t+\psi ·{\displaystyle \frac{{\sum }_{k\in \mathcal{K}}{r}_{k,t}}{{\sum }_{i\in \mathcal{N}}{P}_i^{\left(\mathrm{total}\right)}\left(t\right)}},$$

(21)

where $\psi$ is a scaling factor. This shaping term encourages policies that maximize bits per joule, directly optimizing the EE metric.

4.4. Performance Evaluation

MARL-ESA was evaluated on a fixed topology with 1 macro TBS, 4 small cell TBSs, and 3 UAV, with static traffic patterns. The framework achieved:

• Energy Savings: 48% reduction in total power consumption compared to always-on operation (from 2150 W to 1110 W), with only a 1.5% degradation in aggregate throughput.
• QoS Compliance: Maintained 98.5% of user demands above the minimum rate threshold.
• Convergence: Stable policies achieved within 2000 training episodes.

However, as illustrated in Figure 3, when deployed on a new topology with different UAV positions, the pretrained policy’s performance degraded by 46%, necessitating complete retraining,a process requiring approximately 1500 episodes (3.2 hours of simulation time). This limitation motivates the development of Meta-MARL-ESA.

5. Meta-MARL-ESA: Toward Zero Shot Generalization

While MARL-ESA learned one trick for one network, Meta-MARL-ESA learns how to learn. Instead of optimizing for a single layout, it trains on a whole distribution $p\left(\mathcal{T}\right)$ of layouts, extracting the common principles that make sleep scheduling work regardless of where the nodes sit. This is achieved through a synergistic combination of a graph based policy representation and model agnostic meta learning (MAML). Relevant training hyperparameters are summarized in Table 4.

5.1. Graph Neural Network Policy Architecture

To handle the dynamic topology inherent in the task distribution $p\left(\mathcal{T}\right)$, where the number of nodes and their connectivity patterns vary, we propose a shared GNN policy ${\pi }_{\theta }\left(a_i\right|o_i)$, parameterized by $\theta$, that is executed identically by every agent $i\in \mathcal{N}$. The key advantage: the GNN doesn’t care about node numbering or how many neighbors each node has. Swap two antennas’ labels, and the output stays the same. Add a new drone to the network, and the policy adjusts automatically. That’s what makes zero shot generalization possible, the architecture itself handles variability without retraining.

The local observation $o_i\left(t\right)$ from Eq. (15) is formalized as a local graph ${\mathcal{G}}_i\left(t\right)=({\mathcal{V}}_i,{\mathcal{E}}_i)$. The node set ${\mathcal{V}}_i=\left\{i\right\}\cup \mathcal{N}\left(i\right)$ includes the ego agent and its K neighbors. Each node $j\in {\mathcal{V}}_i$ has a feature vector ${\mathbf{f}}_j\left(t\right)$, and each directed edge $(j,k)\in {\mathcal{E}}_i$ has a feature vector ${\mathbf{e}}_{j,k}\left(t\right)$ representing channel gain and distance.

The GNN policy operates via a message passing neural network (MPNN) with L layers. Let ${\mathbf{h}}_j^{\left(l\right)}$ denote the hidden representation of node j at layer l, with ${\mathbf{h}}_j^{\left(0\right)}={\mathbf{f}}_j$. The forward pass for agent i’s policy at time t is:

$$\begin{array}{c}\hfill {\mathbf{m}}_{j\to k}^{\left(l\right)}\left(t\right)={\mathrm{MLP}}_{\mathrm{msg}}^{\left(l\right)}\left({\mathbf{h}}_j^{(l-1)}\left(t\right),{\mathbf{h}}_k^{(l-1)}\left(t\right),{\mathbf{e}}_{j,k}\left(t\right)\right),\end{array}$$

(22)

$$\begin{array}{c}\hfill {\mathbf{h}}_k^{\left(l\right)}\left(t\right)={\mathrm{GRU}}^{\left(l\right)}\left({\mathbf{h}}_k^{(l-1)}\left(t\right),\sum _{j\in \mathcal{N}(k,t)}{\mathbf{m}}_{j\to k}^{\left(l\right)}\left(t\right)\right),\end{array}$$

(23)

$$\begin{array}{c}\hfill {\mu }_i\left(t\right)={\mathrm{MLP}}_{\mathrm{out}}\left({\mathbf{h}}_i^{\left(L\right)}\left(t\right)\right).\end{array}$$

(24)

Eq. (22) computes a message from node j to its neighbor k. Eq. () aggregates incoming messages at node k and updates its hidden state using a Gated Recurrent Unit (GRU), which helps capture temporal dependencies across decision steps. Finally, Eq. () maps the final representation of the ego node i to the logits ${\mu }_i$ of a Bernoulli distribution, from which the binary sleep/active action $a_i$ is sampled (Figure 4).

The GNN policy is permutation-equivariant: applying a permutation to the input node ordering results in the same permutation applied to the output node representations. This property ensures that the policy is independent of node indexing and can process graphs of arbitrary size. For node-level decisions (sleep/active), this equivariance is precisely the desired property, as each agent’s output corresponds to its own decision. The shared parameters across agents further enforce that all nodes are treated symmetrically, enabling zero-shot generalization to networks with different numbers of nodes.

5.2. Meta Training via MAML

The GNN provides a flexible policy representation. To learn parameters $\theta$ that are adaptable to new tasks, we employ the MAML algorithm [20]. The core idea is to train $\theta$ such that a small number of gradient steps on data from a new task $\tau$ produces a significant performance improvement. The meta-training process involves a bilevel optimization loop, executed over a batch of tasks sampled from $p\left(\mathcal{T}\right)$ (Algorithm 1).

Inner Loop (Task Specific Adaptation): For each task ${\tau }_i$ in a metabatch, we sample two trajectories: a support set ${\mathcal{D}}_i^{\mathrm{supp}}$ for adaptation and a query set ${\mathcal{D}}_i^{\mathrm{query}}$ for metaparameter update. In our implementation, both the actor parameters $\theta$ and the critic parameters $\varphi$ are meta-learned. During the inner loop, both sets of parameters are adapted to task ${\tau }_i$ via gradient descent:

$${\theta }_i^{\prime }=\theta -\alpha {\nabla }_{\theta }{\mathcal{L}}_{{\tau }_i}^{\mathrm{supp}}({\pi }_{\theta },V_{\varphi }),{\varphi }_i^{\prime }=\varphi -\alpha {\nabla }_{\varphi }{\mathcal{L}}_{{\tau }_i}^{\mathrm{supp}}({\pi }_{\theta },V_{\varphi }).$$

(25)

where $\alpha$ is the inner loop learning rate.

Outer Loop (Meta Parameter Update): The outer loop subsequently updates both meta-parameters using the query set losses. The critic is neither shared across tasks without adaptation nor retrained separately; it is jointly meta-learned to provide task-specific value estimates during adaptation. The meta objective is to minimize the aggregate loss across all tasks in the meta batch, evaluated with the adapted parameters ${\theta }_i^{\prime }$ on their respective query sets:

$$\underset{\theta }{min}{\mathcal{L}}_{\mathrm{meta}}\left(\theta \right)=\sum _{{\tau }_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\tau }_i}^{\mathrm{query}}\left({\pi }_{{\theta }_i^{\prime }}\right).$$

(26)

The meta parameters $\theta$ are then updated via gradient descent:

$$\theta (t+1)\leftarrow \theta \left(t\right)-\beta \left(t\right){\nabla }_{\theta }\sum _i{\mathcal{L}}_{{\tau }_i}^{\mathrm{query}}\left({\pi }_{{\theta }_i^{\prime }\left(t\right)},t\right),$$

(27)

where $\beta$ is the meta learning rate.    

Algorithm 1: Meta-MARL-ESA Training Procedure

5.3. Zero Shot Inference and Fine Tuning

After meta training, we obtain a meta policy ${\pi }_{{\theta }^{*}}$ that enables two modes of deployment:

• Zero Shot Generalization: For deployment on a completely unseen task ${\tau }_{\mathrm{new}}$, the meta policy can be applied directly without any gradient steps. The GNN architecture allows it to process the new network graph ${\mathcal{G}}_{{\tau }_{\mathrm{new}}}$ immediately.
• Few Shot Fine Tuning: If higher performance is required, the system can perform a few steps of inner loop adaptation using a small amount of real time interaction data collected from the live network.

During online fine-tuning, the adaptation operations are executed at a central controller (e.g., the O-RAN SMO) or at a distributed edge node with sufficient computational resources. For a network with N agents, each inner-loop step requires:

• Trajectory collection: T decision epochs (typically 1-5 seconds of real-time data)
• Gradient computation: $O(N·H)$ floating-point operations, where H is the GNN hidden dimension (128 in our implementation)
• Parameter update: $O(N·|\theta \left|\right)$ operations for K gradient steps ($K=1$ in our experiments)

On a standard GPU (NVIDIA A100), the gradient computation and parameter update for the full network take approximately 50-100 ms, well within the latency budget for 6G NTN control loops. The main latency bottleneck is trajectory collection, which requires 1-5 seconds of network telemetry. This aligns with the 8-epoch recovery time reported in Figure 8, confirming that the proposed framework is compatible with real-time 6G operation.

6. Simulation Setup and Numerical Results

6.1. Simulation Parameters and Scenario Generation

To evaluate the performance of both MARL-ESA and Meta-MARL-ESA, we developed a comprehensive network simulator in Python. The core parameters, aligned with 3GPP studies and prior art [2,32], are summarized in Table 5.

Each decision epoch corresponds to 1 second of real time, consistent with typical 3GPP NTN control loop timings. Thus, the 8-epoch recovery time reported in dynamic repositioning experiments corresponds to 8 seconds of real-time adaptation.

6.1.1. Task Distribution $p\left(\mathcal{T}\right)$ for Meta-Training

We construct the task distribution $p\left(\mathcal{T}\right)$ by randomizing two key elements per training episode:

1.

Topology ${\mathbf{L}}_{\tau }$: The number of TBSs and UABSs is randomly sampled. Their 2D positions are sampled uniformly at random with a minimum separation constraint of 200 m.

2.

Traffic Demand ${\mathbf{D}}_{\tau }$: The peak data rate demand for each UE cluster is sampled from a truncated Gaussian distribution, $d_k\sim {\mathcal{N}}_{[0,10]}(5,3)$ Mbps.

The meta-training set ${\mathcal{T}}_{\mathrm{train}}$ consists of 5000 unique tasks. For testing, we create two distinct sets: ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{seen}}$ (500 tasks from the same distribution) and ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{unseen}}$ (500 tasks with out of distribution characteristics).

The OOD test set ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{unseen}}$ is characterized by the following deviations from the training distribution:

• Network area: 3 km × 3 km (training: 2 km × 2 km, 50% larger)
• Number of UABSs: 4–6 (training: 2–4, up to 50% more)
• Traffic demand: $d_k\sim {\mathcal{N}}_{[0,15]}(10,5)$ Mbps (training: ${\mathcal{N}}_{[0,10]}(5,3)$ Mbps, mean demand doubled)
• Minimum separation constraint: 150 m (training: 200 m, more dense deployments)

These modifications ensure that the test scenarios lie outside the support of the training distribution, providing a meaningful assessment of OOD generalization.

6.1.2. Baseline Algorithms

We compare against:

• MARL-ESA (from scratch): The original algorithm, reimplemented with the same GNN architecture for fairness.
• Greedy Heuristic: A rule based algorithm where a node enters sleep mode if its traffic load is below a threshold.
• Random Sleep Scheduling: Each node independently chooses to sleep with a fixed probability $p=0.3$ at each decision epoch, reflecting a simple baseline where nodes deactivate without considering network state.

6.2. Performance Analysis

6.2.1. MARL-ESA: Static Scenario Performance

In the static topology scenario, MARL-ESA achieves significant energy savings while maintaining QoS. Figure 5 shows the learning curve and energy consumption patterns.

MARL-ESA achieves a 93% reduction in total power consumption compared to always on operation, with a 11% improvement in aggregate throughput. This establishes MARL-ESA as an effective solution for static deployments.

6.2.2. Meta-MARL-ESA: Convergence Rate and Jumpstart

Figure 6 compares the learning curve of Meta-MARL-ESA against MARL-ESA (from scratch) on a single, held out unseen task ${\tau }^{*}\in {\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{unseen}}$.

The results demonstrate the core advantage of meta-learning:

• Jumpstart: The zero shot performance of Meta-MARL-ESA provides an immediate average return of 42.5, which is already 82% of its final converged performance. In contrast, the randomly initialized MARL-ESA starts near zero.
• Convergence Speed: Meta-MARL-ESA reaches 95% of its peak performance in just 120 iterations. MARL-ESA requires over 1500 iterations to achieve the same level, indicating a speed up factor exceeding 12×.

6.2.3. Generalization Gap and Zero Shot Efficacy

To evaluate generalization, we compute the EE metric across 1000 test episodes. Table 6 reports the average EE for different algorithms and task sets.

Key observations:

• In Distribution Generalization: On ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{seen}}$, zero shot performance (3.82) is within 3.3% of converged MARL-ESA (3.95).
• OOD Robustness: On ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{unseen}}$, zero shot EE (3.51) outperforms converged MARL-ESA (2.89) by 21.5%.
• Value of Fine Tuning: With just 10 episodes of fine tuning, performance improves to 3.92, nearly matching optimal in distribution performance.

6.2.4. EE vs. SE Trade-Off

The empirical EE-SE trade-off curves in Figure 7 are obtained by varying the energy penalty weight ${\omega }_2$ from 0.001 to 0.05. For each value of ${\omega }_2$, we run 100 independent episodes and record the average (SE, EE) pair. The solid markers identify the non-dominated operating points (points for which no other configuration yields both higher EE and higher SE). The dashed lines are quadratic fits to the non-dominated points, serving as a visual guide to the optimal trade-off. For Meta-MARL-ESA, the fitted coefficients are $a=-0.045$, $b=0.55$, and $c=2.5$. The greedy heuristic yields $a=-0.035$, $b=0.35$, and $c=1.8$.

Meta-MARL-ESA’s frontier is strictly superior. At a target SE of 4 bps/Hz, Meta-MARL-ESA achieves 50% higher EE than the heuristic, demonstrating learned coordination strategies beyond simple greedy rules.

6.2.5. Adaptation to Dynamic UAV Repositioning

To simulate a key NTN mobility scenario, we design an experiment where, mid episode, one UABS is commanded to reposition to a new hotspot area. Figure 8 shows the instantaneous network EE before, during, and after the move.

The Greedy Heuristic suffers severe, prolonged performance degradation. The zero shot Meta-MARL-ESA policy maintains 85% of premove EE. With just 5 epochs of online fine tuning, it recovers to 98% of optimal within 8 seconds.

6.3. Comparative Analysis with State of the Art

The performance gains achieved by Meta-MARL-ESA substantially outperform recent advances in UAV assisted network optimization. Compared to conventional DRL based approaches that optimize UAV trajectories and resource allocation [34], our framework demonstrates superior adaptability, achieving 82% zero shot performance versus the 60-70% typical of retrained models. The 21.5% improvement on out of distribution tasks significantly exceeds the 13% satisfaction index improvement reported in recent Multi RAT HetNet optimizations [35]. While prior work on licensed band UAV deployments achieved EE gains through convex optimization, these methods require complete Channel State Information (CSI) and fail under dynamic conditions. Similarly, unlicensed band approaches using heuristic algorithms [36] cannot adapt to topology changes without manual reconfiguration. The 12× convergence speed up of Meta-MARL-ESA contrasts sharply with the 5000+ episodes required by conventional deep RL algorithms in similar multi UAV scenarios [34]. Furthermore, unlike federated or distilled RL approaches that improve security but not generalization [37], our meta learning framework fundamentally enables cross topology transfer. Recent advances in multi-agent coordination, such as the GLIDE framework achieving 68% faster task completion in dynamic military environments [38], still require retraining for new scenarios, whereas our approach maintains high performance zero shot. Similarly, while integrated path planning methods like CRRQN achieve 15% channel capacity improvement through joint optimization [39], they cannot generalize to unseen topologies without complete retraining. These comparisons collectively validate that Meta-MARL-ESA establishes a new paradigm for generalizable, energy efficient network control that addresses the critical limitations of existing methods, combining the adaptability of meta learning with the structural invariance of GNN.

The ’Target Performance’ line in Figure 5(a) represents the peak performance achieved by MARL-ESA after full convergence (2000 episodes), serving as a benchmark for evaluating convergence speed. The ’Optimal’ curve in Figure 8 is obtained by running the fully converged Meta-MARL-ESA policy on the pre-move topology, representing the best achievable performance before the UAV repositioning event. This curve is the converged performance of the same policy before the topology change.

The minor performance drop observed near 2000 episodes in Figs. Figure 5 and Figure 6 corresponds to the scheduled reduction in the learning rate (annealing), which temporarily destabilizes the policy before stabilization at a higher performance level. This is a known phenomenon in PPO-based training when the learning rate is reduced too aggressively. In all experiments, the performance recovers and stabilizes within 100 additional episodes, confirming that the drop is transient and does not indicate policy collapse. The final converged performance remains stable for an additional 1000 episodes (verified in extended runs).

6.4. Ablation Study: Component Contribution Analysis

To isolate the contribution of each component, we evaluate four configurations on ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{unseen}}$:

1.

MARL + MLP: Conventional MARL with MLP policy (no topology awareness, no meta-learning)

2.

MARL + GNN (no meta): GNN policy trained across randomized tasks without MAML

3.

MAML + MLP: MAML applied to MLP policy (meta-learning without topology awareness)

4.

Meta-MARL-ESA (GNN + MAML): Complete framework

Table 7 reports the zero-shot EE on ${\mathcal{T}}_{\mathrm{test}\text{-}\mathrm{unseen}}$.

The GNN alone provides a 38.6% improvement over the MLP baseline by enabling topology-aware processing. MAML with MLP yields a 33.5% improvement through rapid adaptation. However, the full combination (GNN + MAML) achieves a 63.3% improvement over the baseline and outperforms each individual component, demonstrating that both the permutation-equivariant architecture and the meta-learning algorithm are necessary for the reported generalization gains.

6.5. UAV Repositioning Scenario

The UAV repositioning event is modeled as a sudden change in the position of one UABS at time $t=50$ epochs. The repositioning trajectory is assumed instantaneous for the purpose of evaluating sleep scheduling adaptation (the movement itself takes 1 epoch, during which the UAV is unavailable for service). After repositioning, the UAV assumes its new position, and the network topology is updated. The updated channel gains and SINR values are recomputed based on the new distances, and UEs re-associate with the best active node according to the max-SINR rule. The propulsion energy during movement is modeled as an additional 150 W for the duration of the move (1 epoch), after which the UAV resumes its normal hovering power at the new location.

7. Discussion and Conclusion

7.1. Contribution: From MARL to Meta-MARL-ESA

This paper has presented a comprehensive framework for energy efficient sleep scheduling in UAV assisted 6G HetNets, evolving from static MARL to generalizable meta reinforcement learning. We first introduced MARL-ESA, which successfully optimizes energy consumption in static scenarios through a CTDE paradigm, achieving 93% energy savings. However, its policies are tied to specific network configurations, requiring computationally prohibitive retraining for every topology change.

To overcome this limitation, we proposed Meta-MARL-ESA, a novel meta RL framework that enables zero shot generalization and rapid adaptation to unseen network configurations. By formulating the problem as a Dec-POMDP over a distribution of topological tasks and synergistically integrating a permutation invariant GNN policy with MAML, we have achieved a paradigm shift from task specific optimization to generalizable intelligence.

Our comprehensive simulation results demonstrate the framework’s significant advantages:

• Zero Shot Generalization: 82% of optimal performance from the first decision epoch on unseen topologies.
• Rapid Adaptation: 12× faster convergence than training from scratch.
• Superior Robustness: 21.5% improvement on OOD tasks over converged task specific baselines.
• Dynamic Agility: Recovery to 98% of optimal within 8 seconds during UAV repositioning.

These contributions establish a foundation for practical, autonomous, and energy-sustainable 6G NTN operations.

7.2. Practical Feasibility and Implementation Considerations

The proposed framework is designed with practical deployment in mind. The local observation $o_i\left(t\right)$ in Eq. (15) comprises node features ${\mathbf{f}}_i\left(t\right)$ (traffic load, remaining energy) and neighbor features ${\mathbf{e}}_{i,j}\left(t\right)$ (channel gain, distance). These quantities are obtainable through existing 3GPP signaling: traffic load from buffer status reports, remaining energy from battery telemetry, and channel gain from CSI reports or reference signal measurements. Neighbor discovery is performed through standard discovery procedures (e.g., SSB or SIB broadcasts). The varying number of neighbors is handled by the GNN’s permutation-equivariant architecture, which processes an arbitrary number of incoming messages.

The global reward in Eq. (16) is computed centrally during training. This requires collecting from each agent: (i) the achievable rates of served UEs, (ii) the total power consumption, and (iii) the QoS violation indicator. In practice, this information can be aggregated at a central controller (e.g., a network management system or a cloud-based orchestrator) via the O-RAN SMO (Service Management and Orchestration) framework. During online adaptation, the reward is computed locally or at a distributed edge node using available telemetry, enabling decentralized execution.

Recent works on feasible and generalized MARL-based wireless protocol learning [40] provide complementary perspectives on practical implementation constraints and signaling overhead.

7.3. Computational Complexity Analysis

The computational complexity of Meta-MARL-ESA is analyzed in terms of training, adaptation, and inference:

• Training cost: $O(E·T·N·H^2)$, where E is the number of meta-iterations (20,000), T is episode length (100 steps), N is the number of agents (3-10), and H is the GNN hidden dimension (128). Training requires approximately 48 hours on a single NVIDIA A100 GPU.
• Adaptation cost (fine-tuning): $O(K·N·H^2)$, where K is the number of inner-loop steps ($K=1$). Fine-tuning requires 50-100 ms per step on the same hardware.
• Inference latency: $O(N·H^2)$ per decision epoch, corresponding to < 10 ms for a network with 10 agents, well within the 1-second decision epoch duration.

7.4. Future Work

Future work will develop heterogeneous GNN architectures with distinct processing pipelines for macro cells, small cells, and UAVs to enable role-specific strategies. We will also extend our meta-learning framework to handle time-varying operator priorities (such as shifting from energy minimization to throughput maximization during peak hours) through multi-objective meta-learning that generates a continuous Pareto frontier of policies. Finally, domain randomization techniques will be implemented during meta-training, systematically varying channel models, UAV mobility patterns, and sensor noise characteristics to bridge the simulation-to-reality gap and ensure robust real-world deployment.