Federated Multi-Agent Deep Reinforcement Learning for Joint Channel Selection and Power Control in Cognitive Radio Networks

1. Introduction

The exponential growth of wireless communication services and the proliferation of Internet of Things (IoT) devices have imposed unprecedented demands on limited radio spectrum resources [1]. Cognitive radio networks (CRNs) have emerged as a promising paradigm to address spectrum scarcity by enabling dynamic spectrum access (DSA), allowing unlicensed secondary users (SUs) to opportunistically access licensed frequency bands without causing harmful interference to primary users (PUs) [2]. Effective resource management in CRNs—including joint channel selection, power control, and spectrum allocation—is critical for maximizing spectral efficiency while ensuring quality of service for both primary and secondary users [3].

Figure 1. Overview of the evolution from traditional optimization to federated multi-agent deep reinforcement learning for cognitive radio resource management, highlighting the key challenges and the proposed FedMA-DRL framework.

Figure 1. Overview of the evolution from traditional optimization to federated multi-agent deep reinforcement learning for cognitive radio resource management, highlighting the key challenges and the proposed FedMA-DRL framework.

Traditional optimization-based approaches for spectrum management in CRNs, such as convex optimization and game-theoretic formulations, have demonstrated their utility in controlled settings [4]. However, these methods suffer from several fundamental limitations: (1) they require accurate and instantaneous channel state information (CSI), which is often unavailable in rapidly varying wireless environments; (2) the computational complexity scales exponentially with the number of users and channels, rendering real-time decision-making infeasible; and (3) they fail to adapt to the dynamic, partially observable nature of real-world CRN scenarios [5]. The transition from fog IoT spectrum management via traditional optimization to deep neural network (DNN)-based resource allocation marked an important step toward data-driven solutions, yet single-agent approaches still lack the ability to model the complex interactions among multiple cognitive users [6,7].

Deep reinforcement learning (DRL) has recently attracted significant attention as a powerful framework for autonomous decision-making in complex, dynamic environments [8]. Multi-agent DRL (MADRL) further extends this capability by enabling distributed decision-making among multiple SUs, where each agent learns to cooperate or compete in a shared spectrum environment [9]. The centralized training with decentralized execution (CTDE) architecture has shown particular promise, allowing agents to leverage global information during training while making independent decisions during deployment [10]. Despite these advances, several challenges remain: sparse reward signals in spectrum access tasks hinder effective learning, exploration inefficiency leads to suboptimal policies, and single-domain training results in poor generalization across heterogeneous wireless environments [11]. Notably, attention mechanisms have shown remarkable effectiveness across diverse domains, from medical image segmentation [12] to speech enhancement [13,14], motivating their application in cross-domain wireless adaptation.

Federated learning (FL) offers a privacy-preserving paradigm for distributed model training, enabling multiple devices to collaboratively learn a shared model without exchanging raw data [15]. Integrating FL with DRL in CRNs can potentially combine the adaptive decision-making capability of DRL with the privacy protection and communication efficiency of FL. Recent studies have explored federated DRL for dynamic spectrum access, demonstrating improved convergence and stability compared to traditional federated approaches [16]. However, existing federated DRL methods often overlook the structural relationships among wireless devices and fail to effectively address cross-domain generalization.

In this paper, we propose FedMA-DRL, a Federated Multi-Agent Deep Reinforcement Learning framework for joint channel selection and power control in cognitive radio networks. Our approach integrates four key innovations: (1) a CTDE-based multi-agent architecture that models the spectrum access problem as a multi-agent Markov decision process (MAMDP) with nonlinear reward shaping to address sparse rewards; (2) a graph neural network (GNN)-augmented Q-value predictor that captures the topological relationships among SUs and environmental features; (3) a federated learning mechanism with an age-aware aggregation strategy (FedAge) for privacy-preserving knowledge sharing across devices; and (4) an attention-based domain adaptation module that enhances cross-domain generalization by aligning feature distributions across heterogeneous wireless environments.

We evaluate FedMA-DRL on a realistic CRN testbed comprising 10 PU channels and 15 heterogeneous SUs. Extensive experiments demonstrate that FedMA-DRL achieves a secondary user throughput of 14.87 Mbps, outperforming R2D2 (13.92 Mbps) and C-DRL (12.34 Mbps). The collision probability is reduced to 0.038, significantly lower than competing methods. FedMA-DRL also achieves the highest energy efficiency (4.35 bits/Joule) and spectrum efficiency (6.23 bits/s/Hz), confirming the effectiveness of our proposed components.

Our main contributions are summarized as follows:

• We propose FedMA-DRL, a federated multi-agent DRL framework that integrates CTDE architecture with nonlinear reward shaping and action-guided exploration for efficient joint channel selection and power control in CRNs.
• We introduce a GNN-augmented Q-value predictor that leverages the topological structure of wireless devices to improve prediction accuracy and a FedAge-based federated aggregation strategy for privacy-preserving distributed learning.
• We develop an attention-based domain adaptation module that enhances cross-domain generalization, enabling robust performance across heterogeneous wireless environments without requiring domain-specific retraining.

2. Related Work

2.1. Deep Reinforcement Learning for Dynamic Spectrum Access

Deep reinforcement learning has emerged as a powerful paradigm for addressing the dynamic spectrum access problem in cognitive radio networks. Kumar and Kumar [17] provided a comprehensive review of RL-based approaches for DSA, evaluating algorithms ranging from Q-learning to deep policy gradient methods such as DDPG in CRN scenarios. Xu et al. [18] proposed a DRL-based scheme for DSA networks that enables SUs to opportunistically access PU spectrum with minimal interference, demonstrating significant throughput improvements over traditional sensing-based approaches. The application of DQN and QR-DQN for television whitespace CRNs was explored by Ukpong et al. [19], who achieved interference avoidance rates exceeding 96% through DRL-based channel selection. Malhotra [20] conducted a comparative study of PPO, DQN, and R2D2 for wireless resource allocation, finding that R2D2 offers the best stability and convergence due to its recurrent architecture and prioritized experience replay. Bai et al. [21] proposed MA-JCSPC, a multi-agent DRL method with CTDE architecture and nonlinear reward shaping for joint channel selection and power control, achieving up to 9.1% throughput improvement over centralized DRL baselines. Li and Wang [22] developed a decentralized multi-agent deep reinforcement learning approach formulating spectrum access as a stochastic game among distributed learning agents. Venkatesan and Kumaratharan [23] introduced a hierarchical multi-agent RL controller with graph-driven occupancy prediction and federated meta-reinforcement learning for efficient CRN management. These works collectively demonstrate the progression from single-agent to multi-agent DRL in CRNs, yet they often overlook the benefits of federated knowledge sharing and structured device relationship modeling.

2.2. Federated Learning and Graph Neural Networks in Wireless Networks

Federated learning and graph neural networks have gained significant traction for distributed, privacy-preserving wireless resource management. Aggarwal and Gupta [15] presented a comprehensive review of FL methods and their privacy-preserving mechanisms, highlighting the challenges of heterogeneous data distributions and communication efficiency in distributed training. Li et al. [16] proposed a dynamic spectrum access scheme for IoT that integrates federated learning with GNN and DQN, where GNN captures device relationships for Q-value prediction and FedAge coordinates multi-device knowledge sharing. Shah and Ali [24] introduced a federated fog computing framework for IoT resource allocation, combining predictive scheduling with energy-aware resource allocation and adaptive mobility management. The intersection of FL and GNN was explored by Wu et al. [25], who developed privacy-preserving federated GNN with local differential privacy for graph-structured data analysis. He et al. [26] proposed a federated GNN framework with privacy-preserving graph expansion protocols for incorporating high-order interaction information. For NOMA networks, Banday [27] combined DRL with hybrid optimization algorithms for RIS-assisted user clustering and phase shift optimization, achieving substantial rate improvements. Yang et al. [6] laid the foundation for fog IoT spectrum management, while their subsequent work [7] introduced DNN-based resource allocation in NOMA networks and [28] extended this to partially observable multi-agent deep RL for cognitive resource management. Similarly, adaptive attention selection mechanisms have proven effective in other signal processing domains, such as visual speech enhancement [29] and adaptive local/non-local attention for speech enhancement [14]. Despite these advances, existing federated DRL methods often fail to explicitly model the topological relationships among wireless devices or address cross-domain generalization, which our work specifically targets through GNN-augmented Q-value prediction and attention-based domain adaptation.

3. Method

In this section, we present the proposed FedMA-DRL framework, a Federated Multi-Agent Deep Reinforcement Learning approach for joint channel selection and power control in cognitive radio networks. FedMA-DRL integrates four core modules: a CTDE-based multi-agent architecture with nonlinear reward shaping, a graph neural network (GNN)-augmented Q-value predictor, a federated aggregation mechanism with age-aware weighting (FedAge), and an attention-based domain adaptation module. The overall architecture is illustrated in Figure 2.

3.1. Problem Formulation

We model the joint channel selection and power control problem in CRNs as a multi-agent Markov decision process (MAMDP). Consider a CRN with N secondary users (SUs) and M primary user (PU) channels. Each SU i at time step t observes a local state ${\mathbf{o}}_i^t$ comprising the channel availability vector ${\mathbf{a}}^t\in {\{0,1\}}^M$, the previous transmission outcome $r_i^{t-1}$, and the interference level ${\mathbf{I}}_i^t$. The action of SU i is defined as ${\mathbf{u}}_i^t=(c_i^t,p_i^t)$, where $c_i^t\in \{1,\dots ,M\}$ denotes the selected channel and $p_i^t\in [0,P_{max}]$ denotes the transmit power level.

The joint objective is to maximize the long-term cumulative reward across all SUs:

$$\begin{array}{c}\hfill \underset{\mathit{\pi }}{max}\mathbb{E}\left[\sum _{t=0}^T{\gamma }^t\sum _{i=1}^N{R}_i^t({\mathbf{s}}^t,{\mathbf{u}}^t)\right]\end{array}$$

(1)

where $\gamma \in (0,1)$ is the discount factor, $\mathit{\pi }=\{{\pi }_1,\dots ,{\pi }_N\}$ is the joint policy, and $R_i^t$ is the reward function for SU i.

3.2. Nonlinear Reward Shaping

To address the sparse reward problem inherent in spectrum access tasks, we design a nonlinear reward function that incorporates both performance incentives and constraint penalties:

$$\begin{array}{c}\hfill R_i^t={\alpha }_1·log(1+{\mathrm{SIN}\mathrm{R}}_i^t)-{\alpha }_2·\mathbb{1}\left[{\mathrm{collision}}_i^t\right]·P_{\mathrm{coll}}-{\alpha }_3·{\displaystyle \frac{p_i^t}{P_{max}}}\end{array}$$

(2)

where ${\mathrm{SIN}\mathrm{R}}_i^t$ is the signal-to-interference-plus-noise ratio, $\mathbb{1}\left[{\mathrm{collision}}_i^t\right]$ is an indicator for PU collision events, $P_{\mathrm{coll}}$ is the collision penalty, and ${\alpha }_1,{\alpha }_2,{\alpha }_3$ are weighting coefficients. The logarithmic transformation of SINR provides diminishing returns at high signal quality, encouraging more balanced spectrum utilization across SUs.

Furthermore, we introduce an action-guided initial exploration strategy. During the initial training phase, each SU’s action selection is biased toward feasible actions derived from spectrum sensing results:

$$\begin{array}{c}\hfill {\pi }_i^{\left(0\right)}\left(c_i^t\right|{\mathbf{o}}_i^t)={\displaystyle \frac{exp(\beta ·\mathbb{1}\left[c_i^t\mathrm{idle}\right])}{{\sum }_{c=1}^{M}exp(\beta ·\mathbb{1}\left[c\mathrm{idle}\right])}}\end{array}$$

(3)

where $\beta$ is a temperature parameter that controls the exploration bias strength, gradually annealed as training progresses.

3.3. GNN-Augmented Q-Value Predictor

We employ a graph neural network to capture the topological relationships among SUs and enhance Q-value prediction. Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ represent the interference graph, where each node $v_i\in \mathcal{V}$ corresponds to SU i and an edge $(v_i,v_j)\in \mathcal{E}$ exists if SUs i and j can potentially interfere.

For each SU i, the initial node feature is constructed from the local observation:

$$\begin{array}{c}\hfill {\mathbf{h}}_i^{\left(0\right)}=f_{\mathrm{enc}}\left({\mathbf{o}}_i^t\right)\end{array}$$

(4)

where $f_{\mathrm{enc}}$ is a feature encoding network consisting of fully connected layers with ReLU activation.

The GNN performs K rounds of message passing to aggregate neighborhood information:

$$\begin{array}{c}\hfill {\mathbf{h}}_i^{\left(k\right)}=\sigma \left({\mathbf{W}}^{\left(k\right)}·\mathrm{AGG}\left(\{{\mathbf{h}}_j^{(k-1)}:j\in \mathcal{N}\left(i\right)\}\right)+{\mathbf{b}}^{\left(k\right)}\right)\end{array}$$

(5)

where $\mathcal{N}\left(i\right)$ denotes the neighbors of node i, AGG is an attention-weighted aggregation function, and $\sigma$ is the ELU activation function.

The attention-weighted aggregation is defined as:

$$\begin{array}{c}\hfill {\alpha }_{ij}^{\left(k\right)}={\displaystyle \frac{exp\left(\mathrm{LeakyReLU}\left({\mathbf{a}}^{\left(k\right)T}[{\mathbf{h}}_i^{(k-1)}\parallel {\mathbf{h}}_j^{(k-1)}]\right)\right)}{{\sum }_{j^{\prime }\in \mathcal{N}\left(i\right)}exp\left(\mathrm{LeakyReLU}\left({\mathbf{a}}^{\left(k\right)T}[{\mathbf{h}}_i^{(k-1)}\parallel {\mathbf{h}}_{j^{\prime }}^{(k-1)}]\right)\right)}}\end{array}$$

(6)

$$\begin{array}{c}\hfill \mathrm{AGG}\left(\{{\mathbf{h}}_j^{(k-1)}:j\in \mathcal{N}\left(i\right)\}\right)=\sum _{j\in \mathcal{N}\left(i\right)}{\alpha }_{ij}^{\left(k\right)}·{\mathbf{h}}_j^{(k-1)}\end{array}$$

(7)

The final Q-value for SU i given action ${\mathbf{u}}_i^t$ is computed as:

$$\begin{array}{c}\hfill Q_i({\mathbf{o}}_i^t,{\mathbf{u}}_i^t)=f_{\mathrm{out}}\left({\mathbf{h}}_i^{\left(K\right)}\parallel {\mathbf{u}}_i^t\right)\end{array}$$

(8)

where $f_{\mathrm{out}}$ is the output network mapping the concatenated GNN feature and action to a scalar Q-value.

3.4. Federated Aggregation with FedAge

To enable privacy-preserving knowledge sharing across distributed SUs, we adopt a federated learning framework with an age-aware aggregation strategy (FedAge). In each communication round r, each SU i trains its local model parameters ${\mathit{\theta }}_i^{\left(r\right)}$ using local experience replay buffers and sends the model update $\Delta {\mathit{\theta }}_i^{\left(r\right)}={\mathit{\theta }}_i^{\left(r\right)}-{\mathit{\theta }}^{(r-1)}$ to the central server.

The server aggregates the updates using age-aware weights:

$$\begin{array}{c}\hfill {\mathit{\theta }}^{\left(r\right)}={\mathit{\theta }}^{(r-1)}+\sum _{i=1}^N{w}_i^{\left(r\right)}·\Delta {\mathit{\theta }}_i^{\left(r\right)}\end{array}$$

(9)

where the age-aware weight $w_i^{\left(r\right)}$ is computed as:

$$\begin{array}{c}\hfill w_i^{\left(r\right)}={\displaystyle \frac{exp(-\lambda ·{\mathrm{age}}_i^{\left(r\right)})·{\parallel \Delta {\mathit{\theta }}_i^{\left(r\right)}\parallel }^{-1}}{{\sum }_{j=1}^{N}exp(-\lambda ·{\mathrm{age}}_j^{\left(r\right)})·{\parallel \Delta {\mathit{\theta }}_j^{\left(r\right)}\parallel }^{-1}}}\end{array}$$

(10)

Here, ${\mathrm{age}}_i^{\left(r\right)}$ measures the staleness of SU i’s update (i.e., the number of rounds since the last contribution), $\lambda$ is a decay parameter, and the inverse norm term penalizes excessively large updates that may indicate instability.

3.5. Attention-Based Domain Adaptation Module

To enhance cross-domain generalization, we introduce an attention-based domain adaptation module that aligns feature distributions across heterogeneous wireless environments. Given features ${\mathbf{f}}_i^{\left(s\right)}$ from source domain s and ${\mathbf{f}}_i^{\left(t\right)}$ from target domain t, the module computes cross-domain attention:

$$\begin{array}{c}\hfill {\mathbf{A}}_{s\to t}=\mathrm{softmax}\left({\displaystyle \frac{\left({\mathbf{W}}_Q{\mathbf{f}}^{\left(t\right)}\right){\left({\mathbf{W}}_K{\mathbf{f}}^{\left(s\right)}\right)}^T}{\sqrt{d_k}}}\right)\end{array}$$

(11)

$$\begin{array}{c}\hfill {\widehat{\mathbf{f}}}^{\left(t\right)}={\mathbf{A}}_{s\to t}\left({\mathbf{W}}_V{\mathbf{f}}^{\left(s\right)}\right)\end{array}$$

(12)

The adapted target feature is obtained by residual connection and layer normalization:

$$\begin{array}{c}\hfill {\tilde{\mathbf{f}}}^{\left(t\right)}=\mathrm{LayerNorm}\left({\mathbf{f}}^{\left(t\right)}+{\widehat{\mathbf{f}}}^{\left(t\right)}\right)\end{array}$$

(13)

A domain discriminator $D_{\varphi }$ is trained adversarially to classify the domain origin, while the feature encoder is trained to confuse the discriminator, encouraging domain-invariant representations:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{adv}}=-{\mathbb{E}}_s[log{D}_{\varphi }\left({\tilde{\mathbf{f}}}^{\left(s\right)}\right)]-{\mathbb{E}}_t[log(1-D_{\varphi }\left({\tilde{\mathbf{f}}}^{\left(t\right)}\right))]\end{array}$$

(14)

3.6. Overall Training Procedure

The overall training objective combines the DRL loss, federated regularization, and domain adaptation loss:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{DRL}}+{\eta }_1{\mathcal{L}}_{\mathrm{FedReg}}+{\eta }_2{\mathcal{L}}_{\mathrm{adv}}\end{array}$$

(15)

where ${\mathcal{L}}_{\mathrm{DRL}}$ is the standard DQN loss with target network, ${\mathcal{L}}_{\mathrm{FedReg}}={\parallel {\mathit{\theta }}_i-{\mathit{\theta }}^{(r-1)}\parallel }^2$ is a proximal term ensuring local models do not deviate excessively from the global model, and ${\eta }_1,{\eta }_2$ are balancing coefficients. The target network is updated every C steps with a soft update:

$$\begin{array}{c}\hfill {\mathit{\theta }}_{\mathrm{target}}\leftarrow \tau \mathit{\theta }+(1-\tau ){\mathit{\theta }}_{\mathrm{target}}\end{array}$$

(16)

where $\tau \ll 1$ is the soft update coefficient.

4. Experiments

4.1. Experimental Setup

We evaluate the proposed FedMA-DRL framework on a realistic cognitive radio network testbed. The network consists of 10 primary user (PU) channels and 15 heterogeneous secondary users (SUs) with varying transmission requirements and mobility patterns. The channel model follows Rayleigh fading with an average SNR of 15 dB. The PU activity is modeled as an ON/OFF Markov process with an average duty cycle of 40%. Each SU is equipped with a local experience replay buffer of size 10,000 and a mini-batch size of 64. The GNN module uses $K=2$ message passing rounds with hidden dimension 128. The discount factor $\gamma$ is set to 0.99, and the soft update coefficient $\tau =0.005$. We compare FedMA-DRL against five baseline methods: C-DRL (centralized DRL without federated learning), DQN, QR-DQN (Quantile Regression DQN), PPO (Proximal Policy Optimization), and R2D2 (Recurrent Replay Distributed DQN). All methods are trained for 50,000 episodes with 5 independent runs.

4.2. Main Results

Table 1 presents the main performance comparison. FedMA-DRL achieves the highest secondary user throughput of 14.87 Mbps, outperforming the second-best R2D2 by 6.8%. The collision probability is reduced to 0.038, representing a 25.5% improvement over R2D2. The energy efficiency and spectrum efficiency also show consistent improvements, confirming the effectiveness of our proposed components.

4.3. Effectiveness of FedMA-DRL

To validate the contribution of each component in FedMA-DRL, we conduct ablation studies by progressively removing key modules. Table 2 summarizes the results.

The ablation results demonstrate that each component contributes to the overall performance. Removing the GNN module causes the largest performance drop (6.4% throughput decrease), highlighting the importance of capturing topological relationships among SUs. The nonlinear reward shaping also plays a crucial role, as its removal leads to a 8.0% throughput reduction and significantly higher collision probability.

4.4. Human Evaluation

We further conduct human evaluation to assess the practical deployment quality of the spectrum access policies generated by different methods. Five domain experts in wireless communications rate each method on three criteria: spectrum utilization rationality, interference management quality, and policy interpretability. Each criterion is scored on a 1–5 Likert scale.

As shown in Table 3, FedMA-DRL receives the highest scores across all three criteria, with particularly notable improvements in interference management (4.0 vs. 3.5 for R2D2) and interpretability (3.8 vs. 3.5 for C-DRL). The attention-based domain adaptation module provides interpretable cross-domain attention maps that help experts understand the policy transfer mechanism.

4.5. Scalability Analysis

We investigate the scalability of FedMA-DRL by varying the number of secondary users from 5 to 30 while keeping the number of PU channels fixed at 10.

Figure 3 shows that FedMA-DRL maintains reasonable performance even as the number of SUs doubles from 15 to 30. The throughput decrease is gradual, and the GNN module effectively scales by leveraging the interference graph structure. The collision probability remains below 0.1 even with 30 SUs, demonstrating the robustness of our collaborative multi-agent framework.

4.6. Cross-Domain Generalization

We evaluate the cross-domain generalization capability by training FedMA-DRL on one wireless environment (Source) and testing on three different target environments with varying channel fading models, PU activity patterns, and SU mobility profiles.

Figure 4 reports the SU throughput (Mbps) on each target domain. FedMA-DRL significantly outperforms all baselines, demonstrating the effectiveness of the attention-based domain adaptation module. The performance gap is most pronounced on Target C (the most challenging domain with high mobility), where FedMA-DRL achieves 10.83 Mbps compared to 9.15 Mbps for R2D2, an 18.4% improvement.

5. Conclusion

In this paper, we proposed FedMA-DRL, a federated multi-agent deep reinforcement learning framework for joint channel selection and power control in cognitive radio networks. By integrating CTDE architecture with nonlinear reward shaping, a GNN-augmented Q-value predictor, FedAge federated aggregation, and attention-based domain adaptation, our approach effectively addresses the challenges of sparse rewards, exploration inefficiency, and poor cross-domain generalization in dynamic spectrum access. Extensive experiments demonstrated that FedMA-DRL achieves superior performance with 14.87 Mbps SU throughput, 0.038 collision probability, and improved energy and spectrum efficiency compared to existing DRL methods. Ablation studies validated the contribution of each component, while cross-domain evaluations confirmed the robustness of the domain adaptation module. Future work will explore integrating reconfigurable intelligent surfaces (RIS) and extending the framework to ultra-dense IoT scenarios with massive device connectivity.