Energy Efficient Power Allocation in Massive MIMO based on Parameterized Deep DQN

1. Introduction

With the increasing demand for mobile communications and Internet of Things technologies, wireless networks are facing increased data traffic and resource management issues owing to the rapid growth of wireless applications. Fifth-generation cellular networks have gained considerable attention for achieving spectrum efficiency and storage capacity. Massive multiple-input multiple-output (M-MIMO) networks are a reliable option to overcome data storage and capacity issues to satisfy diverse user requirements. The main concept in M-MIMO technology is to equip the base stations (BSs) with a large number (i.e., 100 or more) of wireless antennas to simultaneously serve numerous users, enabling significant improvement in spectrum efficiency [1,2].

The presence of a huge number of antennas in M-MIMO, data multiplexing and management would make MIMO transceiver optimization more challenging compared to single-antenna networks. The multi-objective nature of M-MIMO transceivers have resulted in various optimization strategies being performed in the past, including user association [3], power allocation [4], and user scheduling [5]. A joint user association and the resource allocation problem was investigated in [7, 8]. Given the non-convex multiple objective function in the M-MIMO problem, achieving the Pareto optimal solution set in multi-objective environment becomes more challenging. Recently proposed methods to solve multi-objective problems include approaches based on linear programming [9], game-theory [10], and Markov approximation [11, 12]. Success of these methods requires complete knowledge of the system, which is rarely available.

Thus, emerging machine learning (ML) is an efficient tool to solve such complex multi-objective problems. In this ML field, Reinforcement Learning (RL) is the most appropriate branch to solve a non-convex problem. In RL-based optimization methods, three major elements (i.e., agents, reward, and action) of the proposed solution enable the self-learning abilities from the environment.

The Q-learning algorithm is one of the widely used RL methods because it requires a minimum computation. It can be expressed by single equations and, does not need to know the state transition probability The RL agents maximize the long-term rewards over the current optimal reward function [13-14] using a Q-learning algorithm [14,15,16]. The agents are free to change their actions independently in a single-agent RL method, leading to a fluctuation in the overall action space, as well as action and rewards of the different agents in the process [16]. Q-learning methods have been used for power and resource allocation in heterogeneous and cellular networks [17]. However, it may be considerably difficult to handle such large state and action spaces in M-MIMO systems using Q-learning methods. To handle these issues of RL methods, deep reinforcement learning (DRL) methods are coupled with deep learning and RL to enhance the performance of RL for large scale scenario problems. Nowadays, DRL methods [18] are promising to handle these complicated objective functions. DRL methods have already been applied to several tasks, such as resource allocation, fog radio access networks, dynamic channel, access, and mobile computing [19-21].

In DRL, deep Q-network (DQN) method is mostly employed to train the agents to achieve an optimal scheme from a large state and action space. In [23], Rahimi et al. gave an algorithm of DQN which was based on deep neural networks and has been previously used in past literature data. In [23], Zhao et al. used the DRL method for the efficient management of user association and resource allocation for maximizing the network utility and maintains the quality of service (QoS) requirements. In [25-27], the authors proposed a DQN algorithm to allocate power using a multi-agent DRL method. Recent advancements DRL, particularly techniques like Deep Q-Networks (DQN), have opened up new avenues for addressing resource allocation challenges in wireless networks. However, when it comes to applying DQN to solve the combined problem of power allocation and user association, a critical step involves converting the continuous action space for power allocation into a discrete action space. This quantization process can potentially result in suboptimal power allocation decisions, limiting the overall performance. Additionally, the complexity of DQN grows significantly as the dimension of the action space increases. This exponential complexity can lead to high power consumption and slow convergence rates, which are highly undesirable in practical applications. To address these challenges, our paper introduces the use of Parameterized Deep Q-Network (PD-DQN) techniques which deal with parameterized state spaces. However, it falls short in terms of estimation capabilities and tends to produce sub-optimal policies due to its tendency to overestimate Q-values [28]. PD-DQN is well-suited for solving problems involving hybrid action spaces, making it a more efficient choice for the joint power allocation and user association problem, which uses discrete and continuous action space [25]. This hybrid approach is designed to address the challenges posed by a mixed discrete-continuous action space.

In this study, we introduced a novel approach PD-DQN algorithm [25]. The main contributions of this paper are listed in the following:

• This paper proposes a user association and power allocation problem with objective of maximizing EE in a massive MIMO network.
• To solve the power allocation problems, the action space, the state space, and the reward function have been considered. We apply the model-free DQN framework and PD-DQN to update policies in action space. We also employ the novel PD-DQN framework that is able to updating policies in a hybrid discrete-continuous action space.
• The simulation results show that the proposed user association and power allocation method based on PD-DQN perform better than DRL and RL method.

2. System Model

In this study, we considered single cell massive MIMO network which consists of N remote radio heads (RRHs) and single antenna users. Here, RRHs are connected to a baseband unit via backhaul connections. Each RRH is equipped with $M_{max}$ antennas. There are U single-antenna users served by N RRHs together operating in the same time frequency domain. It is assumed that $M_{max}$ > U. In this network, we associate each user with a single RRH [7]. The set of users is denoted by U. Figure 1 shows the network architecture based on a DRL.

It is assumed that the channel between the uth users and the nth RRH is given by

$${\mathit{h}}_{\mathit{n},\mathit{u}}=\sqrt{{\mathit{\beta }}_{\mathit{n},\mathit{u}}{\mathbf{g}}_{\mathit{n},\mathit{u}}}$$

(1)

where ${\mathit{\beta }}_{\mathit{n},\mathit{u}}$ signifies the large-scale fading coefficient, and ${\mathbf{g}}_{\mathit{n},\mathit{u}}$signifies the small-scale fading coefficient. ${\mathbf{g}}_{\mathit{n},\mathit{u}}$is also known as Rayleigh fading and the elements are independent and identically distributed (i.i.d) random variables having zero mean and unit variance[8].

The received signal of the uth user on the nth RRH can be given by [7]

$${\mathit{Y}}_{\mathit{n},\mathit{u}}=\sqrt{{\mathit{p}}_{\mathit{u}}{\mathbf{h}}_{\mathit{n},\mathit{u}}{\mathbf{w}}_{\mathit{n},\mathit{u}}}{\mathit{s}}_{\mathit{n},\mathit{u}}+{\sum }_{\mathit{j}=1,\mathit{j}\ne \mathit{u}}^{\mathit{U}}\sqrt{{\mathit{p}}_{\mathit{j}}{\mathbf{h}}_{\mathit{n},\mathit{u}}}{{\mathbf{w}}_{\mathit{n},\mathit{u}}\mathit{s}}_{\mathit{n},\mathit{j}}+{\mathbf{z}}_{\mathit{n},\mathit{u}}$$

(2)

where ${\mathit{p}}_{\mathit{u}}$ is power transmitted through uth user, ${\mathit{s}}_{\mathit{n},\mathit{u}}$is the ${\mathit{N}}_{\mathit{r}}\times 1$data symbol of the uth user on the nth RRH, ${\mathbf{w}}_{\mathit{n},\mathit{u}}$ is the ${\mathit{N}}_{\mathit{t}}\times {\mathit{N}}_{\mathit{r}}$beamforming vector of the uth user on the nth RRH which is given by ${\mathbf{w}}_{\mathit{n},\mathit{u}}=\raisebox{1ex}{${\mathbf{h}}_{\mathit{n},\mathit{u}}$}\!\left/ \!\raisebox{-1ex}{$\sqrt{\mathbb{E}\left\{{‖{\widehat{\mathbf{h}}}_{\mathit{n},\mathit{u}}‖}^2\right\}}$}\right.$and ${\mathbf{h}}_{\mathit{n},\mathit{u}}$is the channel matrix of the uth user on the nth RRH. ${\mathbf{z}}_{\mathit{n},\mathit{u}}$ is the noise vector of independent identically distributed (i.i.d.) additive complex Gaussian noise having zero mean and variance of $\mathit{\sigma }$.

Without loss of generality, we set

2.1. Power consumption

We considered the downlink phase for power consumption. The overall ${\mathit{P}}_{\mathit{c}}$ is expressed as the sum of the transmit power and fixed power consumption of the RRHs and base units denoted as ${\mathit{P}}_{\mathit{F}\mathit{I}\mathit{X}}$ and the power consumed by the components of the active antennas [5]. The total ${\mathit{P}}_{\mathit{c}}$ can then be given by

$${\mathit{P}}_{\mathit{c}}={\mathit{P}}_{\mathit{F}\mathit{i}\mathit{x}}+{\mathit{P}}_{\mathit{a}}{\sum }_{\mathit{n}=1}^{\mathit{N}}{\mathit{M}}_{\mathit{n}}+{\sum }_{\mathit{n}=1}^{\mathit{N}}{\sum }_{\mathit{u}=1}^{\mathit{U}}\frac{1}{\mathit{\nu }}{\mathit{P}}_{\mathit{n},\mathit{u}}$$

(3)

where ${\mathit{P}}_{\mathit{a}}$ denotes power assumed for active antenna and $\mathit{\nu }$ power amplifier efficiency, $\mathit{\nu }\in \left(\mathrm{0,1}\right).$

3. Problem formulation

According to the system model, the ergodic achievable rate of the uth user is given by[10]

$${\mathit{R}}_{\mathit{n},\mathit{u}}={\mathbf{log}}_2\left(1+{\mathsf{\Gamma }}_{\mathit{u},\mathit{n}}\right)$$

(4)

where ${\mathsf{\Gamma }}_{\mathit{u},\mathit{n}}$ is signal-to-interference-plus-noise ratio of uth and nth RRH and given by

$${\mathit{R}}_{\mathit{n},\mathit{u}}=\frac{{\mathit{M}}_{\mathit{n}}{\mathit{p}}_{\mathit{n}}{\mathit{\Gamma }}_{\mathit{n},\mathit{u}}}{{\sum }_{\mathit{j}=1}^{\mathit{N}}{\sum }_{\mathit{q}=1}^{\mathit{U}}{\mathit{p}}_{\mathit{j},\mathit{q}}{\mathit{\beta }}_{\mathit{j},\mathit{u}}+{\mathit{\sigma }}^2}$$

(5)

To deal with the above problem, the association between uth users and nth RRH is given by

$${\mathit{x}}_{\mathit{n},\mathit{u}}=\left\{\begin{array}{l}1,\mathit{u}\in {\mathit{L}}_{\mathit{l}}\\ 0,\mathit{u}\in \frac{\mathit{U}}{{\mathit{L}}_{\mathit{l}}}\mathit{l}=1,...\mathit{L}\end{array}\right.$$

(6)

The system energy efficiency (EE) can be expressed as

$$\underset{\mathrm{X},\mathrm{P}}{\mathbf{m}\mathbf{a}\mathbf{x}}\mathit{\eta }=\frac{{\sum }_{\mathit{u}=1}^{\mathit{U}}{\mathit{R}}_{\mathit{u}}}{{\mathit{P}}_{\mathit{F}\mathit{I}\mathit{X}}+{\mathit{p}}_{\mathit{a}}{\sum }_{\mathit{n}=1}^{\mathit{N}}{\mathit{M}}_{\mathit{n}}+{\sum }_{\mathit{n}=1}^{\mathit{N}}{\sum }_{\mathit{u}=1}^{\mathit{U}}\frac{1}{\mathit{v}}{\mathit{p}}_{\mathit{n},\mathit{u}}}$$

(7)

The optimization problem maximizing the system EE can be formulated as

$$\mathrm{P}1:\underset{\mathit{X},\mathit{P}}{\mathbf{m}\mathbf{a}\mathbf{x}}\mathit{\eta }=\frac{{\sum }_{\mathit{u}=1}^{\mathit{U}}{\mathit{R}}_{\mathit{u}}(\mathit{X},\mathit{P})}{{\mathit{P}}_{\mathit{c}}\left(\mathit{X},\mathit{P}\right)},$$

(8)

$$\mathrm{s}.\mathrm{t}\hspace{1em}\hspace{1em}\mathrm{C}1:0<{\mathit{P}}_{\mathit{u},\mathit{m}}\le {\mathit{P}}_{\mathit{u}}^{\mathbf{m}\mathbf{a}\mathbf{x}}$$

$$\mathrm{C}2:{\mathit{a}}_{\mathit{u},\mathit{m}}\in \left\{\mathrm{0,1}\right\}$$

$$\mathrm{C}3:{\sum }_{\mathit{m}}{\mathit{a}}_{\mathit{u},\mathit{m}}=1$$

$$\mathrm{C}4:{\mathit{R}}_{\mathit{u}}\ge {\mathit{R}}_{\min }$$

In the above problem, C1 denotes that the transmitted power consumption is smaller than the transmit power limit of each RRH. Constraints C2 and C3 indicate that one user can only be associated with one RRH. C4 maintains the QoS requirement of each user and signifies the lower limit of the required transmit rate of users. Problem (8) is NP-hard and is usually difficult to find a feasible solution [22]. Therefore, a multi-agent DRL approach was used to solve this problem, as described in the next section.

4. Multi-Agent DRL Optimization Scheme

The problem P1 is a non-convex problem where user association as well as power allocation approaches are involved. To solve this tractable problem, a multi-agent DQN-based RL technique was applied. The major component of the RL approach is based on the Markov decision-making process (MDP), which is a new proposed reward function, prior to the application of the multi-agent DQN approach.

4.1. Overview of RL method

In this section, we present the overview of RL. In RL the aim is to find optimal policy. The problem P1 is converted into a MDP (s, a, r, ${\mathit{P}}_{\mathit{s}{\mathit{s}}^{\mathit{n}\mathit{e}\mathit{w}}}$) similar to the existing work [26-27], where s, a, and r represent the set of state, set of action, and reward functions, respectively. ${\mathit{P}}_{\mathit{s}{\mathit{s}}^{\mathit{n}\mathit{e}\mathit{w}}}$ is the transition probability from state s to ${\mathit{s}}^{\mathit{n}\mathit{e}\mathit{w}}$ with reward r. In the DRL, these state variables are defined as follows:

State space: In problem P1, the users as agents select the BSs for communication at time t. The network consists of U agents. The state space can be expressed as

$$\mathit{s}(\mathit{t})=\left\{{\mathit{s}}_1(\mathit{t}),{\mathit{s}}_2(\mathit{t}),...{\mathit{s}}_{\mathit{u}}(\mathit{t}),...,{\mathit{s}}_{\mathit{U}}(\mathit{t})\right\}$$

(9)

Action space: At time t, the action of the agent is to control the transmit power level between the user association and BS. The action space consisting of each user can be defined as

$$\mathit{a}(\mathit{t})=\left\{{\mathit{a}}_1(\mathit{t}),{\mathit{a}}_2(\mathit{t}),..\mathit{a}{\mathit{s}}_{\mathit{u}}(\mathit{t}),...,{\mathit{a}}_{\mathit{U}}(\mathit{t})\right\}$$

(10)

Reward function: The energy efficiency of all users can be expressed as a system reward function

$$\mathit{r}(\mathit{t})={\sum }_{\mathit{u}}^{\mathit{U}}{\mathit{r}}_{\mathit{u}}(\mathit{t})={\sum }_{\mathit{u}=1}^{\mathit{U}}{\mathit{\eta }}_{\mathit{u}}(\mathit{t})$$

(11)

where is $\mathit{r}(\mathit{t})$ the reward function, which is maximized to achieve the optimal policy with interaction with the outer environment.

Therefore, within the RL framework, the problem P1 can be transformed into problem P2, as follows:

$$\mathrm{P}1:\underset{\mathrm{X},\mathrm{P},\mathrm{M}}{\max }{\mathit{r}}_{\mathit{n}}$$

(12)

where X represents the user association matrix and P denotes the power allocation vector. The agent identifies its state s(t) at time t and follows a policy $\mathit{\pi }$ to perform an action a(t) that is, $\mathit{a}(\mathit{t})=\mathit{\pi }(\mathit{s}(\mathit{t}))$ . Following this, the users communicate with the BSs and the reward function becomes $\mathit{r}\left(\mathit{t}\right)=\mathit{r}\left(\mathit{t}|\mathit{s}=\mathit{s}(\mathit{t}),\mathit{a}=\mathit{a}(\mathit{t})\right).$ Therefore, the future cumulative discounted reward at time t can be given by

$$\mathit{R}(\mathit{t})={\sum }_{\mathit{\tau }=\mathit{t}}^{\mathit{T}}{\mathit{\gamma }}^{\mathit{\tau }-\mathit{t}}\mathit{r}(\mathit{\tau })$$

(13)

where $\mathit{\gamma }$ ∈ [0, 1] denotes the discount factor for the upcoming rewards. To solve the P2, a value function for policy ${\mathit{\pi }}_{\mathit{u}}$ is defined as

$${\mathit{V}}_{\mathit{u}}^{{\mathit{\pi }}_{\mathit{u}}}=\mathit{{\rm E}}\left[{\sum }_{\mathit{\tau }=\mathit{t}}^{\mathit{T}}{\mathit{\gamma }}^{\mathit{\tau }-\mathit{t}}{\mathit{r}}_{\mathit{u}}(\mathit{\tau })|{\mathit{s}}_{\mathit{u}}^{\mathit{\tau }},{\mathit{a}}_{\mathit{u}}(\mathit{\tau })\right]$$

(14)

where E[·] denotes the expectation operator. By Markov property, the value function is defined as

$${\mathit{V}}_{\mathit{u}}^{{\mathit{\pi }}_{\mathit{u}}}=\mathit{r}({\mathit{s}}_{\mathit{u}}^{\mathit{\tau }},{\mathit{\pi }}_{\mathit{u}})+\mathit{\gamma }{\sum }_{{\mathit{s}}^{\mathit{n}\mathit{e}\mathit{w}}\in \mathit{S}}{\mathit{P}}_{\mathit{s}{\mathit{s}}^{\mathit{n}\mathit{e}\mathit{w}}}({\mathit{\pi }}_{\mathit{u}}){\mathit{V}}_{\mathit{u}}^{{\mathit{\pi }}_{\mathit{u}}}({\mathit{s}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}})$$

(15)

The Q-function when performing action ${\mathit{a}}_{\mathit{u}}(\mathit{\tau })$ in state ${\mathit{s}}_{\mathit{u}}^{\mathit{\tau }}$ with policy ${\mathit{\pi }}_{\mathit{u}}$ can be expressed as [24], that is,

$${\mathit{Q}}_{\mathit{\pi }}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)=\mathit{{\rm E}}\left[\mathit{R}\left(\mathit{\tau }\right)|{\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right]$$

(16)

The optimal Q-value function satisfies the Bellman equation [29-30] derived as

$${\mathit{Q}}_{{\mathit{\pi }}^{\mathit{*}}}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)=\mathit{r}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)+\mathit{\gamma }{\sum }_{{\mathit{s}}^{\prime }\in \mathit{S}}{\mathit{P}}_{\mathit{s}{\mathit{s}}^{\prime }}\mathit{a}(\mathit{\tau }){\mathit{V}}^{\mathit{\pi }\mathit{*}}({\mathit{s}}^{{\mathit{\tau }}^{\prime }})$$

(17)

Accordingly, the Bellman optimality equation (17) [24], ${\mathit{V}}_{\mathit{u}}^{{\mathit{\pi }}_{\mathit{u}}^{\mathit{*}}}({\mathit{s}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}})$ can be obtained as

$${\mathit{V}}_{\mathit{u}}^{{\mathit{\pi }}_{\mathit{u}}^{\mathit{*}}}({\mathit{s}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}})=\underset{{{\mathit{a}}^{\prime }}_{\mathit{u}}}{\mathit{m}\mathit{a}\mathit{x}}{\mathit{Q}}_{{\mathit{\pi }}_{\mathit{u}}^{\mathit{*}}}\left({\mathit{s}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}},{\mathit{a}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}}(\mathit{\tau })\right)$$

(18)

Adding Eq (17) and (18), we get

$${\mathit{Q}}_{{\mathit{\pi }}^{\mathit{*}}}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)=\mathit{r}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)+\mathit{\gamma }{\sum }_{{\mathit{s}}^{\prime }\in \mathit{S}}{\mathit{P}}_{\mathit{s}{\mathit{s}}^{\prime }}\mathit{a}(\mathit{\tau })\underset{{{\mathit{a}}^{\prime }}_{\mathit{u}}}{\mathit{m}\mathit{a}\mathit{x}}{\mathit{Q}}_{{\mathit{\pi }}_{\mathit{u}}^{\mathit{*}}}\left({\mathit{s}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}},{\mathit{a}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}}(\mathit{\tau })\right)$$

(19)

The update of the Q-value function is given by [14] as

$${\mathit{Q}}_{{\mathit{\pi }}^{\mathit{*}}}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)=(1-\mathit{\alpha }){\mathit{Q}}_{\mathit{\pi }}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)+\mathit{\alpha }\left[\mathit{r}\left({\mathit{s}}^{\mathit{\tau }},\mathit{a}(\mathit{\tau })\right)\right]+\mathit{\gamma }\underset{{\mathit{a}}^{\prime }}{\mathit{m}\mathit{a}\mathit{x}}{\mathit{Q}}_{{\mathit{\pi }}^{\prime }}\left({\mathit{s}}^{\mathit{\tau }},{\mathit{a}}^{\prime }({\mathit{\tau }}^{\prime })\right)$$

(20)

where $\mathit{\alpha }$ is learning rate scaled between 0 and 1 and updating speed of ${\mathit{Q}}_{{\mathit{\pi }}_{\mathit{u}}}\left({\mathit{s}}_{\mathit{u}}^{\mathit{\tau }},{\mathit{a}}_{\mathit{u}}(\mathit{\tau })\right)$ The RL algorithm shows good performance if size of states is small. In case of high dimensional state space, classical RL approaches fail to perform. Some states are not sampled because of the high dimensional state space and require several restrictions. First, the convergence rate might become slow and storage of lookup table becomes impractical. Thus, the use of the DRL method was explored to solve the problem with the large space.

4.2. Multi-agent DQN frameworks

In contrast to the classical Q-learning approach, the author in [ 23 ] proposed the DQN method which was basically DRL method. This DQN method relies on two components, e.g., replay memory and target network. The agent stores transitions $\left({\mathit{s}}_{\mathit{u}}^{\mathit{\tau }},{\mathit{a}}_{\mathit{u}}(\mathit{\tau }),{\mathit{r}}_{\mathit{u}}(\mathit{\tau }),{\mathit{s}}_{\mathit{u}}^{{\mathit{\tau }}^{\mathit{n}\mathit{e}\mathit{w}}}\right)$in a replay memory D. Then extract this transition from memory D by using random sampling to compute Q-value function. The agent uses the memory D in a part of mini-batch to train the Q-network and then a gradient descent method is applied to update the weight parameter $\mathit{\theta }$ of behavior network.

$${\mathit{\pi }}_{\mathit{u}}=\underset{{\mathit{a}}^{\prime }}{\mathbf{max}{\mathit{Q}}_{{\mathit{\pi }}_{\mathit{u}}}^{\mathit{*}}\left({\mathit{s}}_{\mathit{u}}^{\mathit{\tau }},{\mathit{a}}_{\mathit{u}}^{\mathit{n}\mathit{e}\mathit{w}}(\mathit{\tau })\right)}$$

(21)

In the DQN method, the types of networks are included, that is, DQN sets the ${\mathit{\theta }}_{\mathrm{target}}$ target networks. The learning model calculates the target value${\mathit{y}}_{\mathit{j}}$with a weight parameter${\mathit{\theta }}_{\mathrm{target}}$for a certain time t, which can mitigate the volatility of the learning scheme. During the learning process, after several iterations H the weight parameter θ is synchronized with the target network $\mathit{\theta }\to {\mathit{\theta }}_{\mathrm{target}}$. The agent utilizes a greedy random policy means that the agent randomly selects an action ${\mathit{a}}_{\mathit{u}}(\mathit{\tau })$ parameter $\mathit{\theta }$ for the behavior network. Consequently, $\mathit{a}(\mathit{\tau })$ value and $\mathit{\theta }$ are updated iteratively using the minimum loss function [31]:

$$\mathit{L}(\mathit{\theta })=\sum \left[{\mathit{y}}_{\mathit{j}}-{\mathit{Q}}_{{\mathit{\pi }}_{\mathit{u}}}({\mathit{s}}_{\mathit{u}},{\mathit{a}}_{\mathit{u}};\mathit{\theta })\right],$$

(22)

where ${\mathit{y}}_{\mathit{j}}={\mathit{r}}_{\mathit{j}}+\mathit{\gamma }\underset{{\mathit{a}}_{\mathit{u}}^{\mathit{n}\mathit{e}\mathit{w}}(\mathit{j})}{\mathit{m}\mathit{a}\mathit{x}}{\mathit{Q}}_{{\mathit{\pi }}_{\mathit{u}}}({\mathit{s}}_{\mathit{u}}^{{\mathit{j}}^{\mathit{n}\mathit{e}\mathit{w}}},{\mathit{a}}_{\mathit{u}}^{\mathit{n}\mathit{e}\mathit{w}}(\mathit{j});{\mathit{\theta }}_{\mathrm{target}})$The proposed DQN algorithm for user association and power allocation is shown in Algorithm 1.

4.3. Parameterized Deep Q-Network Algorithm

The combined user association and power allocation procedure in a hybrid action space can be solved via parameterization, but it still has generalization problems. In order to solve this problem, we used the epsilon greedy exploration method, which enables the DQN to explore a wide variety of states and actions and improves generalization. In a hybrid action space, the Q-value is recast as $\mathit{Q}(\mathit{s},\mathit{a})=\mathit{Q}(\mathit{a},\mathit{\beta },\mathit{\chi })$ where $\mathit{\beta }$ denotes a discrete action and $\mathit{\chi }$ denotes a continuous action. The reward is defined as parameterized double DQN with replay buffer EE. Whether UE is associated with BS or not, the user association will only have the two discrete values $\mathit{\beta }$ = 0 and 1. On the other hand, $\mathit{\chi }$ is a matrix that represents various levels of power distribution. PD-DQN for user association and power allocation algorithm is presented in Algorithm 2.

5. Simulation Result

In this section, the results of the simulation with the DRL algorithms are presented. We considered a distributed M-MIMO with three RRHs equipped with 300 antennas in the cell with diameter of 2000 m. We consider K= 20 randomly distributed users within the cell. The PC of each RRH is set to 10000 mW.

The power consumed by component of active antennas is 200 mW, and the power ampli fier efficiency, ν is 0.25. We assumed a transmission bandwidth of 10 MHz [7]. The other parameters are given in Table 1. We consider the Hata-COST231 propagation model [8]. The large-scale-fading β in eq (1) borrowed from literature [8] which is given by $10{\mathit{log}}_{10}\left({\mathit{\beta }}_{\mathit{n},\mathit{u}}\right)=\mathit{P}{\mathit{L}}_{\mathit{n},\mathit{u}}+\mathit{\Omega }$ where PL is path loss and $\Omega$ represents standard deviation. Here, d_n,u is the distance between uth users and nth RRH.

Figure 2 shows the energy efficiency of the proposed DRL algorithm and RL algorithm. Figure 2 gives two observations indicate that the EE achieved by the DRL method outperforms the RL methods. As the number of episodes increases up to 50, the system EE increases and tends to converge after 250 episodes for both schemes. Additionally, the learning speed of the Q-learning method is lower than that of the multi-agent DQN algorithm. For the Q-learning method, there is a slight improvement in the system EE at episode 120, whereas in DRL approach, the system EE tends to be stable at episode 257. The EE is unstable at the beginning as seen in the DRL scheme, and the stability increases as the episodes increase, and thereafter increases slowly. This is because the agent selects actions in a random manner and stores the transition information in D.

Figure 3 shows the EE versus the number of user fix M= 20. From the figure we can see that, the EE generally first increases with K from 5 to 45 and then decreases flatten. This is due to the fact that when scheduling more users, more RRHs are activated to serve users creating more interference noise. Furthermore, proposed DRL algorithm performs superior to QL.

Figure 4 compares the EE performance at different discounted factors, γ = {0.1, 0.5, and 0.8}. It can be observed that a lower discount factor results in higher EE through different discount factors. Figure 4 illustrates that the PD-DQN methods optimize the user association and power allocation. Moreover, when the number of epochs increases, the EE performance of each user is better at a different discount factor.

From the figure, we can observe that PDQN consistently outperforms both DQN and RL in terms of energy efficiency. The EE values achieved by PDQN show a steady increase, starting from 1.5 and reaching 5.25, indicating significant improvement. DQN and RL also exhibit improvements, but their EE values remain below that of PDQN. In terms of percentage improvement, PDQN surpasses DQN by an average of around 35%, while PDQN outperforms RL by approximately 40%. Additionally, DQN exhibits a slight advantage over RL, with an average improvement of about 5%.