Joint Computing Offloading, Task Caching and Resource Allocation Based on TD3 Algorithm in Cache-Assisted Vehicular NOMA-MEC Networks

1. Introduction

With the rapid development of information and communication technologies, the data traffic generated by vehicles (mobile devices, MDs) has also significantly increased [1]. For wireless communication networks, more spectrum resources are required for data traffic transmission [2]. In addition, higher computing power is required by MDs for supporting large amounts of task calculation. However, due to the limited battery capacity of MDs, it may be challenging to process these computation tasks for them. By deploying edge computing servers at base stations (BSs), mobile edge computing (MEC) can support MDs in processing tasks at the adjacent edge servers [3,4]. Compared with cloud computing (CC), which requires tasks to be uploaded to a remote cloud, MEC can provide additional computing resources for MDs within its coverage area and thus reduce the computing overhead of MDs [5,6,7,8,9].

Although the edge servers can reduce the computing overhead of MDs by providing more computing resources, the extra delay and energy consumption caused by offloading tasks through wireless channels cannot be ignored, especially for high-size computation tasks. In order to further reduce the delay and energy consumption caused by offloading tasks, edge caching technology is also introduced into MEC networks. By caching tasks of MDs at edge servers in advance, the overhead caused by offloading tasks can be greatly reduced [10,11,12,13].

To upload tasks from MDs to edge servers, orthogonal multiple access (OMA) is often widely used, but it may be greatly challenging to provide a high transmission rate and support massive connections. As another type of resource utilization manners, non-orthogonal multiple access (NOMA) technologies can let multiple users share the same frequency bands, achieve higher spectral efficiency and support massive connections [14,15,16,17]. It is evident that NOMA is a good type of resource utilization manners for reducing the cost of task transmission in MEC networks.

Although the application of caching and NOMA technologies in MEC networks can bring lower delay and energy consumption, such a framework will make the design of computation offloading and edge caching schemes more complex. To the best of our knowledge, until now, how to jointly perform the device association, computation offloading, edge caching, subchannel selection and resource allocation is still an important and open topic in cache-assisted NOMA-MEC networks.

1.1. Related Work

So far, there exists a lot of work done on joint computation offloading and resource optimization in NOMA-MEC networks. In [14], joint radio and computation resource allocation were optimized to maximize the offloading energy efficiency in NOMA-MEC-enabled IoT networks, and a solution based on a multi-layer iterative algorithm was proposed. In [15], local computation resource, offloading ratio, uplink transmission time and power, and subcarrier assignment were jointly optimized to minimize the sum of weighted energy consumed by users in NOMA-MEC networks, and some effective iterative algorithms were designed for single-user and multi-user cases. In [18], joint task offloading, power allocation, and computing resource allocation were optimized to achieve delay minimization using a deep reinforcement learning (DRL) algorithm in NOMA-MEC networks. In [19], joint optimization of offloading decisions, local and edge computing resource allocation, and power and subchannel allocation were realized to minimize energy consumption in heterogeneous NOMA-MEC networks, and an effective iterative algorithm was designed. In [20], the power and computation resource allocation were jointly optimized to minimize overall computation and transmission delay for massive MIMO and NOMA-assisted MEC systems, and a solution based on an interior-point algorithm was given. In [21], the channel resource allocation and computation offloading policy was jointly optimized to minimize the sum of weighted energy and latency in NOMA-MEC networks, and some efficient solutions were found using a DRL algorithm based on actor-critic and deep Q-network (DQN) methods.

To further reduce the offloading delay and energy consumption, edge caching technology is introduced into conventional MEC networks. Such a framework has attracted more and more attention. In [22], the offloading and caching decisions, uplink power and edge computing resources were jointly optimized to minimize the sum of weighted local processing time and energy consumption in two-tier cache-assisted MEC networks, and a distributed collaborative iterative algorithm was proposed. In [23], a problem of adaptive request scheduling and cooperative service caching was studied in cache-assisted MEC networks. After formulating the optimization problems as partially observable Markov decision process (MDP) problems, an online DRL algorithm was proposed to improve the service hitting ratio and latency reduction rate. In [24], optimal offloading and caching strategies were established to minimize overall delay and energy consumption of all regions using a deep deterministic policy gradient (DDPG) framework in cache-assisted multi-region MEC networks. In [25], joint MD association and resource allocation were done to minimize the sum of MDs’ weighted delay in heterogeneous cellular networks with MEC and edge caching functions, and an effective iterative algorithm was developed using coalitional game and convex optimization theorems.

To enhance spectral efficiency and support massive connections, NOMA technology has attracted increasing attention in cache-assisted MEC networks. In [26], joint optimization of offloading and caching decisions and computation resource allocation was done to maximize long-term reward in cache-assisted NOMA-MEC networks under the predicted task popularity, and single-agent and multi-agent Q-learning algorithms were proposed to find feasible solutions. In [27], joint optimization of offloading and caching decisions was done to minimize the system delay in cache-assisted NOMA-MEC networks, and a multi-agent DQN algorithm was used for finding efficient solutions under the predicted popularity. In [29], local task processing time was minimized by jointly optimizing offloading and caching decisions and the allocation of edge computing resources and uplink power in cache-assisted NOMA-MEC networks with single BS, and blocking successive upper-bound minimization method was utilized to achieve efficient solutions.

Although the framework of cache-assisted (vehicular) NOMA-MEC networks can greatly reduce the task processing delay and energy consumption and support massive connections, there exist very few relevant efforts. Unlike the mentioned-above work, we jointly optimize the edge computing resource allocation, subchannel selection, device association, offloading and caching decisions for the cache-assisted vehicular NOMA-MEC networks with multiple BSs, minimizing the energy consumed by MDs under the latency and resource constraints. In addition, unlike existing efforts, we develop an effective dynamic joint computation offloading and task caching algorithm based on the twin delayed deep deterministic policy gradient algorithm (TD3) to find efficient solutions, which is named as TD3-based offloading (TD3O) algorithm.

1.2. Contribution and Organization

In this paper, we jointly optimize the edge computing resource allocation, subchannel selection, device association, offloading and caching decisions in cache-assisted vehicular NOMA-MEC networks, minimizing the energy consumed by MDs under the latency and resource constraints. Specifically, the main contributions and work of this paper can be listed as follows.

• Edge computing resource allocation, subchannel selection, device association, computation offloading and edge caching are jointly performed in cache-assisted vehicular NOMA-MEC networks. To the best of our knowledge, such work that concerns subchannel selection should be a new investigation for the cache-assisted vehicular NOMA-MEC networks with multi-server scenarios.
• Formulating a problem of jointly optimizing the edge computing resource allocation, subchannel selection, device association, offloading and caching decisions in cache-assisted vehicular NOMA-MEC networks. Its goal is to minimize the energy consumed by MDs under the constraints of latency, computing resources, caching capacity, the number of MDs associated with each BS, and the number of MDs associated with each subchannel. As far as we know, such an optimization problem should be a new concentration in cache-assisted vehicular NOMA-MEC networks.
• Designing effective algorithms to find feasible solutions to the formulated problem. Considering that the formulated problem is in a mixed-integer nonlinear multi-constraint form, a simple map between actions and actual policies in a conventional twin delayed deep deterministic policy gradient (TD3) algorithm cannot be well applied. In addition, too large an action space will cause the TD3 algorithm to fail to search for correct actions and thus fail to converge. In view of these, we develop an effective TD3O algorithm integrating with the AT algorithm to solve the formulated problem. Moreover, in order to solve this problem in a non-iterative manner, an effective heuristic algorithm (HA) is also designed.
• Performance analyses of the designed algorithms. Some analyses are made for the computation complexity and convergence of the designed algorithms in detail. In addition, some meaningful simulation analyses are also made by introducing other benchmark algorithms for comparison, and some good results and insights are achieved.

The rest of the paper is organized as follows. Section 2 introduces the system model. Section 3 formulates a problem of minimizing local energy consumption in cache-assisted vehicular NOMA-MEC networks. Section 4 designs the HA and TD3O algorithm. Section 5 gives the computation complexity and convergence analyses for the designed algorithms. Section 6 investigates the performance of the designed algorithms through simulation. Section 7 gives conclusions and discussions.

2. System Model

2.1. Network Model

Figure 1 shows the cache-assisted vehicular NOMA-MEC networks. In such networks, there exist M MDs, and the index set of them is denoted as $\mathcal{M}=\{1,2,\cdots ,M\}$; B BSs are deployed, and the index set of them is given by $\mathcal{I}=\left\{1,2,\cdots ,I\right\}$. In addition, each BS is equipped with one edge computing server and one edge caching server, and these BSs connect to each other through wired links. We assume that each MD has one computation task at any timeslot, which can be processed by itself, its associated BS or another auxiliary BS selected by this associated BS. When tasks have been cached at BSs used for processing them, they don’t need to be uploaded to these BSs; when the associated BSs have not cached tasks, MDs need to upload tasks to these BSs; when the auxiliary BSs have not cached tasks, the associated BSs need to upload tasks to their selected auxiliary BSs.

Assume that the association index between MD m and BS i is $x_{m,i}\in \left\{0,1\right\}$, where $\mathbf{X}=\left\{x_{m,i}\left|\forall m\in \mathcal{M},\forall i\in \mathcal{I}\right.\right\}$. $x_{m,i}=1$ if MD m is associated with BS i, $x_{m,i}=0$ otherwise. In addition, we assume that the caching index of task of MD m at BS i is denoted as $y_{m,i}\in \left\{0,1\right\}$, where $\mathbf{Y}=\left\{y_{m,i}|\forall m\in \mathcal{M},\forall i\in \mathcal{I}\right\}$. $y_{m,i}=1$ if the task of MD m is cached at BS i, $y_{m,i}=0$ otherwise. We also assume that the offloading (execution) index of task of MD m at BS i is denoted as $u_{m,i}$, where $\mathbf{U}=\left\{u_{m,i}|\forall m\in \mathcal{M},\forall i\in \mathcal{I}\right\}$. $u_{m,i}=1$ if the task of MD m is executed at BS i, $u_{m,i}=0$ otherwise. At last, we assume that the association index between MD m and subchannel k of BS i is denoted as $z_{m,i,k}$, where $\mathbf{Z}=\left\{z_{m,i,k}|\forall m\in \mathcal{M},\forall i\in \mathcal{I},\forall k\in \mathcal{K}\right\}$. If $x_{m,i}\left(1-y_{m,i}\right)\left(1-y_{m,\overline{i}}\right)u_{m,\overline{i}}=1$ or $x_{m,i}\left(1-y_{m,i}\right)u_{m,i}=1$ under $\overline{i}\ne i$, MD m can select (be associated with) some subchannel k of BS i, which means $z_{m,i,k}=1$. Otherwise, the subchannel k of BS i cannot be selected by MD m, which means $z_{m,i,k}=0$.

2.2. Communication Model

In this paper, the system bandwidth W is divided into K subchannels with equal bandwidth, which are indexed by $\mathcal{K}=\left\{1,2,\cdots ,K\right\}$. These subchannels can be shared by different MDs through NOMA manner. Significantly, each MD can occupy at most one subchannel, the number of MDs selecting each subchannel cannot exceed the upper limit $\rho$, and the number of MDs associated with any BS that need to upload tasks should be less than or equal to the number of subchannes K [27].

As revealed in [28], the channel gains of MDs sharing the same subchannel of a BS should be sorted in descending order at first, and then the uplink NOMA signals received by this BS can be decoded in this order. We assume that ${\mathcal{M}}_k^{SC}$ is the set of MDs selecting subchannel k, and $o_{m,i,k}$ represents the sequence number of channel gain between MD m and BS i on subchannel k. When MD i and MD m access the subchannel k of BS i simultaneously, and the channel gain $h_{j,i,k}$ between MD j and BS i on subchannel k is lower than the channel gain $h_{m,i,k}$ between MD m and BS i on subchannel k, $o_{j,i,k}<o_{m,i,k}$ is satisfied. Then, the signal of MD m is decoded, but the signal of MD j will be treated as noise. Therefore, when MD m selects subchannel k of BS i, its uplink data rate $r_{m,i,k}$ can be given by

$$r_{m,i,k}=W{log}_2\left(1+p_m{h}_{m,i,k}/\left({\Gamma }_{m,i,k}+{\sigma }^2\right)\right)/K,$$

(1)

where ${\Gamma }_{m,i,k}={\sum }_{j\in {\mathcal{M}}_k^{sc}/\left\{m\right\}:o_{j,i,k}<o_{m,i,k}}{p}_j{h}_{j,i,k}$ is the interference caused by other MDs (excluding MD m) sharing subchannel k of BS i through NOMA manner; $p_m$ is the transmission power of MD m; ${\sigma }^2$ is the noise power.

2.3. Caching and Offloading Models

In this paper, we assume that any MD m has a delay-sensitive task denoted as ${\mathcal{L}}_m=\left\{d_m,c_m,{\tau }_m^{max}\right\}$ at each timeslot, where $d_m$ is the data size of task of MD m, $c_m$ is the number of CPU cycles required to complete one-bit task, and ${\tau }_m^{max}$ is the maximum task processing time (delay) of MD m.

Figure 2 illustrates the caching and offloading models. At each timeslot, BSs precache the tasks for processing at the next timeslot. When MD m is associated with BS i, it first checks whether the associated BS has cached the corresponding task. If ${\sum }_{i\in \mathcal{I}}{u}_{m,i}=0$, the task of MD m is calculated by itself, e.g., MD 1 in Figure 2; if $x_{m,i}{y}_{m,i}{u}_{m,i}=1$, the task of MD m can be directly calculated at its associated BS i and the results will be fed back from BS i to MD m, e.g., MD 2 in Figure 2; if $x_{m,i}{y}_{m,i}\left(1-y_{m,\overline{i}}\right)u_{m,\overline{i}}=1$, the task of MD m is offloaded from its associated BS i to another auxiliary BS $\overline{i}\ne i$ for computing through a wired link, e.g., MD 3 in Figure 2; if $x_{m,i}{y}_{m,i}{y}_{m,\overline{i}}{u}_{m,\overline{i}}=1$, the task of MD m can be directly calculated at auxiliary BS $\overline{i}\ne i$, e.g., MD 4 in Figure 2; if $x_{m,i}\left(1-y_{m,i}\right)u_{m,i}=1$, the task of MD m will be offloaded to its associated BS i for computing, e.g., MD 5 in Figure 2; if $x_{m,i}\left(1-y_{m,i}\right)\left(1-y_{m,\overline{i}}\right)u_{m,\overline{i}}=1$, the task of MD m first needs to be offloaded to its associated BS i, and then it is transmitted from this BS to another auxiliary BS $\overline{i}\ne i$ for computing through a wired link, e.g., MD 6 in Figure 2; if $x_{m,i}\left(1-y_{m,i}\right)y_{m,\overline{i}}{u}_{m,\overline{i}}=1$, the task of MD m can be directly calculated at auxiliary BS $\overline{i}\ne i$, e.g., MD 4 in Figure 2.

2.3.1. Local Computing

If ${\sum }_{i\in \mathcal{I}}{u}_{m,i}=0$ is satisfied, the task of MD m should be executed locally, and the processing delay and energy consumption are respectively given by

$${\tau }_m^{\mathrm{loc}}=c_m{d}_m/\phantom{c_m{d}_m{f}_m^{\mathrm{loc}}}{f}_m^{\mathrm{loc}},$$

(2)

$${\epsilon }_m^{\mathrm{loc}}=\xi c_m{d}_m{\left(f_m^{\mathrm{loc}}\right)}^2,$$

(3)

where $f_m^{\mathrm{loc}}$ is the computing capacity of MD m, and $\xi$ is an energy-consumption coefficient depending on the hardware architecture.

2.3.2. Task Transmission

If $x_{m,i}\left(1-y_{m,i}\right)\left(1-y_{m,\overline{i}}\right)u_{m,\overline{i}}=1$ or $x_{m,i}\left(1-y_{m,i}\right)u_{m,i}=1$ is satisfied under $\overline{i}\ne i$, the task of MD m should be respectively uploaded to BS $\overline{i}$ or i for execution through NOMA manner. Then, the uploading delay and energy consumption of MD m are respectively given by

$${\tau }_m^{\mathrm{trs}}={\sum }_{i\in \mathcal{I}}{\sum }_{k\in \mathcal{K}}{z}_{m,i,k}{d}_m/r_{m,i,k},$$

(4)

$${\epsilon }_m^{\mathrm{trs}}=p_m{\tau }_m^{\mathrm{trs}}.$$

(5)

In addition, if $x_{m,i}\left(1-y_{m,i}\right)\left(1-y_{m,\overline{i}}\right)u_{m,\overline{i}}=1$ or $x_{m,i}{y}_{m,i}\left(1-y_{m,\overline{i}}\right)u_{m,\overline{i}}=1$ is satisfied under $\overline{i}\ne i$, the task of MD m should be transmitted from its associated BS i to auxiliary BS $\overline{i}$ through a wired link, and the corresponding delay is given by

$${\tau }_m^{\mathrm{bh}}=d_m/\phantom{d_m{r}^{\mathrm{bh}}}{r}^{\mathrm{bh}},$$

(6)

where $r^{\mathrm{bh}}$ is the backhualing rate between any two BSs.

In this paper, we mainly concentrate on the energy consumption of MDs but not the energy consumed by BSs. In addition, the downlink transferring delay of results is often ignored since they are fairly small [30].

2.3.3. Edge Computing

When MD m executes its task at BS i, the task processing time at this BS can be given by

$${\tau }_{m,i}^{\mathrm{exe}}=c_m{d}_m/\phantom{c_m{d}_m{f}_{m,i}}{f}_{m,i},$$

(7)

where $f_{m,i}$ is the computing capacity allocated to MD m by BS i.

Then, the total time used for processing the task of MD m can be given by

$$\begin{array}{cc}\hfill {\tau }_m^{\mathrm{tot}}=& {\sum }_{i\in \mathcal{I}}(\left(1-{\sum }_{i\in \mathcal{I}}{u}_{m,i}\right){\tau }_m^{\mathrm{loc}}\hfill \\ \hfill & +x_{m,i}\left(1-y_{m,i}\right){\sum }_{\overline{i}\in \mathcal{I}\setminus \left\{i\right\}}{u}_{m,\overline{i}}\left(1-y_{m,\overline{i}}\right){\tau }_m^{\mathrm{trs}}\hfill \\ \hfill & +x_{m,i}\left(1-y_{m,i}\right){\sum }_{\overline{i}\in \mathcal{I}\setminus \left\{i\right\}}{u}_{m,\overline{i}}\left(1-y_{m,\overline{i}}\right){\tau }_m^{\mathrm{bh}}\hfill \\ \hfill & +x_{m,i}\left(1-y_{m,i}\right){\sum }_{\overline{i}\in \mathcal{I}\setminus \left\{i\right\}}{u}_{m,\overline{i}}{\tau }_{m,\overline{i}}^{\mathrm{exe}}\hfill \\ \hfill & +x_{m,i}{y}_{m,i}{\sum }_{\overline{i}\in \mathcal{I}\setminus \left\{i\right\}}{u}_{m,\overline{i}}\left(1-y_{m,\overline{i}}\right){\tau }_m^{\mathrm{bh}}\hfill \\ \hfill & +x_{m,i}{y}_{m,i}{\sum }_{\overline{i}\in \mathcal{I}\setminus \left\{i\right\}}{u}_{m,\overline{i}}{\tau }_{m,\overline{i}}^{\mathrm{exe}}\hfill \\ \hfill & +x_{m,i}\left(1-y_{m,i}\right)u_{m,i}\left({\tau }_m^{\mathrm{trs}}+{\tau }_{m,i}^{\mathrm{exe}}\right)\hfill \\ \hfill & +x_{m,i}{y}_{m,i}{u}_{m,i}{\tau }_{m,i}^{\mathrm{exe}}).\hfill \end{array}$$

(8)

On the right side of equality sign in (8), the 1st item represents the local executing time; the 2nd item is the time used for uploading the task from MD m to the associated BS i that doesn’t cache this task and further transmits it to auxiliary BS for computing; the 3rd item is the time used for transmitting task from the associated BS i to another auxiliary BS, where these two BSs don’t cache this task; the 4th item is the time used for executing the task of MD m at an auxiliary BS, where the associated BS doesn’t cache this task; the 5th item is the time used for transmitting task from the associated BS i to another auxiliary BS, where the associated BS caches this task but the auxiliary BS doesn’t; the 6th item is the time used for executing task of MD m at an auxiliary BS, where the associated BS caches this task; the 7th item includes the time used for transmitting task from MD m to the associated BS i that doesn’t cache this task, and the time used for executing the task of MD m at this BS; the 8th item is the time used for executing the task of MD m at the associated BS i that caches this task.

Then, the total local energy consumption used for processing the task of MD m can be given by

$$\begin{array}{cc}\hfill {\epsilon }_m^{\mathrm{tot}}=& {\sum }_{i\in \mathcal{I}}(\left(1-{\sum }_{i\in \mathcal{I}}{u}_{m,i}\right){\epsilon }_m^{\mathrm{loc}}\hfill \\ \hfill & +x_{m,i}\left(1-y_{m,i}\right){\sum }_{\overline{i}\in \mathcal{I}\setminus \left\{i\right\}}{u}_{m,\overline{i}}\left(1-y_{m,\overline{i}}\right){\epsilon }_m^{\mathrm{trs}}\hfill \\ \hfill & +x_{m,i}\left(1-y_{m,i}\right)u_{m,i}{\epsilon }_m^{\mathrm{trs}}).\hfill \end{array}$$

(9)

On the right side of equality sign in (9), the 1st item represents the local executing energy consumption; the 2nd item is the energy consumption caused by offloading the task from MD m to its associated BS i that further transmits this task to auxiliary BS $\overline{i}\ne i$ for computing; the 3rd item is the energy consumption caused by transmitting task from MD m to the associated BS i that doesn’t cache this task.

3. Problem Formulation

Until now, we can formulate a problem of minimizing local energy consumption at each period. Specifically, under the constraints of latency, computing resources, caching capacity, the number of MDs associated with each BS, and the number of MDs associated with each subchannel, we jointly optimize the edge computing resource allocation, subchannel selection, device association, offloading and caching decisions to minimize the energy consumed by MDs in cache-assisted vehicular NOMA-MEC networks. Mathematically, it is formulated as

$$\begin{array}{cc}\hfill \mathrm{P}1:& \underset{\mathbf{X},\mathbf{Y},\mathbf{U},\mathbf{Z},\mathbf{F}}{min}{\sum }_{m\in \mathcal{M}}{\epsilon }_m^{\mathrm{tot}}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& C_1:{\tau }_m^{\mathrm{tot}}\le {\tau }_m,\forall m\in \mathcal{M},\hfill \\ \hfill & C_2:{\sum }_{i\in \mathcal{I}}{x}_{m,i}=1,\forall m\in \mathcal{M},\hfill \\ \hfill & C_3:{\sum }_{i\in \mathcal{I}}{\sum }_{k\in \mathcal{K}}{z}_{m,i,k}\le 1,\forall m\in \mathcal{M},\hfill \\ \hfill & C_4:{\sum }_{m\in \mathcal{M}}{\sum }_{k\in \mathcal{K}}{z}_{m,i,k}\le K,\forall i\in \mathcal{I},\hfill \\ \hfill & C_5:{\sum }_{i\in \mathcal{I}}{u}_{m,i}\le 1,\forall m\in \mathcal{M},\hfill \\ \hfill & C_6:x_{m,i}\in \left\{0,1\right\},\forall m\in \mathcal{M},\forall i\in \mathcal{I},\hfill \\ \hfill & C_7:y_{m,i}\in \left\{0,1\right\},\forall m\in \mathcal{M},\forall i\in \mathcal{I},\hfill \\ \hfill & C_8:z_{m,i,k}\in \left\{0,1\right\},\forall m\in \mathcal{M},\forall i\in \mathcal{I},\forall k\in \mathcal{K},\hfill \\ \hfill & C_9:u_{m,i}\in \left\{0,1\right\},\forall m\in \mathcal{M},\forall i\in \mathcal{I},\hfill \\ \hfill & C_{10}:{\sum }_{m\in \mathcal{M}}{y}_{m,i}{d}_m\le {\vartheta }_i,\forall i\in \mathcal{I},\hfill \\ \hfill & C_{11}:{\sum }_{m\in \mathcal{M}}{\sum }_{i\in \mathcal{I}}{z}_{m,i,k}\le \rho ,\forall k\in \mathcal{K},\hfill \\ \hfill & C_{12}:{\sum }_{m\in \mathcal{M}}{u}_{m,i}{f}_{m,i}\le f_i^{\mathrm{BS}},\forall i\in \mathcal{I},\hfill \end{array}$$

(10)

where $\mathbf{F}=\left\{f_{m,i}\left|\forall m\in \mathcal{M},\forall i\in \mathcal{I}\right.\right\}$; the constraint $C_1$ gives the maximum task processing time of MD m; $C_2$ and $C_6$ indicate that any MD m just can select only one BS; $C_3$ and $C_8$ indicate that any MD m can occupy at most one subchannel; $C_4$ and $C_8$ mean that the number of MDs selecting any BS who need to upload tasks should be less than or equal to the number of subchannels; $C_5$ and $C_9$ mean that any MD m can select at most one BS to execute its task; $C_7$ and $C_{10}$ indicate that the data size of tasks cached at BS i doesn’t exceed the caching capacity ${\vartheta }_i$ of this BS; $C_8$ and $C_{11}$ show that the number of MDs selecting a subchannel cannot exceed its upper limit; $C_7$ and $C_{12}$ reveal that the total computing capacity allocated to MDs by BS i cannot exceed the computing capacity of this BS.

4. Algorithm Design

As previously mentioned, the optimization problem P1 refers to minimizing local energy consumption within a period. In view of this, we adopt the DRL algorithm to solve it. DRL is based on MDP, which implements the environment-based output of agent policy in MDP through neural networks, maximizing certain rewards. Considering that the overestimation of some conventional DRL algorithms (e.g., DQN and DDPG), the TD3 algorithm has been widely advocated [31,32].

Evidently, the problem P1 has both continuous and discrete variables and it is in a highly-complex form, a simple map between actions and actual policies in a conventional TD3 algorithm cannot be well applied to this problem. Considering that too large action space will cause the TD3 algorithm to fail to search for correct actions and thus fail to converge, we develop an effective TD3O algorithm integrating with the AT algorithm to solve the problem P1.

Considering that the optimization problem P1 needs to be tackled within a period, in order to apply TD3O to the problem P1, such a period is divided into T timeslots and denoted as $\mathcal{T}=\left\{1,2,\cdots ,T\right\}$. Furthermore, the problem of joint computing offloading, task caching and resource allocation is described as a MDP, the state space, action space and reward function are defined as follows.

❶ State space: At each timeslot, the state space contains the information used for decisions made by the network. Here, the state $s_t$ at timeslot t can be denoted as $s_t=\{\overline{\mathbf{D}}\left(t+1\right),\overline{\mathbf{Y}}\left(t\right)\}$. The detailed definitions can be found as follows.

• $\overline{\mathbf{D}}\left(t+1\right)=\left\{{\overline{d}}_m\left(t+1\right)\left|\forall m\in \mathcal{M}\right.\right\}$ are the standardized data sizes of tasks of MDs at timeslot $t+1$, where

  $${\overline{d}}_m\left(t\right)=\frac{d_m\left(t\right)-d^{min}\left(t\right)}{d^{max}\left(t\right)-d^{min}\left(t\right)},$$

  (11)
  $d^{min}\left(t\right)$ is the minimum data size of tasks of all MDs at timeslot t, and $d^{max}$ is the maximum data size of tasks of all MDs at timeslot t.
• $\overline{\mathbf{Y}}\left(t\right)=\left\{{\overline{y}}_m\left(t\right)\left|\forall m\in \mathcal{M}\right.\right\}$ are the task caching decision factors at BSs at timeslot t, where ${\overline{y}}_m\in \left[0,1\right]$.

❷ Action space: At each timeslot, the action space refers to the decisions made by the network according to the state $s_t$. The action $a_t$ at timeslot t can be denoted as $a_t=\left\{\overline{\mathbf{X}}\left(t\right),\overline{\mathbf{Y}}\left(t+1\right),\overline{\mathbf{Z}}\left(t\right),\overline{\mathbf{U}}\left(t\right),\overline{\mathbf{F}}\left(t\right)\right\}$. Specifically,

• $\overline{\mathbf{X}}\left(t\right)=\left\{{\overline{x}}_m\left(t\right)\left|\forall m\in \mathcal{M}\right.\right\}$ are the association decision factors of MDs at timeslot t, where ${\overline{x}}_m\in \left[0,1\right]$.
• $\overline{\mathbf{Y}}\left(t+1\right)=\left\{{\overline{y}}_m\left(t+1\right)\left|\forall m\in \mathcal{M}\right.\right\}$ are the caching decision factors at timeslot t for the next timeslot.
• $\overline{\mathbf{Z}}\left(t\right)=\left\{{\overline{z}}_i\left(t\right)\left|\forall i\in \mathcal{I}\right.\right\}$ are the subchannel allocation decision factors of BSs at timeslot t, where ${\overline{z}}_i\in \left[0,1\right]$.
• $\overline{\mathbf{U}}\left(t\right)=\left\{{\overline{u}}_m\left(t\right)\left|\forall m\in \mathcal{M}\right.\right\}$ are the offloading decision factors of MDs at timeslot t, where ${\overline{u}}_m\in \left[0,1\right]$.
• $\overline{\mathbf{F}}\left(t\right)=\left\{{\overline{f}}_m\left(t\right)\left|\forall m\in \mathcal{M}\right.\right\}$ are the computing resource allocation factors of MDs at timeslot t, where ${\overline{f}}_m\in \left[0,1\right]$.

It is noteworthy that the dimension of the above-mentioned state and action spaces have been greatly reduced compared to the actual ones. The actual state and action spaces can be achieved by executing AT algorithm in the following parts.

❸ Reward: Considering that the goal of problem P1 is to minimize local energy consumption, and the constraints $C_1$ and $C_{10}$ cannot be strictly satisfied in the DRL-based iteration procedure, the reward $w_t$ at timeslot t is given by

$$w_t=-{\omega }_1{\sum }_{m\in \mathcal{M}}{\epsilon }_m^{\mathrm{tot}}\left(t\right)-{\omega }_2\varphi \left(t\right)-{\omega }_3\phi \left(t\right),$$

(12)

where $\varphi \left(t\right)={\sum }_{m\in \mathcal{M}}max\left({\tau }_m^{\mathrm{tot}}\left(t\right)-{\tau }_m,0\right)$ is the penalty function added for guaranteeing the constraint $C_1$, and $\phi \left(t\right)={\sum }_{i\in \mathcal{I}}max({\sum }_{m\in \mathcal{M}}{y}_{m,i}\left(t\right)d_m\left(t\right)-{\vartheta }_i\left(t\right),0)$ is the penalty function introduced for guaranteeing the constraint $C_{10}$; ${\omega }_1$ is the energy-consumption discount factor; ${\omega }_2$ and ${\omega }_3$ are penalty coefficients.

When the network obtains action $a_t$ according to the state $s_t$, the state space will obtain the next state $s_{t+1}$ according to the action $a_t$. Specifically, the task caching decisions of BSs can be directly achieved from $\mathbf{Y}\left(t+1\right)$ in $a_t$. Therefore, the total return of minimizing long-term local energy consumption within T timeslots can be given by

$$R={\sum }_{t\in \mathcal{T}}\gamma w_t,$$

(13)

where $\gamma$ is the reward discount factor satisfying $\gamma \in \left(0,1\right)$.

4.1. TD3O Algorithm

TD3 algorithm is an actor-critic-based framework, it comprises the policy ($\mu$) network, critic (Q) network and their corresponding target networks, and updates the network parameters using gradient algorithms. It is characterized by using two critic networks and two critic target networks in the design of critic networks. The TD3 algorithm is often divided into two parts consisting of experience collection and training. In the phase of collecting experience, new action $a_t$ can be generated by adding random Gaussian noise into the output of policy network at the state $s_t$, i.e.,

$$a_t=\mu \left(s_t,{\theta }^{\mu }\right)+{\overline{\sigma }}^2.$$

(14)

where ${\theta }^{\mu }$ is the parameter of policy network, and ${\overline{\sigma }}^2$ is the additive Gaussian noise.

After that, the environment is rewarded with $w_t$ and the next state $s_{t+1}$ can be achieved according to the state and action $\left(s_t,a_t\right)$. To enable the algorithm to obtain better decisions through past experience-assisted training, we put the quadruple $\left(s_t,a_t,w_t,s_{t+1}\right)$ into the experience replay buffer as a historical experience. In the training process, a certain number of quadruples are randomly selected from the experience replay buffer for training. Specifically, the training process can be divided into the following steps.

4.1.1. Training Policy Network

The training process of policy network is shown in Figure 3. In the training phase, N quadruples are extracted from the experience replay buffer and denoted as $\mathcal{N}=\left\{1,2,\cdots ,N\right\}$. For any quadruple $n\in \mathcal{N}$, the policy network outputs an new action ${a^{\prime }}_n=\mu \left(s_n,{\theta }^{\mu }\right)$ according to the state $s_n$. It should be noted that the policy ${a^{\prime }}_n$ is different from $a_n$ existing in the experience replay buffer. After $s_n$ and ${a^{\prime }}_n$ are inputted into any critic network (e.g., critic ${\mathrm{Q}}_1$ network), such network outputs $q_n=Q_1\left(s_n,\mu \left(s_n,{\theta }^{\mu }\right),{\theta }^{Q_1}\right)$, where ${\theta }^{Q_1}$ is the parameter of the critic $Q_1$ network. After achieving all $q_n$, their mathematical expectation is given by

$$J\left({\theta }^{\mu }\right)=\mathbb{E}\left[Q_1\left(\mathcal{S},\mu \left(\mathcal{S},{\theta }^{\mu }\right),{\theta }^{Q_1}\right)\right],$$

(15)

where $\mathcal{S}=\left\{s_n|n\in \mathcal{N}\right\}$. Then, the policy gradient of function J with respect to ${\theta }^{\mu }$ can be given by

$${\nabla }_{{\theta }^{\mu }}J=\mathbb{E}\left[{\nabla }_{\mathcal{A}}{Q}_1\left(\mathcal{S},\mathcal{A},{\theta }^{Q_1}\right){\nabla }_{{\theta }^{\mu }}\mu \left(\mathcal{S},{\theta }^{\mu }\right)\right],$$

(16)

where $\mathcal{A}=\left\{a_n|n\in \mathcal{N}\right\}$.

Significantly, the calculated gradient requires gradient clipping, which can avoid skipping the optimal solution because the gradient is too large. The calculated policy gradients will be used to update the parameters of the policy networks. We assume that the learning rate of the policy network is ${\beta }^{\mu }$, and Adaptive moment estimation (Adam) is used for achieving optimal ${\theta }^{\mu }$.

4.1.2. Training Critic Network

Figure 4 shows the training process of the critic network. During the critic network training, the policy at the next time is first estimated through the policy target (${\mu }^{-}$) network, i.e., ${a^{\prime }}_n={\mu }^{-}({s^{\prime }}_n,{\theta }^{{\mu }^{-}})+{\widehat{\sigma }}^2$, where ${\widehat{\sigma }}^2$ is the clipped additive Gaussian noise. Then, the action ${a^{\prime }}_n$ and the state ${s^{\prime }}_n$ are used as the input of the critic target ($Q_1^{-})$ network and critic target ($Q_2^{-}$) network, where ${\theta }^{Q_1^{-}}$ and ${\theta }^{Q_2^{-}}$ are their parameters. After that these two networks output ${\tilde{q}}_{n,1}$ and ${\tilde{q}}_{n,2}$ respectively. At the same time, the action $a_n$ and the state $s_n$ are used as the input of the critic $Q_1$ network and critic $Q_2$ network, where ${\theta }^{Q_1}$ and ${\theta }^{Q_2}$ are their parameters. After that these two networks output $q_{n,1}$ and $q_{n,2}$. Then, the approximation of Q value is ${\overline{q}}_n=r_n+\gamma {\tilde{q}}_n$ achieved using Behrman equation, where ${\tilde{q}}_n=min\left({\tilde{q}}_{n,1},{\tilde{q}}_{n,2}\right)$. At last, for all ${\overline{q}}_n$, according to the theorem of mean-squared error (MSE), the expectation function of squared loss between $Q_1\left(\mathcal{S},\mathcal{A},{\theta }^{Q_1}\right)$ and $\overline{Q}$ is

$$L_1\left({\theta }^{Q_1}\right)=0.5\mathbb{E}\left[{\left(Q_1\left(\mathcal{S},\mathcal{A},{\theta }^{Q_1}\right)-\overline{Q}\right)}^2\right],$$

(17)

and the expectation function of squared loss between $Q_2\left(\mathcal{S},\mathcal{A},{\theta }^{Q_2}\right)$ and $\overline{Q}$ is given by

$$L_2\left({\theta }^{Q_2}\right)=0.5\mathbb{E}\left[{\left(Q_2\left(\mathcal{S},\mathcal{A},{\theta }^{Q_2}\right)-\overline{Q}\right)}^2\right],$$

(18)

where $\overline{Q}=\left\{{\overline{q}}_n|n\in \mathcal{N}\right\}$. Then, the gradient of the loss function $L_1\left({\theta }^{Q_1}\right)$ with respect to the parameter ${\theta }^{Q_1}$ is

$${\nabla }_{{\theta }^{Q_1}}{L}_1=\mathbb{E}\left[\left(Q_1\left(\mathcal{S},\mathcal{A},{\theta }^{Q_1}\right)-\overline{Q}\right){\nabla }_{{\theta }^{Q_1}}{Q}_1\left(\mathcal{S},\mathcal{A},{\theta }^{Q_1}\right)\right],$$

(19)

and the gradient of the loss function $L_2\left({\theta }^{Q_2}\right)$ with respect to the parameter ${\theta }^{Q_2}$ is given by

$${\nabla }_{{\theta }^{Q_2}}{L}_2=\mathbb{E}\left[\left(Q_2\left(\mathcal{S},\mathcal{A},{\theta }^{Q_2}\right)-\overline{Q}\right){\nabla }_{{\theta }^{Q_2}}{Q}_2\left(\mathcal{S},\mathcal{A},{\theta }^{Q_2}\right)\right].$$

(20)

Similar to calculating the policy gradient, the gradient clipping needs to be performed after calculating the gradients using (19) and (20). In addition, ${\beta }^Q$ is the learning rate of the critic network, and the parameters of the two critic networks are updated using the Adam algorithm. Certainly, the parameters of critic target networks also need to be updated using soft update manner, i.e.,

$${\theta }^{{\mu }^{-}}=\lambda {\theta }^{\mu }+\left(1-\lambda \right){\theta }^{{\mu }^{-}},$$

(21)

$${\theta }^{Q_1^{-}}=\lambda {\theta }^{Q_1}+\left(1-\lambda \right){\theta }^{Q_1^{-}},$$

(22)

$${\theta }^{Q_2^{-}}=\lambda {\theta }^{Q_2}+\left(1-\lambda \right){\theta }^{Q_2^{-}},$$

(23)

where $\lambda$ is the learning rate of target networks.

It is noteworthy that a lower network updating frequency is adopted in this paper. We assume that the update interval of the critic network is $t^{\mathrm{cti}}$ and the update interval between the policy and critic networks is $t^{\mathrm{pti}}$. The critic networks are trained many times to ensure the stability of Q value. After that the policy network can be updated. The detailed procedure of TD3O algorithm can be summarized in Algorithm 1, where $t^{\mathrm{mep}}$ is the maximal number of epochs.

Algorithm 1: TD3-based Offloading (TD3O)

1: Initialization: ${\theta }^{Q_1}$, ${\theta }^{Q_2}$, ${\theta }^{\mu }$, ${\theta }^{Q_1^{-}}$, ${\theta }^{Q_2^{-}}$, ${\theta }^{{\mu }^{-}}$, $t^{\mathrm{step}}=0$, $t^{\mathrm{epoch}}=0$.

2: While $t^{\mathrm{epoch}}<t^{\mathrm{mep}}$

3: Let $t=0$, state $s_t$ and reward $R=0$.

4: While $t<T$

5:  Generate action $a_t$ using (14).

6:  Achieve actual action by executing Algorithm 2.

7:  Calculate reward $w_t$ using (12) and obtain the state $s_{t+1}$.

8:  If $t^{\mathrm{step}}\ge \kappa$

9:   Replace the previous quadruple with $\left(s_t,a_t,w_t,s_{t+1}\right)$.

10:  Else

11:   Put the quadruple $\left(s_t,a_t,w_t,s_{t+1}\right)$ into the queue.

12:  EndIf

13:  Update state $s_t=s_{t+1}$.

14:  If $t^{\mathrm{step}}\%t^{\mathrm{cti}}=0$ and $t^{\mathrm{step}}>N$

15:   Extract N quadruples for training.

16:   For any sample n, $Q_1^{-}$ and $Q_2^{-}$ networks output ${\tilde{q}}_{n,1}$ and

17:     ${\tilde{q}}_{n,2}$ respectively, and obtain the minimum value ${\tilde{q}}_n$.

18:   Calculate $L\left({\theta }_1^Q\right)$ and $L\left({\theta }_2^Q\right)$ using (17)-(18) respectively.

19:   Calculate Q gradient using (19)-(20), and clip it.

20:   Find ${\theta }^{Q_1}$ and ${\theta }^{Q_2}$ using Adam optimizer.

21:   If $t^{\mathrm{step}}\%t^{\mathrm{pti}}=0$

22:    Calculate q through $Q_1$.

23:    Calculate policy gradient using (16), and clip it.

24:    Find ${\theta }^{\mu }$ using Adam optimizer.

25:   EndIf

26:   Calculate ${\theta }^{Q_1^{-}}$, ${\theta }^{Q_2^{-}}$ and ${\theta }^{{\mu }^{-}}$ using (21)-(23) respectively.

27:  EndIf

28:  $R=R+\gamma w_t$.

29:  $t^{\mathrm{step}}=t^{\mathrm{step}}+1$; $t=t+1$.

30: EndWhile

31:  $t^{\mathrm{epoch}}=t^{\mathrm{epoch}}+1$.

32: EndWhile

4.2. AT Algorithm

In order to apply TD3O algorithm to solving the problem P1, it is necessary to convert the achieved continuous action $a_t=\left\{\overline{\mathbf{X}}\left(t\right),\overline{\mathbf{Y}}\left(t+1\right),\overline{\mathbf{Z}}\left(t\right),\overline{\mathbf{U}}\left(t\right),\overline{\mathbf{F}}\left(t\right)\right\}$ into discrete one [33]. To this end, we consider the following transformations for $a_t$.

4.2.1. The Discretization of Device Association Array

In $\overline{\mathbf{X}}=\left\{{\overline{x}}_m\left|\forall m\in \mathcal{M}\right.\right\}$, ${\overline{x}}_m$ is the non-integer association index of MD m, which is the continuous action achieved by TD3 algorithm. Then, it is converted into an integer form, i.e.,

$$\left\{\begin{array}{c}{x}_{m,\mathrm{ceil}\left(I{\overline{x}}_m\right)}=1,\mathrm{if}I{\overline{x}}_m\ne 0,\hfill \\ x_{m,1}=1,\mathrm{otherwise},\hfill \end{array}\right.$$

(24)

where $\mathrm{ceil}\left(b\right)$ is an upward rounding function with respect to b. Such a transformation can ensure that each MD can be associated with one BS.

4.2.2. The Discretization of Task Caching Array

In $\overline{\mathbf{Y}}$, ${\overline{y}}_m$ represents the non-integer caching index of MD m, which is the continuous action achieved by TD3 algorithm. Since each MD can store its task at all BSs, there exist $2^I$ storage options for it. Consequently, in order to convert ${\overline{y}}_m$ into a discrete form, we first need to perform

$$\left\{\begin{array}{c}{\widehat{y}}_m=\mathrm{floor}\left(2^I{\overline{y}}_m\right),\mathrm{if}{2}^I{\overline{y}}_m\ne 0,\hfill \\ {\widehat{y}}_m=0,\mathrm{otherwise},\hfill \end{array}\right.$$

(25)

where $\mathrm{floor}\left(b\right)$ is a downward rounding function with respect to b. Then, in order to achieve the binary caching index, the decimal ${\widehat{y}}_m$ needs to be converted into a binary number of I 0-1 digits, which is given by $\mathrm{bin}\left({\overline{y}}_m\right)$. In it, $\mathrm{bin}\left(b\right)$ is a function used for calculating the binary number of decimal b. Then, $y_{m,i}=\mathrm{bin}{\left({\overline{y}}_m\right)}_i$, where $\mathrm{bin}{\left({\overline{y}}_m\right)}_i$ represents the i-th digit of the binary number $\mathrm{bin}\left({\overline{y}}_m\right)$.

4.2.3. The Discretization of Task Offloading Array

In $\overline{\mathbf{U}}$, ${\overline{u}}_m$ is the non-integer offloading index of MD m, which is the continuous action achieved by TD3 algorithm. Considering that each MD can offload its task to at most one BS, ${\overline{u}}_m$ is converted into an integer form, i.e.,

$$\left\{\begin{array}{c}{u}_{m,\mathrm{ceil}\left(I{\overline{u}}_m\right)}=1,\mathrm{if}I{\overline{u}}_m\ne 0,\hfill \\ u_{m,i}=0,\forall i\in \mathcal{I},\mathrm{otherwise}.\hfill \end{array}\right.$$

(26)

4.2.4. The Discretization of Subchannel Allocation Array

In $\overline{\mathbf{Z}}$, ${\overline{z}}_i$ is the non-integer index of the subchannels allocated by BS i to its associated MDs who need to offload tasks, which is the continuous action achieved by TD3 algorithm. To achieve the integer form of ${\overline{z}}_i$, we first need to perform

$$\left\{\begin{array}{c}{\widehat{z}}_i=\mathrm{ceil}\left(\mathrm{C}\left(M_i,K_i\right){\overline{z}}_i\right),\mathrm{if}\mathrm{C}\left(M_i,K_i\right){\overline{z}}_i\ne 0,\hfill \\ {\widehat{z}}_i=1,\mathrm{otherwise},\hfill \end{array}\right.$$

(27)

where $M_i$ is the number of MDs who are associated with BS i and need to offload tasks; $K_i$ is the number of available subchannels at BS i; $\mathrm{C}\left(M_i,K_i\right)=\mathrm{fac}\left(K_i\right)/\mathrm{fac}\left(M_i\right)\mathrm{fac}\left(K_i-M_i\right)$ is a function with respect to $M_i$ and $K_i$, and used for calculating the number of feasible subchannel allocation policies between $M_i$ MDs and $K_i$ subchannels at BS i; $\mathrm{fac}\left(b\right)$ is a factorial function with respect to b; $M_i\le K_i$ shall be satisfied.

Then, we assume that ${\mathcal{Z}}_i=\left\{1,2,\cdots ,\mathrm{C}\left(M_i,K_i\right)\right\}$ is the set of $\mathrm{C}\left(M_i,K_i\right)$ feasible subchannel allocation policies between $M_i$ MDs and $K_i$ subchannels at BS i. After that, the subchannel allocation policy ${\widehat{z}}_i$ in the set ${\mathcal{Z}}_i$ is selected according to the equation (27). It is noteworthy that $\mathrm{C}\left(M_i,K_i\right)$ feasible subchannel allocation policies are generated in advance. That is to say, in the policy ${\widehat{z}}_i$, we can easily know the utilized indices of $K_i$ subchannels for $M_i$ MDs. According to these rules, we can easily find the subchannel allocation index $\mathbf{Z}$.

4.2.5. The Transformation of Computing Resource Allocation Array

In $\overline{\mathbf{F}}=\left\{{\overline{f}}_m\left|\forall m\in M\right.\right\}$, ${\overline{f}}_m$ represents the computing resource score of MD m at target BS that executing its task. If ${\sum }_{i\in I}{u}_{m,i}=1$ is satisfied between MD m and BS i, according to the proportional allocation of computing resources, the computing resources allocated to MD m by BS i can be given by

$$f_{m,i}=u_{m,i}{f}_i^{\mathrm{BS}}{\overline{f}}_m/{\sum }_{j\in \mathcal{M}}{u}_{j,i}{\overline{f}}_j.$$

(28)

Based on the above-mentioned operations, the output action $a_t=\left\{\overline{\mathbf{X}},\overline{\mathbf{Y}},\overline{\mathbf{Z}},\overline{\mathbf{U}},\overline{\mathbf{F}}\right\}$ of TD3O algorithm can be effectively converted into an actual decision, which is summarized as Algorithm 2.

Algorithm 2: Action transformation (AT)

1: For each MD $m\in \mathcal{M}$

2:  Achieve MD association matrix $\mathbf{X}$ using discretization rule.

3:  Achieve task caching matrix $\mathbf{Y}$ using discretization rule.

4:  Achieve task offloading matrix $\mathbf{U}$ using discretization rule.

5: EndFor

6: For each BS $i\in \mathcal{I}$

7: Returns the set ${\mathcal{K}}_i$ of available subchannels and the set ${\mathcal{M}}_i$ of

8:   offloading MDs.

9: If $M_i>K_i$

10:  $M_i-K_i$ associated MDs are randomly selected, disassociated

11:    and execute tasks locally.

12: EndIf

13: Achieve subchannel allocation matrix $\mathbf{Z}$ using discretization rule.

14: EndFor

15: For each MD $m\in \mathcal{M}$

16: If ${\sum }_{i\in \mathcal{I}}{u}_{m,i}=1$

17:  If ${\overline{f}}_m=0$

18:   Assign small enough computing capacity to MD m to avoid

19:    zero division.

20:  Else

21:   Allocate computing resources to MD m using (28).

22:  EndIf

23: EndIf

24: EndFor

4.3. HA

To solve the problem P1 in a non-iterative manner, we design an effective heuristic algorithm, which is summarized in Algorithm 3. In such an algorithm, in order to reduce the uplink transmission time and energy consumption, some MDs are associated with the nearest BSs. To guarantee time constraints, a part of MDs are disassociated with BSs without sufficient subchannels, and execute tasks by themselves.

Algorithm 3: Heuristic Algorithm (HA)

1: Initialization: energy consumption ${\overline{\epsilon }}^{\mathrm{tot}}=0$.

2: Each MD selects (is associated with) the nearest BS.

3: For each BS $i\in \mathcal{I}$

4:  If $M_i>K_i$

5:  $M_i-K_i$ associated MDs are randomly selected, disassociated

6:   and execute tasks locally.

7:  EndIf

8:  Randomly select the tasks of MDs associated with BS i for caching

9:  until the caching space is full.

10: EndFor

11: For $t\in \mathcal{T}$

12: Randomly select a target BS for each MD without cached task.

13: Randomly allocate subchannels to MDs associated with each BS.

14: If subchannels are insufficient

15:   Extra MDs are randomly selected to execute tasks locally.

16: EndIf

17: Proportionally allocate computing resources to MDs associated with

18:   each BS according to the CPU cycles required by tasks.

19: Calculate the total local energy consumption $\overline{\epsilon }$.

20: ${\overline{\epsilon }}^{\mathrm{tot}}={\overline{\epsilon }}^{\mathrm{tot}}+\overline{\epsilon }$.

21: EndFor

5. Algorithm Analysis

5.1. Computation Complexity Analysis

In this section, the computation complexity of proposed algorithms are analysed as follows.

Proposition 1: The computation complexity of Algorithm 2 is $\mathcal{O}\left(MIK\right)$ at the worst case.

proof: In Algorithm 2, the computation complexity of Steps 1-5 is $\mathcal{O}\left(M\right)$, the computation complexity of Steps 6-14 is $\mathcal{O}\left(MIK\right)$ at the worst case, and the computation complexity of Steps 15-24 is $\mathcal{O}\left(MI\right)$. In general, the computation complexity of Algorithm 2 is $\mathcal{O}\left(MIK\right)$ at the worst case. ❑

Proposition 2: The computation complexity of Algorithm 1 is $\mathcal{O}(max({\sum }_{l=0}^{L^Q}{\psi }_l^Q{\psi }_{l+1}^Q,{\sum }_{l=0}^{L^{\mu }}$ ${\psi }_l^{\mu }{\psi }_{l+1}^{\mu }\left)\right)$ at each timeslot, where $L^{\mu }$ is the number of layers of the policy network, $L^Q$ is the number of layers of the critic network, ${\psi }_l^{\mu }$ is the number of neurons at the l-th layer of the policy network, and ${\psi }_l^Q$ is the number of neurons at l-th layer of the critic network.

proof: In Algorithm 1, the computation complexity is mainly related to the action transformation, the calculation of reward and task processing time, and the structure of the neural network. As previously mentioned, the computation complexity of the action transformation should be $\mathcal{O}\left(MIK\right)$ at the worst case. Seen from the formulas (8) and (12), the computation complexity of the calculation of reward and task processing time is $\mathcal{O}\left(MIK\right)$.

In Algorithm 1, there exist four critic networks and two policy networks. We assume that the structure of the policy network and its target network is the same, and the structure of the two critic networks and its target network is the same. Then, we can easily deduce that the computation complexity of establishing policy networks is $\mathcal{O}\left({\sum }_{l=0}^{L^{\mu }}{\psi }_l^{\mu }{\psi }_{l+1}^{\mu }\right)$, and the computation complexity of establishing critic networks is $\mathcal{O}\left({\sum }_{l=0}^{L^Q}{\psi }_l^Q{\psi }_{l+1}^Q\right)$. Therefore, the computation complexity of establishing neural networks is $\mathcal{O}(max({\sum }_{l=0}^{L^Q}{\psi }_l^Q{\psi }_{l+1}^Q,{\sum }_{l=0}^{L^{\mu }}{\psi }_l^{\mu }$${\psi }_{l+1}^{\mu }\left)\right)$.

Since the computation complexity of establishing neural networks is much higher than the one of other operations in Algorithm 2. In general, the computation complexity of Algorithm 2 is $\mathcal{O}\left(max\left({\sum }_{l=0}^{L^Q}{\psi }_l^Q{\psi }_{l+1}^Q,{\sum }_{l=0}^{L^{\mu }}{\psi }_l^{\mu }{\psi }_{l+1}^{\mu }\right)\right)$ at each timeslot. ❑

Proposition 3: The computation complexity of Algorithm 3 is $\mathcal{O}\left(MI\right)$ at each timeslot.

proof: In Algorithm 3, the computation complexity of Step 2 is $\mathcal{O}\left(MI\right)$, the computation complexity of Steps 3-10 is $\mathcal{O}\left(I\right)$, the computation complexity of Steps 12-16 is $\mathcal{O}\left(M\right)$, the computation complexity of Steps 17-19 is $\mathcal{O}\left(MI\right)$. In general, the computation complexity of Algorithm 3 is $\mathcal{O}\left(MI\right)$ at each timeslot. ❑

5.2. Convergence Analysis

Since Algorithm 2 is a part of Algorithm 1 and Algorithm 3 is non-iterative, we just need to concentrate on the convergence of Algorithm 1. In detail, it is established as follows.

Theorem 1: Algorithm 1 can be guaranteed to converge after finite iterations.

proof: In Algorithm 1, the neural networks are updated by the gradient descent method used in the Adam optimizer. It utilizes the gradient information of the functions $J\left({\theta }^{\mu }\right)$, $L_1\left({\theta }^{Q_1}\right)$ and $L_2\left({\theta }^{Q_2}\right)$ to guide the updating directions of the parameters ${\theta }^{\mu }$, ${\theta }^{Q_1}$ and ${\theta }^{Q_2}$ so that the values of objective functions can reach the optimal or suboptimal. When these values tend to be stable, the parameters ${\theta }^{\mu }$, ${\theta }^{Q_1}$ and ${\theta }^{Q_2}$ also tend to be stable. At this time, Algorithm 1 is deemed convergent. ❑

6. Performance Evaluation

In order to verify the performance of the designed algorithms, we introduce the following algorithms for comparison.

DDPG-based Offloading (DDPGO): DDPG is a classical DRL algorithm. Compared with the TD3 algorithm, the DDPG algorithm reduces a critic network and a critic target network. In addition, both the critic network and policy network are updated at each timeslot in the DDPG algorithm. In this paper, the DDPG algorithm used for solving the problem P1 is named as DDPG-based offloading (DDPGO) algorithm.

Completely Offloading (CO): In CO algorithm, the task of each MD is offloaded to the nearest BS for computing. Such BS proportionally allocates the computing capacity to its associated MDs according to the CPU cycles required by the tasks of these MDs.

Completely Local Executing (CLE): In CLE algorithm, the tasks of all MDs can be executed by themselves.

In this paper, we consider that each BS is deployed in a non-overlapping area with a radius of 400 m, and the power spectral density is -174 dBm/Hz. In addition, $I=3$, $f_m^{\mathrm{loc}}=1\mathrm{GHz}$, $f_i=8\mathrm{GHz}$, $W=40\mathrm{MHz}$, $K=4$, $d_m=2\sim 5\mathrm{MB}$, $c_m=50\mathrm{cycles}/\mathrm{bit}$, $\xi ={10}^{-27}$, ${\tau }_m=10\mathrm{s}$, $\rho =2$, $r^{\mathrm{bh}}=1\mathrm{Gbps}$, $p_m=23\mathrm{dBm}$, $\kappa =80000$, $N=128$, $\gamma =0.94$, and $\lambda =0.04$. In the DRL algorithm, we consider that both the policy network and the critic network are composed of three-layer fully connected neural networks, where the numbers of neurons of three-layer neural networks in the policy network are 300, 200 and 128 respectively, and the corresponding target network has the same structure with this policy network; the number of neurons of three-layer neural networks in the critic network are 300, 128 and 32 respectively, and the corresponding target network has the same structure with this critic network. Significantly, the first-layer fully connected neural network of the policy network and the critic network utilizes the Rectified Linear Unit 6 (RELU6) suppressing the maximum value as the activation function, and other layers use RELU as the activation function.

Figure 5 shows the convergence of TD3O and DDPGO algorithms. As shown in Figure 5, DDPGO may have a higher convergence rate than TD3O, but the former may have worse convergence stability than the latter. The reason for this may be that the critic network and the policy network are updated synchronously in DDPGO. In DDPGO, the network parameters are updated in each training, which speeds up the convergence. Synchronously, the policy network parameters are updated in the training, which results in the instability of the long-term reward value and training bias. As we know, TD3O is composed of two sets of critic networks. Consequently, it can be trained in a relatively stable Q value so that the algorithm can converge stably. In the simulation, it is also easy to find that TD3O may achieve a more stable and better solution to the problem P1 than DDPGO in general.

Figure 6 shows the impact of training interval $t^{\mathrm{pti}}$ on the convergence of TD3O algorithm. As we know, under the same number of iterations, a larger $t^{\mathrm{pti}}$ can effectively reduce the overall training time of the network. However, it will reduce the total learning times of the policy network and its target network. As illustrated in Figure 6, the convergence rate of TD3O may decrease with $t^{\mathrm{pti}}$ in general.

Figure 7 shows the impacts of learning rates ${\beta }^Q$ and ${\beta }^{\mu }$ on the convergence of TD3O algorithm. As we know, when the learning rate ${\beta }^Q$ of the critic network increases, the parameters of such network will be updated at a larger scale, which speeds up the convergence of TD3O. However, it may lead to the failure of stable evaluation of environmental information, which weakens the convergence stability of TD3O. As illustrated in Figure 7, when ${\beta }^Q=0.001$, the convergence rate of TD3O is relatively high, but the achieved long-term reward dramatically fluctuates at this moment. On the other hand, the learning rate ${\beta }^{\mu }$ of the policy network can affect the optimization capability of TD3O. Specifically, a lower ${\beta }^{\mu }$ means a smaller amplitude of updating the policy network, which is better for finding better solutions. Seen from Figure 7, TD3O can achieve better long-term reward when ${\beta }^Q=0.0001$ and ${\beta }^{\mu }=0.0001$.

Figure 8 shows the impact of the number of MDs on the long-term local energy consumption ${\epsilon }^{\mathrm{MD}}$, where ${\epsilon }^{\mathrm{MD}}$ is the sum of total local energy consumption in T timeslots. In general, the ${\epsilon }^{\mathrm{MD}}$ increases with the number of MDs since more energy consumption is used for tackling more tasks of more MDs. Since CLE executes tasks in maximal computation capacity, it may achieve the highest ${\epsilon }^{\mathrm{MD}}$ among all algorithms. In CO, MDs are associated with the nearest BSs, which may result in a relatively imbalanced load distribution. Then, some overloaded BSs cannot provide good services for their associated MDs because of limited resources, which may result in high ${\epsilon }^{\mathrm{MD}}$. Consequently, CO may achieve higher ${\epsilon }^{\mathrm{MD}}$ than other algorithms excluding CLE. As illustrated in Figure 8, TD3O may achieve lower ${\epsilon }^{\mathrm{MD}}$ than DDPGO since the former can mitigate the overestimation existing in the latter well. Although HA lets MDs be associated with the nearest BSs, some MDs associated with overloaded BSs will disassociate and execute tasks locally. Such an operation may result in relatively low ${\epsilon }^{\mathrm{MD}}$.

Figure 9 shows the impacts of the number of MDs on the long-term reward (R). As illustrated in Figure 9, R may decrease with the number of MDs since more MDs result in higher energy consumption. Since both TD3O and DDPGO try to maximize the reward, but other algorithms are not the case, they may achieve higher R than other algorithms in general. Since TD3O may achieve lower ${\epsilon }^{\mathrm{MD}}$ than DDPGO, the former may achieve a higher R than the latter. In view of the unstable convergence of DDPGO, its reward may dramatically fluctuate. Since CLE may achieve the highest ${\epsilon }^{\mathrm{MD}}$ among all algorithms, it may achieve the lowest R in general. In addition, CO may achieve a lower R than HA since the former consumes more energy than the latter.

Figure 10 shows the impacts of the size of caching space of each BS on the relative long-term energy consumption ${\eta }^{\epsilon }$ denoted as the ratio of ${\epsilon }^{\mathrm{MD}}$ achieved by an offloading algorithm to the one attained by CLE. Evidently, a smaller ${\eta }^{\epsilon }$ means a higher energy gain caused by offloading tasks. As illustrated in Figure 10, besides DDPGO and CO, ${\eta }^{\epsilon }$ in other algorithms decreases with the size of caching space of each BS in general. The reason for this may be that a larger caching space can hold more tasks to reduce the transmission energy consumption. However, as revealed in Figure 5 and Figure 9, the unstable convergence of DDPGO may result in dramatically fluctuating performance. Therefore, ${\eta }^{\epsilon }$ in DDPGO may be evidently fluctuating. In addition, ${\eta }^{\epsilon }$ in CO may not change with the size of caching space of each BS since it doesn’t utilize caching space. By minimizing the ${\epsilon }^{\mathrm{MD}}$, TD3O and DDPGO may achieve a lower ${\eta }^{\epsilon }$ than other algorithms in general. In addition, TD3O may achieve a lower ${\eta }^{\epsilon }$ than DDPGO since the former mitigates the overestimation existing in DDPGO. Seen from Figure 10, CO may achieve the highest ${\eta }^{\epsilon }$ among all algorithms since it has no sufficient resources to provide for MDs associated with overloaded BSs.

Figure 11 shows the impacts of the size of caching space of each BS on the relative long-term reward ${\eta }^R$ denoted as the ratio of long-term reward achieved by an offloading algorithm to the one attained by CLE. Evidently, a smaller ${\eta }^R$ means a higher reward gain caused by offloading tasks. As illustrated in Figure 11, ${\eta }^R$ in TD3O and HA decreases with the size of the caching space of each BS in general. The reason for this may be that a larger caching space can hold more tasks to reduce the transmission energy consumption, and then bring a higher reward. However, due to the unstable convergence of DDPGO, ${\eta }^R$ in DDPGO may be fluctuating. Moreover, ${\eta }^R$ in CO may not change with the size of caching space of each BS since it doesn’t utilize caching space. By minimizing the ${\epsilon }^{\mathrm{MD}}$ and thus increasing reward, TD3O and DDPGO may achieve a lower ${\eta }^R$ than other algorithms in general. In addition, TD3O may achieve a lower ${\eta }^R$ than DDPGO since the former mitigates the overestimation existing in DDPGO. Seen from Figure 11, CO may achieve the highest ${\eta }^R$ because of the high energy consumed by MDs associated with overloaded BSs.

As seen from the above-mentioned simulation figures, although HA is a non-iterative algorithm, it may achieve better performance than DDPGO sometimes. In addition, it may always achieve fairly better performance than CO and CLE.

7. Conclusion

In this paper, the problem of minimizing the local energy consumption is concentrated in the cache-assisted vehicular NOMA-MEC networks under the latency and resource constraints, which refers to joint optimization of the computing resource allocation, subchannel selection, device association, offloading and caching decisions. To solve the formulated problem, we developed an effective TD3O algorithm integrating with the AT algorithm, and designed HA simultaneously. As for the designed algorithms, we give some analyses of the convergence and computation complexity. Simulation results show that such TD3O may achieve local lower energy consumption than several benchmark algorithms, and the HA may achieve lower local energy consumption than the CO and CLE algorithms. Future work can include power allocation and secure communications.