Securing Snapshot Pruning in IOTA Tangle 2.0: A Cooperative Deep Reinforcement Learning Approach for Edge-Cloud Smart Meter Networks

1. Introduction

Web 3.0 is the next evolution in Internet technology, shifting the paradigm of centralized trusted architecture to decentralized architecture with the utility of Artificial Intelligence [1]. Blockchain is the underlying technology in Web 3.0 that fosters decentralized trust on the Internet, controlling identities and data by users rather than companies. Smart contracts, decentralized finance, and bitcoin are examples of blockchain applications [2]. Industries are adopting blockchain in their service architecture as it facilitates a unique dimension of user autonomy. Blockchains are categorized into three types: public, private and consortium. The public blockchain is decentralized and open, such as Bitcoin, Ethereum and Solana. The private blockchain is permissioned. For example, Hyperledger Fabric, Alibaba blockchain, IBM blockchain, Microsoft Azure blockchain, Meta blockchain and Google Cloud blockchain are private blockchain offerings from big tech companies. The consortium blockchains, like R3 Corda and Quorum, are partially decentralized. Needless to say the sole purpose of blockchain is lost when they are offered as a service on a payment basis. However, targeting the Internet of Things (IoTs), the IOTA Tangle is an open-source Distributed Ledger Technology (DLT) where there are no miners and chargeable transactions like Ethereum 1. Especially, as the transactions (smart meter readings) and utility are not charged, anyone can use the stack to avail of free blockchain in their service architecture. On the flip side, it is under development, which means that people are actively researching to improve it from known vulnerabilities and bottlenecks in deployment [3].

Differing from the traditional blockchain with the proof of work (PoW) and Proof of Stake (PoS), the IOTA Tangle works by Directed Acyclic Graph (DAG). Every transaction in the Tangle network is referred to as a site (approved). Accordingly, the first approved transaction is the genesis site in the network. To add new transactions in the Tangle, transactions are locally verified (signature) by a node. To add the verified transactions (site) in the network, the local node attaches the transaction (tip) to two sites with the tip (not yet approved). Thereafter, the Tip Selection Algorithm (TSA) executes the process to approve the tip in order to convert it to a site [4]. It is noted that every site in the Tangle carries the specifics of a traditional blockchain, say transaction hashes, block signature and Merkle root. The recent version of the IOTA Tangle is IOTA 2.0 (Coordicide), and its previous version is IOTA 1.5 (Chrysalis). Using conventional blockchain for the energy industry is in practice [5], and its applicability is even wider, starting from the energy-producing sector to the energy-selling market [6].

We consider a network architecture with the IoT setting, shown in Figure 1 where the smart meters from homes and industries are connected to the edge servers via LORA technology. The edge servers are rented by the energy companies from cloud service providers (CSPs), such as Amazon Web Services (AWS), Microsoft Azure, Oracle Cloud and Google Cloud. CSPs and energy companies are in a software-level agreement (SLA). Energy companies utilize IOTA 2.0 as the blockchain architecture (Tangle network) to offer services, like smart contracts and smart meter users. Particularly, the Tangle network utilizes a deep Q actor-critic network-enables a reinforcement learning algorithm, where the actor (DQ-AN) and critic (DQ-CN) jointly work to obtain the optimal policy of storage, energy and pruning frequency. By pruning, we mean removing expired blocks. For this, an agent trains the network to minimize the loss by updating the parameters, and the agent works in a model-free architecture using the replay memory as a minibatch for sampling the (state, action, reward) tuple. Our architecture is an extension of the blockchain architecture considered for the smart meter application [7]. Notably, blockchain technology is widely applied in the energy sector, focusing the energy trading [8].

When IOTA 2.0 is utilized, energy providers cannot attain security, cost-effectiveness, and data analytics capability, which are prominent in securing the network and offering sophisticated services to users from the supplier’s point of view. For optimizing the network, the pre-Chrysalis version used the global snapshot mechanism, facilitating network-wide coordination. The global snapshot requires a coordinator to carry out the task, and it demands network-wide consistency to attain the pruning. Nonetheless, IOTA 2.0, is upgraded by the local snapshots where individual nodes can execute the pruning to optimize the storage 2. Though the local snapshot mechanism solves the issue of the requirement for a global coordinator and the process for maintaining network-wide unanimity, further improvement is still required to safeguard the security of the network. Specifically, the local snapshot is governed by static rules, i.e., nodes execute the pruning decision when the threshold limit for the storage is reached locally. This decision can lead to over-pruning or under-pruning, violating the storage requirement of minimum transactions to safeguard the security of the network. Also, as the global snapshot is deprecated in IOTA 2.0, we highlight that the lack of a coordinator can lead to failure in data analytics, blocking the utility of machine learning methods for electricity price forecasting [9]. Figure 2 shows the problems associated with the local snapshots and how they are overcome in the proposed method by the Actor-Critic-based Deep Q network. Considering the challenges faced by the local snapshot mechanism in IOTA 2.0, we pose the following question: "How can pruning decisions in IOTA 2.0 be made secure and efficient in decentralized edge networks?" This research question opens avenues to solve the problem in two-fold: developing methods to minimize the cost of the energy providers and developing methods to solve the issues associated with the local snapshot mechanism of IOTA 2.0. As optimising the storage directly reduces the number of edge servers and imposing security constraints while reducing the storage prevents the adversarial attacks, this research question is worth solving.

By answering the research question, we lay out the following contributions of this article:

• We formulate the cost optimization problem as a function of the storage, energy and pruning frequency optimization for optimizing the policy of energy and storage consumption and pruning frequency, respectively. Precisely, we optimize storage and energy by strictly satisfying security constraints in sub-problem 1. Then, we jointly optimize the pruning frequency in sub-problem 2, where constraints are relaxed to attain a feasible solution.
• We develop Actor-Critic method-based deep reinforcement learning algorithms (DQACN) for deep policy optimization, working cooperatively when they are deployed in the edge servers with a cloud service provider.
• Extensively, we perform the Monte Carlo simulation to gain confidence in the performance improvement of proposed algorithms over conventional local snapshots, global snapshots, ordinary deep Q network and random elimination methods.
• We implement and test the proposed algorithms to evaluate their performance robustly in the real-world environment.

The remainder of the article is organized as follows: Section 2 presents the related work. Section 3 presents problem formulation, introducing IOTA 2.0. Section 4 put forth the solution approach, convergence proofs for the proposed algorithms and formal security analysis in the universal composability framework. The next section (Section 5) evaluates the proposed algorithms and other methods using the Monte Carlo simulation. Section 6 carries out the implementation study of the proposed algorithm in a real-world setting. Finally, the work is concluded in Section 7.

2. Related Work

Aiming to present the taxonomy in storage optimization of the blockchain, the study [10] categorizes storage optimization in blockchain into three types: replication-based, redaction-based and content-based. Storage optimization in conventional blockchains is targeted by replication-based methods, whereas the redaction-based and content-based methods find optimization in the service and application layers. This study clarifies that storage optimization in the IOTA Tangle has not been done, so we take our breadth-first approach towards optimizing the storage and its associated cost in the IOTA Tangle by preserving security.

Xu et al. [11] presented the storage optimization in blockchain. Their work formulates the objective function as a multi-objective function, and their storage model captures the disk space of the servers used in the cloud environment. However, they propose the genetic algorithm-based solution approach, which usually takes a longer time to find the global optimum solution.

Zuo et al. [12] put forth a three-level, involving IoT devices, a base station and a cloud server, Stackelberg game-based optimal cloud resource utilization model involving IoT devices, base station, and cloud server. This method shows how to cooperatively allocate cloud resources for blockchain storage. Moreover, it lays out the resource allocation model, computing service pricing model and computing, storage and communication model. This work captures mining in the blockchain; however, their work does not involve optimizing the blockchain itself, which directly contributes to the storage space.

Targeting the minimization of energy consumption and carbon emission, Alofi et al. [13] studied the multi-objective optimization for the Bitcoin environment. Their work considers a variety of objectives ranging from reputation to carbon emissions. They present the Pareto front for the different combinations of objective functions, minimizing the convex function with the obtained hypervolume set. Nonetheless, they used the evolutionary algorithm as the solution approach, which takes a longer time to converge and is unsuitable for the considered application in the edge-based cloud network.

Akrasi-Mensah et al. [14] discussed the usage of the deep reinforcement learning algorithm for solving storage optimization in the cloud. Specifically, they considered a blockchain network for the cloud where storage cost is optimized. The solution approach consists of an Actor-Critic method-based optimal policy selection algorithm. Though this approach presents the applicability of the deep reinforcement algorithm as the solution approach, this method does not consider edge networks, i.e., the method applies to the traditional cloud network model.

Zhou et al. [15] presented a way of optimizing the storage cost in the blockchain. The problem is formulated as a multi-objective optimization and reduced to the block selection problem. For optimizing the blockchain, the storage model captures the blockchain as individual blocks with a finite size. Their solution approach considers the traditional deep Q network, which overcomes the curse of dimensionality in the conventional reinforcement learning methods. However, the blocks in the blockchain do not capture the nuance of transactions, signatures and the Merkle root of the block, which is essential to assess the security of the optimization.

Considering a consortium blockchain network, Tanis et al. [16] proposed a cooperative framework for energy trading using the Nash equilibrium [17]. In contrast to the traditional single-energy model, this work takes into account the multi-energy model. Q-algorithm in reinforcement learning is used to solve the nonlinear multi-objective optimization. However, this work does not consider storage optimization, though it leverages the cooperative framework.

In general, in the storage optimization of the blockchain and the granularity of modeling the blocks (precise modeling of components of a block/site) in the blockchain are absent, though optimization for improving throughput is evident [18]. Importantly, security constraints have not been considered to preserve security. Further, blockchain optimization for the edge-assisted cloud network has not been considered, specifically implementing the IOTA Tangle. Cost optimization for energy service providers renting the cloud server from CSPs using the IOTA Tangle network is not reported in the literature, which is a thriving market.

3. Problem Formulation

In this section, we formulate the energy model and storage model of IOTA 2.0 (Coordicide), followed by formulating optimization problems for energy cost, storage cost and pruning frequency. While IOTA 2.0 does not use the proof of work of the traditional blockchain, it retains many features of the blockchain, such as signature computation and Merkle root retention for transactions (sites), so we introduce the features of IOTA 2.0 and compare them with the conventional blockchain. Then, we prove that the presented optimization problem is NP-hard.

3.1. IOTA 2.0 (Coordicide)

IOTA 2.0 is referred to as a Tangle rather a blockchain since the network is formed by a DAG. Transactions in IOTA 2.0 are called sites. Each site is referenced by two previous transactions (tips). When a node wants to add a transaction, it places the site (unapproved transaction) in the tangle by referring to two previous tips (approved transactions). Then, the Tip Selection Algorithm (TSA) approves the added site to a tip with the help of the referenced two tips. Thus, every site in the tangle is referenced by two tips.

The proof of work is replaced by the mana-based system and Fast Probabilistic Consensus (FPC). Mana is a reputation-based resource bound with every node. This is introduced to have a full-fledged decentralization, as the coordinator in the previous version (IOTA 1.5) is replaced. Every node generates mana by holding IOTA tokens (stake-based) and through its activity (transactions). There are two types of manas, viz, access mana and consensus mana. The access mana determines the node’s access to the network, that is, the number of transactions issued by the node. Consensus mana defines the weight of the node when it participates in voting during FPC. The mana of a node is used for at least three purposes: protection, control and weighing. Briefly, Mana prevents the Sybil attack by giving importance to token ownership. This prevents malicious nodes from gaining disproportionate access in the tangle by creating fake identities. Mana regulates the transaction issuance rate. Nodes with higher mana are preferred during network congestion. Consensus mana of every node carries wight during voting in FPC. That is, nodes with higher consensus mana have an advantage in conflict resolution as their votes are weighed higher. Table A1 (Appendix B) compares IOTA 2.0 with the proof of work (PoW) and proof of stake (PoS) of the traditional blockchain.

Remark: While the coordinator (centralized and recommended in IOTA 1.5) is deprecated in IOTA 2.0, deployment of the global coordinator has some benefits. For instance, aggregated data retention for future data analytics can be easily accomplished with such a central coordinator though it does not hold any important roles centrally, like the global snapshot (pruning decision is centrally coordinated).

3.2. Energy Model

Let the total number of edge servers be $N=\{1,2,...,i,...,N\}$. Out of the edge servers, a subset of servers ($C\left[n\right]$) is responsible for carrying out FPC. Table 1 elaborately presents the different notations and their meaning. The pruning decision for optimizing the storage relates to energy consumption.

The cost for edge servers rented by the energy companies depends on the energy consumed by the tangle network. Therefore, it is essential to precisely model the energy cost associated with the tangle network. Without loss of generality, we present the energy cost for the tangle network as the cost incurred for transaction, tip, consensus and pruning.

The total energy consumption across all edge servers at time n in IOTA 2.0 is given by.

$$E_{\mathrm{total}}\left[n\right]=\sum _{i=1}^N\left(E_{\mathrm{T},i}\left[n\right]+E_{\mathrm{tip},i}\left[n\right]+E_{\mathrm{C},i}\left[n\right]+E_{\mathrm{P},i}\left[n\right]\right).$$

(1)

The cost associated with a transaction can be decomposed into the cost incurred for the hash computation, signature computation, attaching cost and the number of transactions. Accordingly, Equation (2) presents the transaction cost for the $i^{th}$ server.

$$E_{\mathrm{T},i}\left[n\right]={\lambda }_T^i\left[n\right]·\left(k_h·n_{\mathrm{hash}}^i\left[n\right]+k_s·n_{\mathrm{sig}}^i\left[n\right]+k_{\mathrm{attach},i}\right).$$

(2)

The consensus cost is presented in Equation (3).

$$E_{\mathrm{C},i}\left[n\right]={\psi }_{i\in C\left[n\right]}·{\displaystyle \frac{k_{\mathrm{fpc},i}·Q_i\left[n\right]+k_{\mathrm{mana},i}·M_i\left[n\right]}{{\alpha }_i}},$$

(3)

where $M\left[n\right]$ denotes the mana, and it is given in (4). We assume the smart meters delegate their mana to the edge servers (pledging) as they have limited computing resources. The mana of server i at time n is updated to include pledged mana from the smart meters in the below equation that presents the time evolution model of the mana. This equation, also, captures the notion of decreasing mana of a server in the event of inactivity.

$$m_i\left[n\right]=m_i[n-1]·e^{-{\delta }_m}+{\kappa }_m·{\lambda }_T^i[n-1]+\sum _{j\in S_i}{m}_j[n-1]$$

(4)

The energy associated with tip selection for server i at time n in IOTA 2.0 can be expressed as follows:

$$E_{\mathrm{tip},i}\left[n\right]={\lambda }_T^i\left[n\right]·\left(k_h·n_{\mathrm{hash}-\mathrm{tip}}^i\left[n\right]+k_s·n_{\mathrm{sig}-\mathrm{tip}}^i\left[n\right]\right).$$

(5)

The energy associated with pruning operations (local snapshot pruning in IOTA 2.0) for server i at time n in IOTA 2.0 is given by:

$$E_{\mathrm{P},i}\left[n\right]={\psi }_i·{\lambda }_P^i\left[n\right]·k_p.$$

(6)

The total energy associated with the $i^{th}$ server is presented in the Equation (8) by substituting Equations (2), (3), (5) and (6).

$$\begin{array}{cc}\hfill E_{\mathrm{total}}\left[n\right]& =\sum _{i=1}^N{E}_{\mathrm{server},i}\left[n\right]=\sum _{i=1}^N({\lambda }_T^i\left[n\right]·\left(k_h·n_{\mathrm{hash}}^i\left[n\right]+k_s·n_{\mathrm{sig}}^i\left[n\right]+k_{\mathrm{attach},i}\right)\hfill \\ \hfill & +{\psi }_{i\in C\left[n\right]}·{\displaystyle \frac{k_{\mathrm{fpc},i}·Q_i\left[n\right]+k_{\mathrm{mana},i}·M_i\left[n\right]}{{\alpha }_i}}\hfill \\ \hfill & +{\lambda }_T^i\left[n\right]·\left(k_h·n_{\mathrm{hash}-\mathrm{tip}}^i\left[n\right]+k_s·n_{\mathrm{sig}-\mathrm{tip}}^i\left[n\right]\right)\hfill \\ \hfill & +{\psi }_i·{\lambda }_P^i\left[n\right]·k_p)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill E_{\mathrm{total}}\left[n\right]& =\sum _{i=1}^N{\lambda }_T^i\left[n\right]·\left(k_h·n_{\mathrm{hash}}^i\left[n\right]+k_s·n_{\mathrm{sig}}^i\left[n\right]+k_{\mathrm{attach},i}\right)\hfill \\ \hfill & +\sum _{i=1}^N{\psi }_{i\in C\left[n\right]}·{\displaystyle \frac{k_{\mathrm{fpc},i}·Q_i\left[n\right]+k_{\mathrm{mana},i}·M_i\left[n\right]}{{\alpha }_i}}\hfill \\ \hfill & +\sum _{i\in P\left[n\right]}{\lambda }_T^i\left[n\right]·\left(k_h·n_{\mathrm{hash}-\mathrm{tip}}^i\left[n\right]+k_s·n_{\mathrm{sig}-\mathrm{tip}}^i\left[n\right]\right)\hfill \\ \hfill & +\sum _{i=1}^N{\psi }_i·{\lambda }_P^i\left[n\right]·k_p\hfill \end{array}$$

(8)

3.3. Storage Model

The storage dynamics over time are captured by the storage model of the time evolution equation. For the discrete time instance n, the storage is given by $S[n-1]$, i.e., storage contributed from the previous time instance, where $S\left[0\right]\in ⌀$. Equation (9) depicts the total storage contributed at n by the $i^{th}$ server, where $S_{\mathrm{tx}}$ and ${\lambda }_T^i\left[n\right]\in [0,1]$ denotes the storage size per transaction and coefficient for the transaction of the $i^{th}$ server, respectively.

$$\begin{array}{cc}\hfill S_S^i\left[n\right]& ={\lambda }_T^i\left[n\right]·S_{\mathrm{tx}}\hfill \end{array}$$

(9)

The quantity of storage freed by pruning is modeled in the Equation (10). $S_P$ specifies the storage cost eliminated by pruning, and ${\lambda }_P^i\left[n\right]\in [0,1]$ denotes the act of pruning done at the $i^{th}$ server at the instance n.

$$\begin{array}{cc}\hfill S{\left[n\right]}_F^i& ={\lambda }_P^i\left[n\right]·S_{\mathrm{P}}\hfill \end{array}$$

(10)

The total storage cost for N servers can be framed as per Equation (12) by combining Equations (9) and (10).

$$\begin{array}{cc}\hfill S\left[n\right]& =S[n-1]+\sum _{i=1}^N\left({\lambda }_T^i\left[n\right]·S_{\mathrm{tx}}-{\lambda }_P^i\left[n\right]·S_{\mathrm{P}}\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill S\left[n\right]& =S[n-1]+\sum _{i=1}^N{\lambda }_T^i\left[n\right]·S_{\mathrm{tx}}-\sum _{i=1}^N{\lambda }_P^i\left[n\right]·S_{\mathrm{P}}\hfill \end{array}$$

(12)

3.4. Problem Formulation

In this study, our aim is to optimize storage and energy as they directly contribute to the cost of an edge server, thereby increasing the monetary benefit. The key performance indicator of the objective function is linked to the cost-benefit. While optimizing the storage and energy, constraints, namely security constraints, software level agreement (SLA), bill settlement and retention of historical data have to be strictly satisfied. Therefore, we formulate sub-problem 1 (SP 1) to optimize the storage and energy by strictly meeting these constraints (constraint satisfaction problem (CP)). Sub-problem 1 is formulated to optimize storage and energy individually. Then, in the sub-problem 2 (SP 2), we jointly optimize the pruning frequency, energy and storage. We formulate each objective function using regret, as the regret-based formulation captures dynamic scenarios in comparison with the usual static formulation. The proposed optimisation is replication-based.

$$\begin{array}{c}\hfill R\left(T\right)=\sum _{t=1}^T\left(C^{*}\left[t\right]-C\left[t\right]\right)\end{array}$$

(13)

In the above equation, $C\left[t\right]$ represents the cost at time t, and $C^{*}\left[t\right]$ specifies the best possible cost in hindsight. By Equation (14), we intend to minimize the energy consumption by choosing the best policy ${\pi }_i^{*}\in \mathsf{\Pi }$ from the set of policies ${\left\{{\pi }_i\right\}}_{i=1}^{i=n}$, where $\pi :S\to A$, i.e. a mapping function from state to action (refer to next section).

SP 1:

$$\begin{array}{cc}\hfill \underset{{\pi }_i^{*}\in \mathsf{\Pi }}{max}\underset{E\left[n\right]}{min}{R}_{\mathrm{storage}}\left(T\right)& =\underset{\pi \in \mathsf{\Pi }}{max}\underset{S\left[n\right]}{min}\sum _{t=1}^T\left(S_{\mathrm{total}}^{*}\left[t\right]-S_{\mathrm{total}}\left[t\right]\right)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\mathrm{Subject}\mathrm{to}:\hfill & \hfill \end{array}$$

$$\begin{array}{cc}\hfill C_1:& S\left[n\right]\ge min\left({\displaystyle \frac{2^{256}}{B_{\mathrm{hash}}}},{\displaystyle \frac{2^{256}}{B_{\mathrm{tx}}}},{\displaystyle \frac{2^{128}}{B_{\mathrm{hash}}}},{\displaystyle \frac{2^{256}}{B_{\mathrm{sig}}}}\right)\in [{10}^{28},{10}^{71}]\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill C_2:& {\lambda }_P^i\left[n\right]\le B_{\mathrm{paid},i}\left[n\right],\forall B_{\mathrm{paid},i}\left[n\right]\in \{0,1\}\hfill \end{array}$$

(14b)

$$\begin{array}{cc}\hfill C_3:& S_H\left[n\right]\ge N_{\mathrm{ret}}\times S_{\mathrm{tx}},\forall S_{\mathrm{tx}}\in [0,\infty )\hfill \end{array}$$

(14c)

$$\begin{array}{cc}\hfill C_4:& S_{\mathrm{edge}}\left[n\right]\le S_{\mathrm{edge}}^{max},\forall S_{\mathrm{edge}}^{max}\in (0,\infty )\hfill \end{array}$$

(14d)

$$\begin{array}{cc}\hfill C_5:& S_{\mathrm{cloud}}\left[n\right]\ge S_{\mathrm{cloud}}^{min},\forall S_{\mathrm{cloud}}^{min}\in (0,\infty )\hfill \end{array}$$

(14e)

$$\begin{array}{cc}\hfill C_6:& {\lambda }_P^i\left[n\right]\le {\lambda }_P^{max},\forall {\lambda }_P^{max}\in [0,1]\hfill \end{array}$$

(14f)

$$\begin{array}{cc}\hfill C_7:& H_{\mathrm{cloud}}\left[n\right]\ge H_{\mathrm{cloud}}^{min},\forall H_{\mathrm{cloud}}^{min}\in (0,\infty )\hfill \end{array}$$

(14g)

$$\begin{array}{cc}\hfill C_8:& S_H\left[n\right]\ge S_H^{min},\forall S_H^{min}\in (0,\infty )\hfill \end{array}$$

(14h)

Security constraints for a 256-bit security level are presented in $C_1$. In $C_1$, ${\displaystyle \frac{2^{256}}{B_{hash}}}$ presents the security constraint for the first preimage resistance (given a hash, finding its input is computationally infeasible). This constraint imposes search space complexity bounded to computation time for a hash ($B_{hash}$), i.e., when time $t\to \infty$, $2^{256}\to 0$. ${\displaystyle \frac{2^{256}}{B_{block}}}$ presents the security constraint for the second pre-image resistance, that is, given a hash and its input, finding its second input is computationally infeasible. ${\displaystyle \frac{2^{128}}{B_{hash}}}$ presents the constraint for collision resistance property, that is, finding two distinct inputs for the same hash output is computationally infeasible. Due to the birthday paradox, $2^{256}$ is reduced to $2^{128}$. ${\displaystyle \frac{2^{256}}{B_{sig}}}$ presents the integrity constraint. We assume that 1 billion hashes can be produced in a second, though in Bitcoin mining, 1 trillion hashes per second are produced. Accordingly, the minimum values obtained are in the order of ${10}^{28}$. For the integrity, the maximum value is obtained in the order of ${10}^{71}$ as IOTA 2.0 uses Winternitz One-Time Signatures (WOTS), which accounts for 1 million signature verifications per second. $C_2$ presents the pruning constraints on blocks that are settled for bills. $B_{\mathrm{paid},i}\left[n\right]$ is 1 if the bill is settled; otherwise, its value is 0, indicating not ready for pruning. $C_3$ presents the number of blocks retailed as part of the historical data. Constraints from $C_4$ to $C_8$ present software level agreement (SLA) constraints. Electricity and cloud service providers agree on an SLA to rent resources. Specifically, $C_4$ depicts the storage capacity of the edge server should not exceed its maximum capacity. $C_5$ presents the constraint that at the cloud level, the minimum storage has to be ensured for future audits. $C_6$ insists the pruning frequency should not exceed its limit as it increases energy consumption. $C_7$ presents at the cloud, minimum storage has to be ensured for analytics purposes. $C_8$ ensures the retention of local historical data for the smart meter.

Now that we present the objective function based on the regret definition (refer to Equation (13)) for energy optimization.

$$\begin{array}{cc}\hfill \underset{{\pi }^{*}\in \mathsf{\Pi }}{max}\underset{E\left[n\right]}{min}{R}_{\mathrm{enerygy}}\left(T\right)& =\underset{{\pi }^{*}\in \mathsf{\Pi }}{max}\underset{E\left[n\right]}{min}\sum _{t=1}^T\left(E_{\mathrm{total}}^{*}\left[t\right]-E_{\mathrm{total}}\left[t\right]\right)\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\mathrm{Subject}\mathrm{to}:\hfill & \hfill & \hfill \end{array}$$

$$\begin{array}{cc}\hfill C_1:& E_{\mathrm{total}}\left[n\right]\ge min\left({\displaystyle \frac{2^{256}}{P_{\mathrm{compute}}}},{\displaystyle \frac{2^{256}}{P_{\mathrm{block}}}},{\displaystyle \frac{2^{128}}{P_{\mathrm{hash}}}},{\displaystyle \frac{2^{256}}{P_{\mathrm{sig}}}}\right)\in [{10}^{28},{10}^{71}]\hfill \end{array}$$

(15a)

$$\begin{array}{cc}\hfill C_2:& {\lambda }_P^i\left[n\right]\le B_{\mathrm{paid},i}\left[n\right],\forall B_{\mathrm{paid},i}\left[n\right]\in \{0,1\}\hfill \end{array}$$

(15b)

$$\begin{array}{cc}\hfill C_3:& E_H\left[n\right]\ge N_{\mathrm{ret}}\times P_{\mathrm{block}}\hfill \end{array}$$

(15c)

$$\begin{array}{cc}\hfill C_4:& E_{\mathrm{edge}}\left[n\right]\le E_{\mathrm{edge}}^{max},\forall E_{\mathrm{edge}}^{max}\in (0,\infty )\hfill \end{array}$$

(15d)

$$\begin{array}{cc}\hfill C_5:& E_{\mathrm{cloud}}\left[n\right]\ge E_{\mathrm{cloud}}^{min},\forall E_{\mathrm{cloud}}^{min}\in (0,\infty )\hfill \end{array}$$

(15e)

$$\begin{array}{cc}\hfill C_6:& {\lambda }_P^i\left[n\right]·P_{\mathrm{prune}}\le {\lambda }_P^{max},\forall {\lambda }_P^{max}\in (0,1)\hfill \end{array}$$

(15f)

$$\begin{array}{cc}\hfill C_7:& E_H\left[n\right]\ge E_H^{min},\forall E_H^{min}\in (0,\infty )\hfill \end{array}$$

(15g)

$$\begin{array}{cc}\hfill C_8:& E_{\mathrm{edge}}\left[n\right]+E_{\mathrm{cloud}}\left[n\right]\ge E_H\left[n\right],\forall E_H\in (0,\infty )\hfill \end{array}$$

(15h)

$C_1$ ensures the minimum energy is allotted to maintain security. In particular, ${\displaystyle \frac{2^{256}}{P_{\mathrm{compute}}}}$ ensures energy consumption for the first pre-image resistance. ${\displaystyle \frac{2^{256}}{P_{\mathrm{block}}}}$ assures the energy consumption for the second pre-image resistance. ${\displaystyle \frac{2^{128}}{P_{\mathrm{hash}}}}$ ensures the energy consumption for maintaining the collision resistance property. ${\displaystyle \frac{2^{256}}{P_{\mathrm{sig}}}}$ warrant the energy consumption for maintaining the integrity. $C_2$ confirms only if pruning is allowed bill after bill settlement. $C_3$ corroborates that energy is allotted for retaining the relevant blocks as historical data. $C_4$ and $C_5$ meet the energy consumption of the edge server does not exceed its maximum capacity, and the energy capacity of the cloud is allotted to meet its minimum energy utilization. The pruning frequency should not consume much energy. This is met in $C_6$. While $C_3$ allocates the minimum energy consumption for maintaining the historical data, $C_7$ prevents that energy reduction from falling below the minimum level of energy for maintaining the historical data. The final constraint $C_8$ ensures the balance between the energy used by cloud and edge w.r.t. the energy maintained for retaining historical data.

SP 2:

We jointly optimize the pruning frequency by considering the energy consumption and storage consumption costs, and turning security constraints into the objective function.

$$\begin{array}{c}\hfill J_{\mathrm{prune}}\left({\lambda }_P\left[n\right]\right)=w_E·\left(E_{\mathrm{total}}^{*}\left[n\right]-E_{\mathrm{total}}\left[n\right]\right)+w_S·\left(S_{\mathrm{total}}^{*}\left[n\right]-S_{\mathrm{total}}\left[n\right]\right)+\\ \hfill w_C·C_{\sec \mathrm{urity}}\left({\lambda }_P\left[n\right]\right)\end{array}$$

(16)

$$\begin{array}{cc}\hfill C_{\sec \mathrm{urity}}\left({\lambda }_P\left[n\right]\right)& ={\alpha }_1·{\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{256}}{B_{\mathrm{hash}}}}+{\alpha }_2·{\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{256}}{B_{\mathrm{block}}}}\hfill \\ \hfill & +{\alpha }_3·{\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{128}}{B_{\mathrm{hash}}}}+{\alpha }_4·{\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{256}}{B_{\mathrm{merkle}}}}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill \underset{{\pi }_i^{*}\in \mathsf{\Pi }}{max}\underset{{\lambda }_P\left[n\right]}{min}{R}_{\mathrm{joint}}\left(T\right)& =\sum _{t=1}^T\left(J_{\mathrm{prune}}\left({\lambda }_P\left[t\right]\right)-J_{\mathrm{prune}}\left({\lambda }_P^{*}\left[t\right]\right)\right)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\mathrm{Subject}\mathrm{to}:\hfill & \hfill \end{array}$$

$$\begin{array}{cc}\hfill & \left(E_{\mathrm{hash}}\left[n\right]+E_{\mathrm{block}}\left[n\right]+E_{\mathrm{sig}}\left[n\right]+\underset{P_{receive}}{\underbrace{{\displaystyle \frac{D\left[n\right]·P_{\mathrm{receive}}}{R_{\mathrm{LoRa}}}}}}\right)\le E_{\max }\left[n\right],\forall E_{\max }\left[n\right]\in (0,\infty )\hfill \end{array}$$

(18a)

$$\begin{array}{cc}\hfill & {\lambda }_P\left[n\right]\times \left(\underset{B_{block}}{\underbrace{N_{\mathrm{tx}}\times B_{\mathrm{tx}}}}+B_{\mathrm{hash}}+B_{\mathrm{sig}}+B_{\mathrm{merkle}}\right)\le S_{\min }\left[n\right],\forall S_{\min }\left[n\right]\in (0,\infty )\hfill \end{array}$$

(18b)

$$\begin{array}{cc}\hfill S_{\mathrm{total}}\left[n\right]& \ge S_H\left[n\right]+N_{\mathrm{ret}}\times B_{\mathrm{block}}\in (0,\infty )\hfill \\ \hfill \end{array}$$

(18c)

In Equation (16), $w_E$, $w_S$ and $w_C$ are weights for energy consumption, storage and security at time n, respectively. $E_{\mathrm{total}}^{*}$ and $S_{\mathrm{total}}^{*}$ are energy and storage at hindsight. $C_{security}$ is the objective function capturing the security constraint. $C_{security}$ is the objective capturing the security constraints. Precisely, the first pre-image resistance term is covered in ${\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{256}}{B_{\mathrm{hash}}}}$. The second pre-image resistance is absorbed by the term ${\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{256}}{B_{\mathrm{block}}}}$. The collision resistance and integrity are modelled by the terms ${\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{128}}{B_{\mathrm{hash}}}}$ and ${\displaystyle \frac{{\lambda }_P\left[n\right]}{S\left[n\right]}}\times {\displaystyle \frac{2^{256}}{B_{\mathrm{merkle}}}}$, respectively. ${\alpha }_1,...,{\alpha }_4$ are the weights for these security properties. From Equation (18), $J_{prune}\left({\lambda }_P^{*}\left[t\right]\right)$ is the best known pruning frequency at hindsight. Equation (18a) imposes a constraint for pruning energy, i.e. total pruning energy should not exceed the maximum energy. The pruning should not be overwhelmed, and it is met in the constraint (18b) by ensuring minimum blocks. At any instance, $S_{total}\left[n\right]$ should be more than the number of blocks retained as historical data (Equation (18c)).

We show that SP 2 is NP-hard as it includes SP 1 and its components as constraints. The problem of finding the optimal pruning frequency is an NP-hard problem. We prove that the Pruning Optimization Decision Problem (PODP) is NP-hard by reducing it to the Boolean Satisfiability Problem (SAT), which is NP-complete (PODP is polynomial-time reducible to SAT).

Decision Problem formulation: Given N edge servers, time slot T, pruning decisions ${\lambda }_P^i\left[n\right]\in \{0,1\}$, costs $J_{\mathrm{total}}^i\left[n\right]$, and threshold K, does there exist a pruning schedule ${\left\{{\lambda }_P^i\left[n\right]\right\}}_{i\in N,n\in \left[T\right]}$ such that:

$$\sum _{n=1}^T\sum _{i=1}^N{J}_{\mathrm{total}}^i\left[n\right]\le K,$$

(19)

subject to the constraints of SP 2. The proof is presented in Appendix A.

4. Solution Approach Based on Deep Q Actor Critic Network (DQACN)

Figure 3 depicts the architecture of the deep Q actor-critic network in which the actor network is optimized by the gradient ascent method, and the critic network is optimized by the gradient descent method. Initially, the agent observes the state of the environment, and it utilizes the actor network. (policy function network) to take action based on transition probabilities. The action is applied to the environment, resulting in a new state and reward. Then, the critic network evaluates the new state value and current state value by computing the temporal difference (TD) error, followed by tuning the critic and actor networks. The problem of selecting a random server is solved by using the SoftMax probability function $P^k$, i.e., this function turns the energy cost of every server into the probability distribution, where servers can be selected by knowing their lowest probability score. Once the server is selected, it is allowed to solve optimization problems in order to find the cost-efficient storage, energy and pruning frequency. The server broadcast its solutions to the remaining servers. The recipient servers check whether the received solutions satisfy the constraints. If the constraints are satisfied, then other servers accept the current solution; otherwise, they reject the new solution. Once the pruning is executed locally, the edge server updates the retained data to the rented cloud server, facilitating future data analytics.

Now, we present the state encoding variables, action space encoding variables, and reward and penalty to deploy the deep Q actor-critic network-based agent, representing the environment. The state $s^k\left[n\right]$ for server k at time n is defined as follows, and it is common for SP 1 and SP 2. Note that for the mana state, we consider the old value and the recent value, as backlog and finalized, respectively:

$$s^k\left[n\right]=\left(S_{\mathrm{total}}^k\left[n\right],E_{\mathrm{total}}^k\left[n\right],{\lambda }_T^k\left[n\right],m_k\left[n\right],M_{\mathrm{backlog}}^k\left[n\right],M_{\mathrm{finalized}}^k\left[n\right]\right)$$

(20)

The action space server k at time n is defined for SP1 and SP 2 as:

$$a^k\left[n\right]=S_{total}^k\left[n\right]$$

(21)

$$a^k\left[n\right]=E_{total}^k\left[n\right]$$

(21a)

$$a^k\left[n\right]={\lambda }_P^k\left[n\right]$$

(21b)

The reward $r^k\left[n\right]$ for the agent is defined as the negative of the total cost minus a penalty for violating constraints. For SP 1 and SP 2, the reward is defined as below.

$$r^k\left[n\right]=-J_{\mathrm{total}}^k\left[n\right]-P^k\left[n\right]$$

(22)

$$r^k\left[n\right]=-S_{\mathrm{total}}^k\left[n\right]-P^k\left[n\right]$$

(22a)

$$r^k\left[n\right]=-E_{\mathrm{total}}^k\left[n\right]-P^k\left[n\right]$$

(22b)

Let the total cost be $J_{\mathrm{total}}^k\left[n\right]$.

• For Algorithm 1 (SP 1):

  $$E_{\mathrm{total}}^k\left[n\right];S_{\mathrm{total}}^k\left[n\right]$$

  (23)
• For Algorithm 2 (SP 2):

  $$J_{\mathrm{total}}^k\left[n\right]=w_E{E}_{\mathrm{total}}^k\left[n\right]+w_S{S}_{\mathrm{total}}^k\left[n\right]+w_C{C}_{\sec \mathrm{urity}}^k\left[n\right]$$

  (24)

The penalty term $P^k\left[n\right]$ accounts for violations of key constraints, e.g., security constraints. This ensures that the agent chooses the appropriate pruning frequency by respecting constraints and other objectives.

$$P^k\left[n\right]={\beta }_1·max(0,S_{\mathrm{MIN}}-S_{\mathrm{total}}^k\left[n\right])+{\beta }_2·max(0,S_{\mathrm{edge}}^k\left[n\right]-S_{\mathrm{edge}}^{\max })+{\beta }_3·max(0,{\lambda }_P^k\left[n\right]-{\lambda }_P^{\max })...$$

(25)

We brief out the working of Algorithm 1 and Algorithm 2. We note that Algorithm 1 must be executed to preserve the security, before Algorithm 2 is executed. Algorithm 1 solves SP 1, thereby producing the solution for optimal storage and energy cost. Then, Algorithm 2 consumes the solution of Algorithm 1 and produces the optimal pruning frequency. We discuss the generic layout of Algorithms 1 and 2. At first, parameters are initialised, and the softmax probabilities are calculated for the edge servers. Then, storage and energy in hindsight are proposed, and the regret for those is computed. If the regrets for the storage and energy are non-negative, it is discarded. Otherwise, the solution is accepted. Then, the actor and critic networks are updated, and this process is iterated for the maximum number of iterations. In Algorithm 1, the parameters of the actor-critic network are initialised in all edge servers (Steps 3 to 6). Among the edge servers, a server ($k^{th}$) is selected (Steps 8 to 11) by the choice of SoftMax probability function (Step 10). Steps 13 and 14 present the environment to the agent (now, $k^{th}$ server). Using the storage and energy equations, the agent proposes a new value for the storage and the energy by respecting the strict constraints (Steps 15 and 16). In Steps 17 and 18, the agent evaluates the regret. The agent finalises the storage and energy value in Step 19, which is broadcast to other servers. From Steps 21 to 28, other servers evaluate the solution, i.e. whether the new value for the storage and energy satisfies the constraints. The solution is accepted by the remaining servers if the constraints are satisfied (from Steps 29 to 33). In Step 35, the agent computes the advantage function as the computation of the temporal difference learning (the function is computed by targeting future reward as the dominant factor rather than the immediate reward). Additionally, the agent works out such a step by bootstrapping. Step 36 shows the gradient descent step of the critic network with the learning rate ($\alpha$), whereas Step 37 shows the gradient ascent step of the actor network. At last, the agent outputs the storage and energy values. Note that here the value of the pruning frequency is still null, and it is solved in Algorithm 2. Until Step 11, the agent follows the steps of Algorithm 1. In Step 14, the agent turns the security constraints into part of the objective function. Step 18, additionally, includes the pruning frequency. In contrast to Algorithm 1, the agent updates its actor and critic network parameters after Step 29 by evaluating the advantage function. Then, the new solution is broadcast to the remaining edge servers. From Steps 30 to 38, the edge servers are updating their actor and critic parameters as per the received solution. Finally, every edge server gets the updated storage value, energy value and pruning value.

Next, we analyse the convergence of the algorithms to depict whether the algorithms attain such solutions in sub-linear time, as the regret assures such efficiency.

Algorithm 1 Storage and Energy Optimization (SP 1) with Softmax-based Random Server Selection and Actor-Critic Evaluation

1: Input: N; $s^k\left[0\right]$; $\alpha$, $\gamma$, $T_{max}$; ${\psi }_i$, $m_i\left[0\right]$, ${\delta }_m$, ${\kappa }_m$, $S_i$; $k_h$, $k_s$, $k_{\mathrm{attach},i}$, $k_{\mathrm{fpc},i}$, $k_{\mathrm{mana},i}$, $k_p$, ${\alpha }_i$; $S_{\mathrm{tx}}$, $S_{\mathrm{P}}$; ${\lambda }_T^k\left[n\right]$, $n_{\mathrm{hash}}^k\left[n\right]$, $n_{\mathrm{sig}}^k\left[n\right]$, $Q_i\left[n\right]$, $M_i\left[n\right]$, $D_i\left[n\right]$; ${\lambda }_P^k\left[n\right]$, $M_{\mathrm{backlog}}^k\left[n\right]$, $M_{\mathrm{finalized}}^k\left[n\right]$; $S^k\left[0\right]$, $E_{\mathrm{total}}^k\left[0\right]$; 2: Initialization: 3: for ∀k, $k\in N$do 4:     $rand\leftarrow {\theta }_l^k$ (actor parameter) and $rand\leftarrow {\theta }_l^{lk}$ (critic parameter); 5:     $n_l\leftarrow 0$ (learning step counter) and $T\leftarrow 0$ (time step); 6:     $S^k\left[0\right]$, $E_{\mathrm{total}}^k\left[0\right]$, ${\lambda }_P^k\left[0\right]$; 7: end for 8: while  $T<T_{max}$do 9:     for  ∀k, $k\in N$ do 10:         $P^k={\displaystyle \frac{exp(-E_{\mathrm{total}}^k\left[n\right])}{{\sum }_{j\in N}exp(-E_{\mathrm{total}}^j\left[n\right])}}$; 11:     end for 12:     $k_{\mathrm{selected}}\left[k\right]$; * Prioritize servers with lower energy usage for optimization * 13:     $s^{k_{\mathrm{selected}}}\left[n\right]$: $S^{k_{\mathrm{selected}}}\left[n\right]$, $E_{\mathrm{total}}^{k_{\mathrm{selected}}}\left[n\right]$, ${\lambda }_T^{k_{\mathrm{selected}}}\left[n\right]$, $m_{k_{\mathrm{selected}}}\left[n\right]$, $M_{\mathrm{backlog}}^{k_{\mathrm{selected}}}\left[n\right]$, $M_{\mathrm{finalized}}^{k_{\mathrm{selected}}}\left[n\right]$ (observed local states); 14:     $s^{k_{\mathrm{selected}}}\left[n\right]$: ${\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]\sim {\pi }_l(s^{k_{\mathrm{selected}}}\left[n\right]\mid {\theta }_l^{k_{\mathrm{selected}}})$ (actor’s proposal); 15:     $k_{\mathrm{selected}}$: $S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$ using Equation 9; 16:     $k_{\mathrm{selected}}$: $E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$ using Equation 8; 17:     $k_{\mathrm{selected}}$: $R_{\mathrm{storage}}^{k_{\mathrm{selected}}}\left[n\right]\leftarrow S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]-S_{\mathrm{total}}^{k_{\mathrm{selected}}}\left[n\right]$; 18:     $k_{\mathrm{selected}}$: $R_{\mathrm{energy}}^{k_{\mathrm{selected}}}\left[n\right]\leftarrow E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]-E_{\mathrm{total}}^{k_{\mathrm{selected}}}\left[n\right]$; 19:     $k_{\mathrm{selected}}$: $S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1],E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1],{\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]$ (broadcast); 20:     solution_accepted ← True; 21:     for ∀k, $k\in N\setminus \left\{k_{\mathrm{selected}}\right\}$ do 22:         $R_{\mathrm{storage}}^k\left[n\right]\leftarrow S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]-S_{\mathrm{total}}^k\left[n\right]$; 23:         $R_{\mathrm{energy}}^k\left[n\right]\leftarrow E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]-E_{\mathrm{total}}^k\left[n\right]$; 24:         if $R_{\mathrm{storage}}^k\left[n\right]<0$ or $R_{\mathrm{energy}}^k\left[n\right]<0$ then 25:            solution_accepted ← False; 26:            break; 27:         end if 28:     end for * Ensure the solution benefits all servers before proceeding * 29:     if solution_accepted then 30:         for $\forall k$, $k\in N$ do 31:            ${\lambda }_P^k\left[n\right]\leftarrow {\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]$; 32:            $S^k[n+1]\leftarrow S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$; 33:            $E_{\mathrm{total}}^k[n+1]\leftarrow E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$; 34:         end for 35:         $k_{\mathrm{selected}}$: $A^{k_{\mathrm{selected}}}\left[n\right]\leftarrow R_{\mathrm{storage}}^{k_{\mathrm{selected}}}\left[n\right]+R_{\mathrm{energy}}^{k_{\mathrm{selected}}}\left[n\right]+\gamma V^{{\pi }_l}(s^{k_{\mathrm{selected}}}[n+1]\mid {\theta }_l^{l{k}_{\mathrm{selected}}})-V^{{\pi }_l}(s^{k_{\mathrm{selected}}}\left[n\right]\mid {\theta }_l^{l{k}_{\mathrm{selected}}})$; 36:         $k_{\mathrm{selected}}$: ${\theta }_l^{l{k}_{\mathrm{selected}}}\leftarrow {\theta }_l^{l{k}_{\mathrm{selected}}}+\alpha ·A^{k_{\mathrm{selected}}}\left[n\right]·{\nabla }_{{\theta }_l^{l{k}_{\mathrm{selected}}}}{V}^{{\pi }_l}(s^{k_{\mathrm{selected}}}\left[n\right]\mid {\theta }_l^{l{k}_{\mathrm{selected}}})$; 37:         $k_{\mathrm{selected}}$: ${\theta }_l^{k_{\mathrm{selected}}}\leftarrow {\theta }_l^{k_{\mathrm{selected}}}+\alpha ·A^{k_{\mathrm{selected}}}\left[n\right]·{\nabla }_{{\theta }_l^{k_{\mathrm{selected}}}}log{\pi }_l(s^{k_{\mathrm{selected}}}\left[n\right]\mid {\theta }_l^{k_{\mathrm{selected}}})$; * Apply the solution globally and improve the policy through learning * 38:     end if 39:     $T\leftarrow T+1$; 40: end while * Provide the optimised results for storage and energy across servers * 41: Output: 42:    ${\lambda }_P^k\left[n\right]$, $S^k\left[n\right]$, $E_{\mathrm{total}}^k\left[n\right]$, ∀k, $k\in N$.

Algorithm 2 Joint Pruning Optimization (SP 2) with Softmax-based Random Server Selection and Actor-Critic Evaluation

1: Input: N; $s^k\left[0\right]$; $\alpha$, $\gamma$, $T_{max}$; ${\psi }_i$, $m_i\left[0\right]$, ${\delta }_m$, ${\kappa }_m$, $S_i$; $w_E$, $w_S$, $w_C$; ${\alpha }_1$, ${\alpha }_2$, ${\alpha }_3$, ${\alpha }_4$, $B_{\mathrm{hash}}$, $B_{\mathrm{block}}$, $B_{\mathrm{merkle}}$; $k_h$, $k_s$, $k_{\mathrm{attach},i}$, $k_{\mathrm{fpc},i}$, $k_{\mathrm{mana},i}$, $k_p$, ${\alpha }_i$; $S_{\mathrm{tx}}$, $S_{\mathrm{P}}$; ${\lambda }_T^k\left[n\right]$, $n_{\mathrm{hash}}^k\left[n\right]$, $n_{\mathrm{sig}}^k\left[n\right]$, $Q_i\left[n\right]$, $M_i\left[n\right]$, $D_i\left[n\right]$; ${\lambda }_P^k\left[n\right]$, $M_{\mathrm{backlog}}^k\left[n\right]$, $M_{\mathrm{finalized}}^k\left[n\right]$; $S^k\left[0\right]$, $E_{\mathrm{total}}^k\left[0\right]$; 2: Initialization: 3: for ∀k, $k\in N$do 4:     $rand\leftarrow {\theta }_l^k$ and $rand\leftarrow {\theta }_l^{lk}$; 5:     $n_l\leftarrow 0$ and $T\leftarrow 0$; 6:     $S^k\left[0\right]$, $E_{\mathrm{total}}^k\left[0\right]$, ${\lambda }_P^k\left[0\right]$; 7: end for 8: while  $T<T_{max}$do 9:     for  ∀k, $k\in N$ do 10:         $P^k={\displaystyle \frac{exp(-E_{\mathrm{total}}^k\left[n\right])}{{\sum }_{j\in N}exp(-E_{\mathrm{total}}^j\left[n\right])}}$; 11:     end for 12:     $k_{\mathrm{selected}}\left[k\right]$; * Prioritize servers with lower energy usage for optimization * 13:     $s^{k_{\mathrm{selected}}}\left[n\right]$: $S^{k_{\mathrm{selected}}}\left[n\right]$, $E_{\mathrm{total}}^{k_{\mathrm{selected}}}\left[n\right]$, ${\lambda }_T^{k_{\mathrm{selected}}}\left[n\right]$, $m_{k_{\mathrm{selected}}}\left[n\right]$, $M_{\mathrm{backlog}}^{k_{\mathrm{selected}}}\left[n\right]$, $M_{\mathrm{finalized}}^{k_{\mathrm{selected}}}\left[n\right]$; 14:     $k_{\mathrm{selected}}$: $C_{\sec \mathrm{urity}}\left({\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]\right)$ using Equation 15; 15:     $k_{\mathrm{selected}}$: $S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$ using Equation 9; 16:     $k_{\mathrm{selected}}$: $E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$ using Equation 8; 17:     $k_{\mathrm{selected}}$: $J_{\mathrm{proposed}}$ using Equation 16; 18:     $k_{\mathrm{selected}}$: $S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$, $E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$, ${\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]$ (broadcast); 19:     solution_accepted ← True; 20:     for  ∀k, $k\in N\setminus \left\{k_{\mathrm{selected}}\right\}$ do 21:         $J_{\mathrm{local}}^k$ using Equation 16; 22:         if $J_{\mathrm{local}}^k<J_{\mathrm{proposed}}$ then 23:            solution_accepted ← False; 24:            break; 25:         end if 26:     end for * Ensure the proposed solution improves the joint cost for all servers * 27:     if solution_accepted then 28:         $k_{\mathrm{selected}}$: $R_{\mathrm{joint}}^{k_{\mathrm{selected}}}\left[n\right]\leftarrow J_{\mathrm{proposed}}-J_{\mathrm{prune}}\left({\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]\right)$; 29:         $k_{\mathrm{selected}}$: $A^{k_{\mathrm{selected}}}\left[n\right]\leftarrow R_{\mathrm{joint}}^{k_{\mathrm{selected}}}\left[n\right]+\gamma V^{{\pi }_l}(s^{k_{\mathrm{selected}}}[n+1]\mid {\theta }_l^{l{k}_{\mathrm{selected}}})-V^{{\pi }_l}(s^{k_{\mathrm{selected}}}\left[n\right]\mid {\theta }_l^{l{k}_{\mathrm{selected}}})$; 30:         for each edge server $k\in N$ do 31:            ${\lambda }_P^k\left[n\right]\leftarrow {\lambda }_P^{k_{\mathrm{selected}}}\left[n\right]$; 32:            $S^k[n+1]\leftarrow S_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$; 33:            $E_{\mathrm{total}}^k[n+1]\leftarrow E_{\mathrm{proposed}}^{k_{\mathrm{selected}}}[n+1]$; 34:            k: $R_{\mathrm{joint}}^k\left[n\right]\leftarrow J_{\mathrm{proposed}}-J_{\mathrm{prune}}\left({\lambda }_P^k\left[n\right]\right)$; 35:            k: $A^k\left[n\right]\leftarrow R_{\mathrm{joint}}^k\left[n\right]+\gamma V^{{\pi }_l}(s^k[n+1]\mid {\theta }_l^{lk})-V^{{\pi }_l}(s^k\left[n\right]\mid {\theta }_l^{lk})$; 36:            ${\theta }_l^{lk}\leftarrow {\theta }_l^{lk}+\alpha ·A^k\left[n\right]·{\nabla }_{{\theta }_l^{lk}}{V}^{{\pi }_l}(s^k\left[n\right]\mid {\theta }_l^{lk})$; 37:            ${\theta }_l^k\leftarrow {\theta }_l^k+\alpha ·A^k\left[n\right]·{\nabla }_{{\theta }_l^k}log{\pi }_l(s^k\left[n\right]\mid {\theta }_l^k)$; 38:         end for 39:         $n_l\leftarrow n_l+1$; * Apply the solution globally and refine the policy with joint cost optimization * 40:     end if 41:     $T\leftarrow T+1$; 42: end while * Provide the optimized results for storage and energy across servers * 43: Output: 44:    ${\lambda }_P^k\left[n\right]$, $S^k\left[n\right]$, $E_{\mathrm{total}}^k\left[n\right]$, $\forall k$, $k\in N$.

4.1. Convergence Proof

While the deep Q actor-critic-based method is used for solving optimization problems in different contexts, we attempt to present the formulated regret functions indeed converge to the optimal solution in sub-linear time.

We present the assumption for proving theorems in Appendix A.

Theorem 1.

Under the assumptions of finite state and action spaces, bounded rewards, Robbins-Monro learning rates, and an ergodic MDP, Algorithm 1 converges to a locally optimal policy ${\pi }^{*}$ for SP1, minimizing energy and storage costs subject to constraints $C_1$ to $C_8$, with sublinear regret $R\left(T\right)=O\left(\sqrt{T}\right)$.

The proof of this theorem is presented in the appendix section (Appendix A).

Theorem 2.

Under the assumptions of finite spaces, bounded rewards, Robbins-Monro learning rates, an ergodic MDP, Lipschitz continuity, and bounded security costs, Algorithm 2 converges to a locally optimal policy ${\pi }^{*}$ forming a Nash equilibrium for SP2, jointly optimizing pruning frequency, energy, and storage, with sublinear regret $R_{\mathrm{joint}}\left(T\right)=O\left(\sqrt{T}\right)$.

The proof is presented in Appendix A.

Now, we show that the proposed algorithm for SP 1 truly satisfies security properties, namely, first pre-image resistance, second pre-image resistance, collision resistance and integrity against active adversaries.

4.2. Adversary Model

We consider the Dolev-Yao adversary model for the adversary $\mathcal{A}$. $\mathcal{A}$ can overhear communications between the smart meters and edge servers, taking the passive role. The attack surface for the adversary is considered at the edge servers. In the active role, $\mathcal{A}$ can substitute fake transactions in the storage held at every edge server. The adversary can violate the first preimage resistance, i.e., it can find the inputs for a targeted hash. By violating the second preimage resistance property, $\mathcal{A}$ can identify the other inputs that share the same hash value. In the event of breaking the collision resistance property, the adversary can find other inputs for a targeted input and hash. While counterfeating the integrity, $\mathcal{A}$ can substitute fake transactions to claim that they are legitimate.

4.3. Formal Security Analysis

We formally analyze how Algorithm 1 safeguards security against a classical active adversary using the universal composability framework, so we start with the relevant terminology of the framework. We claim the sense of security for the only properties claimed in the below propositions. The security is attained by satisfying the properties.

• Real World: Parties execute the protocol (Algorithm 1) in the presence of an adversary $\mathcal{A}$.
• Ideal World: Parties interact with an ideal functionality $\mathcal{F}$ that securely implements the desired behavior, with a simulator $\mathcal{S}$ emulating the adversary.
• Security Definition: A protocol $\mu$ UC-realizes $\mathcal{F}$ if, for any adversary $\mathcal{A}$, there exists a simulator $\mathcal{S}$ such that every environment $\mathcal{Z}$ cannot distinguish the real-world execution (with $\mu$ and $\mathcal{A}$) from the ideal-world execution (with $\mathcal{F}$ and $\mathcal{S}$).

Refer to Appendix A for the ideal functionality (${\mathcal{F}}_{PRUNE}$).

• Proposition 1: Algorithm 1 ensures the first preimage resistance with non-negligible probability.
• Proposition 2: Algorithm 1 satisfies the second preimage resistance with non-negligible probability.
• Proposition 3: Collision resistance with non-negligible probability is achieved by Algorithm 1.
• Proposition 4: Integrity with non-negligible probability is ratified by Algorithm 1.

We prove the propositions by showing that Algorithm 1 UC-realizes ${\mathcal{F}}_{\mathrm{PRUNE}}$, satisfying the respective security property (refer to Appendix A for the proofs).

5. Simulation and Analysis

In this section, to asses the effectiveness of the proposed algorithms, we ask the following sub-research questions (SRQs):

• SRQ 1: How much efficiency is attained by DQACN (Algorithm 1 and Algorithm 2) w.r.t. the local snapshot mechanism in IOTA 2.0?
• SRQ 2: How far do DQACN (Algorithm 1 and Algorithm 2) satisfy the constraints strictly?
• SRQ 3: What is the cost-benefit of the proposed algorithms over local snapshot in IOTA 2.0?

To answer SRQ 1, we perform the Monte Carlo simulation as per the network setting of IOTA 2.0. Moreover, we simulate the network behavior of the IOTA 2.0 to capture the actual effectiveness of the proposed algorithms. We consider the local and global snapshots supported in IOTA 2.0 and IOTA 1.5, respectively, and the deep Q network and random elimination methods for comparison.

We consider a random process of transactions sent by the smart meters received by edge servers, with the IOTA Tangle network between smart meters and edge servers. We capture the input variables with the appropriate probability distribution, modeling the activity of the mana with the appropriate probability distribution.

Refer Appendix C for the probability distributions considered for the random process.

We use the values specified in Table A3 for different parameters. As we considered the temporal difference learning method in the actor-critic network, the network learns the optimal policy by bootstrapping (initial guess followed by iterative refinement), attaining convergence in the critic network. Figure 4 presents the simulation result. 1 in Figure 4 demonstrates the total cost, whereas Sub-figures 2 and 3 show the storage and energy costs, respectively. In the three sub-figures, Algorithm 2 (Alg 2) outperforms all schemes, typically consuming lower cost than the local snapshot. Sub-figure 1 depicts the total cost w.r.t. increasing edge servers, summing the energy cost and storage cost presented in sub-figures 2 and 3, respectively. When the edge server reaches 100, Algorithm (Alg 1) attains approximately 50% cost reduction in comparison with the local snapshot as it is driven by the static pruning rules. Alg 2 shows the cost reduction of 60% when compared with the local snapshot. This reduction trend is reflected in the ordinary deep-Q network as well; however, the actor-critic-based method outperforms all methods, namely local and global snapshots, random elimination and deep Q method.

In Sub-figure 2, the proposed algorithms’ storage cost is almost zero for proposed algorithms because of the aggressive pruning frequency. This trend shifts for increasing servers due to the time evolution model of the storage function and security constraint satisfaction. Sub-figure 3 demonstrates the energy cost of the proposed algorithms, showing lower consumption of energy. Notably, Algorithm 2 consumes the least energy, whereas the global snapshot incurs the highest energy cost. Remaining methods contribute to the middle values between the proposed algorithms and the global snapshot.

Sub-figure 4 shows the amount of pruning frequency versus the number of servers where the highest values are attained by the random elimination since the pruning decisions are not cost aware; however, the least value comes from the local snapshot because of static rules, say prune only if the storage capacity exceeds 85% of the total storage. Proposed algorithms have higher pruning frequencies to optimize the storage and energy costs.

Sub-figure 5 shows the total cost in relation to transaction volume, with Algorithm 1 performing best by minimizing costs, followed by Algorithm 2 in energy consumption. Sub-figure 6 indicates that both Algorithms 1 and 2 achieve the same lowest storage cost. Sub-figure 7 highlights that Algorithms 1 and 2 outperform others in energy costs. Regarding pruning frequency, the local snapshot shows the lowest costs, but effective pruning decisions are necessary for optimal savings, leading to a higher pruning frequency in the proposed algorithms. Sub-figures 9, 10, and 11 demonstrate the quick convergence of pruning decisions for total cost, storage cost, and energy cost in the proposed algorithms compared to others.

SRQ 2 is answered in Sub-figure 12. For all the pruning decisions, Algorithm 1 strictly satisfies all constraints. Algorithm 2 occasionally satisfies the constraints, whereas the remaining methods, such as local and global snapshots, and the deep Q method, do not satisfy constraints, resulting in security violations and infeasibility in data analytics.

We answer SRQ 3 as follows. We consider five cities, namely Amsterdam, Beijing, Cincinnati, Delhi and Milan for our study. The profile of the cities, including population, smart meter usage, number of edge servers used and renting rate, are depicted in Table A4 (Appendix C). We present the case for Amsterdam. For all cities, we consider the rental rate of the edge server as 5000 USD/year (Appendix C).

Table 2. Average Cost Savings Using Algorithms 1 and 2 for Smart Meters in Five Cities (USD, 2025-2029).

City	1-Year	2-Year	3-Year	4-Year	5-Year
Amsterdam	5,585,700	11,171,400	16,757,100	22,342,800	27,928,500
Beijing	136,652,240	273,304,480	409,956,720	546,608,960	683,261,200
Cincinnati	1,857,440	3,714,880	5,572,320	7,429,760	9,287,200
Delhi	204,978,360	409,956,720	614,935,080	819,913,440	1,024,891,800
Milan	8,693,120	17,386,240	26,079,360	34,772,480	43,465,600

Table 2. Average Cost Savings Using Algorithms 1 and 2 for Smart Meters in Five Cities (USD, 2025-2029).

City	1-Year	2-Year	3-Year	4-Year	5-Year
Amsterdam	5,585,700	11,171,400	16,757,100	22,342,800	27,928,500
Beijing	136,652,240	273,304,480	409,956,720	546,608,960	683,261,200
Cincinnati	1,857,440	3,714,880	5,572,320	7,429,760	9,287,200
Delhi	204,978,360	409,956,720	614,935,080	819,913,440	1,024,891,800
Milan	8,693,120	17,386,240	26,079,360	34,772,480	43,465,600

6. Implementation Study

In this section, we implement the proposed algorithms to answer the following research question.

SRQ 4: What is the effectiveness of the proposed algorithms in the actual edge server environment?

SRQ 5: How do energy and other industries, and the IOTA community benefit from this research?

To answer SRQ 4, we consider the following environment for implementing and testing the proposed algorithms: Linux (operating system type), x86_64 (processor), Intel(R) Xeon(R) CPU @ 2.20GHz (processor brand and operating frequency), 12 GB (RAM) and 2 (CPU cores). We use Python 3 as the programming language for the implementation. In order to assess the proposed algorithms in a real-world setting, we implement the IOTA node with mana and the DQACN agent. We consider the test cases mentioned in Table A2. We test the algorithms to know what extent the storage, energy and pruning frequency are used to understand their baseline operation. For this case, we measure the memory consumed, energy consumed and pruning fraction held initially. To check the robustness of the algorithm towards growing transactions, we subject the second case, which is detailed in Table A2 (Appendix B). To check the security assurance of the proposed algorithms, we use the third test case. To understand the reliability of the algorithms, we use the final test case. Figure A1 (Appendix B) shows the result of the test cases. It is apparent that all of the test cases succeeded, making the proposed algorithms suitable for real-world deployment.

Our developed algorithms can be found in the link 3, and the algorithms can be used for the real-time deployment, or it can serve as the guideline for extended implementation. This answers our SRQ 5. We made the developed algorithms as a standalone file ("dr_agent.py") so that it can be easily used for deployment. Our implementation can be used by the energy companies for the upgradation. Also, the IOTA Tangle foundation can use the developed algorithms as an alternative for the local snapshot.

7. Conclusions

This article identifies key limitations associated with the local snapshot of IOTA 2.0, particularly security violations and the infeasibility of data analytics. We proposed two algorithms based on the deep Q actor-critic network in reinforcement learning. Algorithm 1 optimizes storage cost and energy cost by strictly satisfying constraints (inclusive of security constraints) for a random edge sever, and the solutions are accepted by the remaining servers provided that constraints are satisfied, and Algorithm 2 optimizes pruning frequency without the restriction on the constraint satisfaction. Monte Carlo simulation confirms the practical efficacy of the proposed algorithms. Typically, a significant amount of improvement in efficiency is attained by the algorithms, satisfying security requirements. The study evaluated the cost-benefit of the proposed algorithms by considering five cities. This article studied the real-world implementation of the proposed algorithms and made the algorithms available, ready for real-world deployment. The solutions proposed in this article can benefit the IOTA Tangle foundation for upgrading its future snapshot mechanism. Further, this research can benefit energy companies renting CSPs by following the proposed algorithms over the conventional local snapshot in IOTA 2.0.

Appendix A.1. Proof for the NP-Hardness of SP 2

Proof. 

We reduce the problem, beginning from SAT. Given a SAT instance with v variables and m clauses: Set $N=v$, $T=1$ in Equation (19). Map variable $x_j$ to server $i=j$. Let ${\lambda }_P^j\left[1\right]=1$ if $x_j=\mathrm{true}$, 0 if $x_j=\mathrm{false}$. For each clause $c_k$, the cost is defined as $C_{\mathrm{clause}}^k=1$ if $c_k$ is not satisfied (all literals false), 0 otherwise. Set $J_{\mathrm{total}}^i\left[1\right]={\sum }_{k=1}^m{C}_{\mathrm{clause}}^k$. Set $K=0$. This satisfies constraints.

If SAT is satisfiable, there exists $\left\{{\lambda }_P^i\left[1\right]\right\}$ such that all clauses are satisfied ($C_{\mathrm{clause}}^k=0$), so ${\sum }_{k=1}^m{C}_{\mathrm{clause}}^k=0\le K$, and PODP returns "True". If SAT is not satisfiable, at least one clause is unsatisfied ($C_{\mathrm{clause}}^k=1$), so ${\sum }_{k=1}^m{C}_{\mathrm{clause}}^k\ge 1>K$, and PODP returns "False."

The time complexity associated with the construction takes $O(m·v)$. Since SAT is NP-complete and reduces to PODP in polynomial time, PODP is NP-hard, which is NP-complete. Thus, the optimization problem (SP 2) is NP-hard. □

Appendix A.2. Assumptions

Finite Spaces (A1): To ensure bounded space for the agent, let the state spaces S and action spaces A are finite and discrete [19]. This makes the finite landscape for the agent to explore and make decisions.

Bounded Rewards (A2): Reward $r(s,a)=-E_{\mathrm{total}}\left[n\right]+P\left[n\right]$r $-S_{\mathrm{total}}\left[n\right]+P\left[n\right]$ satisfies $\left|r(s,a)\right|\le R_{max}$ [20]. This ensures the reward is finite and convergent rather than being divergent.

Learning Rates (A3): The learning rate ${\alpha }_t$ satisfies ${\sum }_t{\alpha }_t=\infty$, ${\sum }_t{\alpha }_t^2<\infty$, and $0<\gamma <1$ a.k.a. Robbins-Monro learning rates [21]. The divergent condition indicates that the learning step is not stuck with local optimality, whereas the convergent condition implies that the learning step indeed converges to a stationary point.

Ergodic Markov Decision Process (MDP) (A4): The MDP has a stationary distribution ${\mu }^{\pi }\left(s\right)$ [20]. By the property of the ergodicity, the time average equals the ensemble average, that is, the statistical property observed in different states (pruning decision) is equal to the time average of the system.

Lipschitz Continuity (A5): ${\nabla }_{\theta }J\left(\theta \right)$ is Lipschitz continuous with constant L [22]. In other words, the gradient of the function is continuous and it is bounded by a constant.

Convex Weights (A6): $w_E,w_S,w_C>0$, $w_E+w_S+w_C=1$. This condition underscores the importance of the objective function, meaning that every objective function in a joint optimization problem contributes to certain importance.

Bounded Security (A7): $C_{\sec \mathrm{urity}}$ is continuous and bounded. This assumption ensures that security constraints converted to the objective function have a finite landscape while exploring for solutions.

Appendix A.3. Proof of Theorem 1:

Proof. 

We prove Theorem 1 by establishing the convergence of the Actor-Critic method in Algorithm 1, leveraging the Policy Gradient Theorem [23] and TD(0) convergence [24].

We begin with A1 and A2.

The critic updates the value function using TD(0) (initial guess):

$$\begin{array}{cc}\hfill {\delta }^k\left[n\right]& =r^k\left[n\right]+\gamma V^{\pi }\left(s^k[n+1]\right|{\theta }^{c,k})-V^{\pi }\left(s^k\left[n\right]\right|{\theta }^{c,k}),\hfill \end{array}$$

(A1)

$$\begin{array}{cc}\hfill {\theta }^{c,k}& \leftarrow {\theta }^{c,k}+{\alpha }_n{\delta }^k\left[n\right]{\nabla }_{{\theta }^{c,k}}{V}^{\pi }\left(s^k\left[n\right]\right|{\theta }^{c,k}).\hfill \end{array}$$

(A2)

The TD error’s expectation is:

$$\mathbb{E}\left[{\delta }^k\left[n\right]\right|s^k\left[n\right]]=Q^{\pi }(s^k\left[n\right],a^k\left[n\right])-V^{\pi }\left(s^k\left[n\right]\right).$$

(A3)

By [24], TD(0) is a contraction mapping with factor $\gamma$:

$$\parallel V_{n+1}^{\pi }-V^{\pi }{\parallel }_{\infty }\le \gamma {\parallel V_n^{\pi }-V^{\pi }\parallel }_{\infty }.$$

(A4)

Given the Robbins-Monro conditions (A3), ${\theta }^{c,k}$ converges almost surely to parameters yielding $V^{\pi }\left(s\right)$ [20].

The actor updates:

$$\begin{array}{cc}\hfill {\theta }^k& \leftarrow {\theta }^k+{\alpha }_n{A}^k\left[n\right]{\nabla }_{{\theta }^k}log\pi \left(a^k\left[n\right]\right|s^k\left[n\right],{\theta }^k),\hfill \end{array}$$

(A5)

$$\begin{array}{cc}\hfill A^k\left[n\right]& =r^k\left[n\right]+\gamma V^{\pi }\left(s^k[n+1]\right)-V^{\pi }\left(s^k\left[n\right]\right).\hfill \end{array}$$

(A6)

The policy gradient theorem states:

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\mu }^{\pi }}[{\nabla }_{\theta }log\pi \left(a\right|s,\theta )Q^{\pi }(s,a)].$$

(A7)

Since $A^k\left[n\right]\to Q^{\pi }(s,a)-V^{\pi }\left(s\right)$ as the critic converges, the gradient is unbiased.

Server selection uses:

$$P^k={\displaystyle \frac{exp(-E_{\mathrm{total}}^k\left[n\right])}{{\sum }_{j\in N}exp(-E_{\mathrm{total}}^j\left[n\right])}}.$$

(A8)

By Jensen’s inequality for the concave function $exp(-x)$:

$$\mathbb{E}[exp(-E_{\mathrm{total}}^k\left[n\right])]\ge exp(-\mathbb{E}\left[E_{\mathrm{total}}^k\left[n\right]\right]).$$

(A9)

As $E_{\mathrm{total}}^k\left[n\right]$ stabilizes, $P^k$ favors low-cost servers, ensuring exploration [19].

Regret is given by:

$$R\left(T\right)=\sum _{t=1}^T(C^{*}\left[t\right]-C\left[t\right]),C\left[t\right]=E_{\mathrm{total}}\left[t\right]\mathrm{or}{S}_{\mathrm{total}}\left[t\right].$$

(A10)

The performance difference lemma [25] gives:

$$J\left({\theta }_{t+1}\right)-J\left({\theta }_t\right)\ge {\displaystyle \frac{1}{1-\gamma }}{\mathbb{E}}_{{\mu }^{{\pi }_t}}\left[A^{{\pi }_{t+1}}(s,a)\right].$$

(A11)

With $A^k\left[n\right]>0$ for improvements and constraints enforced, $J\left({\theta }_t\right)$ increases. By Lipschitz continuity (A5), $R\left(T\right)=O\left(\sqrt{T}\right)$.

As $n_l\to \infty$, ${\nabla }_{\theta }J\left(\theta \right)\to 0$, and the ergodic MDP (A4) ensures convergence to a local optimum ${\pi }^{*}$ satisfying $C_1$ to $C_8$.

Thus, Algorithm 1 converges to ${\pi }^{*}$ with sublinear regret. □

Appendix A.4. Proof of Theorem 2:

Proof. 

We prove Theorem 2 using cooperative Actor-Critic convergence [26] and Nash equilibrium properties [27].

The reward is (A7 holds in addition to A2):

$$r^k\left[n\right]=-J_{\mathrm{total}}^k\left[n\right],J_{\mathrm{total}}^k\left[n\right]=w_E{E}_{\mathrm{total}}^k\left[n\right]+w_S{S}_{\mathrm{total}}^k\left[n\right]+w_C{C}_{\sec \mathrm{urity}}^k\left[n\right],$$

(A12)

where $C_{\sec \mathrm{urity}}^k\left[n\right]={\sum }_{i=1}^4{\alpha }_i{\displaystyle \frac{{\lambda }_P^k\left[n\right]}{S^k\left[n\right]}}{\displaystyle \frac{2^{k_i}}{B_i}}$ is bounded (${\lambda }_P^k\left[n\right]\in [0,1]$, $S^k\left[n\right]\ge S_{min}$) (A7).

As in SP1:

$$\begin{array}{cc}\hfill {\theta }^{c,k}& \leftarrow {\theta }^{c,k}+{\alpha }_n{\delta }^k\left[n\right]{\nabla }_{{\theta }^{c,k}}{V}^{\pi }\left(s^k\left[n\right]\right|{\theta }^{c,k}),\hfill \end{array}$$

(A13)

$$\begin{array}{cc}\hfill {\delta }^k\left[n\right]& =r^k\left[n\right]+\gamma V^{\pi }\left(s^k[n+1]\right)-V^{\pi }\left(s^k\left[n\right]\right).\hfill \end{array}$$

(A14)

TD(0) ensures $V^{\pi }\left(s^k\left[n\right]\right|{\theta }^{c,k})\to V^{\pi }\left(s\right)$ [24].

The actor updates:

$${\theta }^k\leftarrow {\theta }^k+{\alpha }_n{A}^k\left[n\right]{\nabla }_{{\theta }^k}log\pi \left(a^k\left[n\right]\right|s^k\left[n\right],{\theta }^k),$$

(A15)

with $A^k\left[n\right]=r^k\left[n\right]+\gamma V^{\pi }\left(s^k[n+1]\right)-V^{\pi }\left(s^k\left[n\right]\right)$, and:

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{{\mu }^{\pi }}[{\nabla }_{\theta }log\pi \left(a\right|s,\theta )Q^{\pi }(s,a)].$$

(A16)

A solution is accepted if:

$$J_{\mathrm{total}}^k\left[n\right]<J_{\mathrm{proposed}},\forall k\in N\setminus \left\{k_{\mathrm{selected}}\right\}.$$

(A17)

This ensures cooperative improvement, approximating a Nash equilibrium [27].

For the convex combination:

$$\mathbb{E}\left[J_{\mathrm{total}}^k\right]\le w_E\mathbb{E}\left[E_{\mathrm{total}}^k\right]+w_S\mathbb{E}\left[S_{\mathrm{total}}^k\right]+w_C\mathbb{E}\left[C_{\sec \mathrm{urity}}^k\right].$$

(A18)

This bounds the expected cost, stabilizing optimization.

Regret is:

$$R_{\mathrm{joint}}\left(T\right)=\sum _{t=1}^T(w_E{E}_{\mathrm{total}}^{*}\left[t\right]+w_S{S}_{\mathrm{total}}^{*}\left[t\right]-J_{\mathrm{total}}\left[t\right]).$$

(A19)

Cooperative updates ensure:

$$J\left({\theta }_{t+1}\right)\le J\left({\theta }_t\right)-ϵ\mathrm{or}J\left({\theta }_{t+1}\right)=J\left({\theta }_t\right).$$

(A20)

By Lipschitz continuity:

$$\parallel J\left({\theta }_{t+1}\right)-J\left({\theta }_t\right)\parallel \le L{\alpha }_t\parallel A^k\left[n\right]{\nabla }_{\theta }log\pi \parallel .$$

(A21)

As ${\alpha }_t\to 0$, ${\nabla }_{\theta }J\left(\theta \right)\to 0$ (A3), and $R_{\mathrm{joint}}\left(T\right)=O\left(\sqrt{T}\right)$.

At convergence, no server can improve $J_{\mathrm{total}}^k$ unilaterally, forming a Nash equilibrium [26].

Thus, Algorithm 2 converges to ${\pi }^{*}$ with sublinear regret. □

Appendix A.5. Ideal Functionality F PRUNE:

We define an ideal functionality ${\mathcal{F}}_{\mathrm{PRUNE}}$ that captures the secure pruning mechanism for IOTA 2.0, ensuring the cryptographic properties claimed in Propositions 1–4. The functionality interacts with edge servers ($P_1,\dots ,P_N$), smart meters ($S{M}_1,\dots ,S{M}_M$), and a CSP, handling transactions and pruning while enforcing security.

• Initialization:
  • Stores a ledger of transactions $\mathcal{L}=\left\{(t{x}_i,h_i,si{g}_i,merkl{e}_i)\right\}$, where $h_i=H\left(t{x}_i\right)$, $si{g}_i$ is the Winternitz One-Time Signature (WOTS), and $merkl{e}_i$ is the Merkle root.
  • Maintains storage $S_i\left[n\right]$ and energy $E_i\left[n\right]$ for each server $P_i$ at time n.
  • Security parameters: $k=256$ (hash length), ensuring $2^{256}$ preimage resistance, $2^{128}$ collision resistance (birthday paradox), and $2^{256}$ integrity for WOTS.
• Transaction Submission:
  • On input $(tx,\mathrm{send})$ from $S{M}_j$:

    –

    Compute $h=H\left(tx\right)$, $sig=\mathrm{WOTS}.\mathrm{Sign}\left(tx\right)$, and update Merkle root.

    –

    If H is collision-resistant ($\mathrm{Pr}[\mathrm{find}t{x}^{\prime }\ne tx:H\left(t{x}^{\prime }\right)=h]\le 2^{-128}$) and $sig$ verifies, add $(tx,h,sig,merkle)$ to $\mathcal{L}$.

    –

    Send $(tx,h,sig)$ to $P_i$ for storage, updating $S_i\left[n\right]\leftarrow S_i[n-1]+S_{\mathrm{tx}}$.

• Pruning Request:
  • On input $(\mathrm{prune},{\lambda }_P^i\left[n\right])$ from $P_i$:

    –

    Check constraints:

    *

    $S_i\left[n\right]\ge S_H^{min}$ (historical data retention).

    *

    ${\lambda }_P^i\left[n\right]·S_P\le S_{max}$ (pruning limit).

    *

    Bills settled ($B_{\mathrm{paid},i}\left[n\right]=1$).

    *

    Security: Ensure pruning retains $2^{256}$ first/second preimage resistance, $2^{128}$ collision resistance, and $2^{256}$ integrity (via sufficient transactions for hash/Merkle checks).

    –

    If valid, remove fraction ${\lambda }_P^i\left[n\right]$ of transactions, updating $S_i\left[n\right]\leftarrow S_i[n-1]-{\lambda }_P^i\left[n\right]·S_P$.

    –

    Output $(S_i\left[n\right],\mathrm{success})$.

• Adversary Interaction:
  • $\mathcal{A}$ can submit fake transactions $(t{x}^{\prime },h^{\prime },si{g}^{\prime })$.
  • ${\mathcal{F}}_{\mathrm{PRUNE}}$ rejects if:

    –

    $h^{\prime }\ne H\left(t{x}^{\prime }\right)$ (first preimage resistance).

    –

    $t{x}^{\prime }\ne tx$ but $H\left(t{x}^{\prime }\right)=H\left(tx\right)$ (second preimage/collision resistance).

    –

    $si{g}^{\prime }$ fails WOTS verification (integrity).

  • $\mathcal{A}$ can query $\mathcal{L}$, but cannot modify valid entries.
• Output:
  • Return updated $S_i\left[n\right]$, $E_i\left[n\right]$, and $\mathcal{L}$ to $P_i$, $S{M}_j$, and CSP, ensuring no security violations.

This functionality ensures pruning preserves cryptographic properties by rejecting adversarial attempts to violate preimage resistance, collision resistance, or integrity.

Appendix A.5.1. Proof of Proposition 1

Algorithm 1 ensures that an adversary $\mathcal{A}$ cannot find an input $t{x}^{\prime }$ such that $H\left(t{x}^{\prime }\right)=h$ for a given hash $h=H\left(tx\right)$ with non-negligible probability.

• Ideal World: In ${\mathcal{F}}_{\mathrm{PRUNE}}$, $\mathcal{A}$ submits $(t{x}^{\prime },h^{\prime })$. The functionality checks $h^{\prime }=H\left(t{x}^{\prime }\right)$, rejecting if false. Since H is modeled as a random oracle with $2^{256}$ output space, $\mathrm{Pr}[H\left(t{x}^{\prime }\right)=h]\le 2^{-256}$, ensuring first preimage resistance.
• Real World: Algorithm 1 prunes transactions while ensuring $S_i\left[n\right]\ge S_H^{min}$, retaining enough transactions to maintain hash security (constraint $C_1$: ${\displaystyle \frac{2^{256}}{B_{\mathrm{hash}}}}$). The hash function H (e.g., SHA-256) is preimage-resistant, and pruning decisions enforce sufficient storage to verify $h=H\left(tx\right)$.
• Simulator $\mathcal{S}$: $\mathcal{S}$ simulates $\mathcal{A}$’s view by generating transactions $(tx,h,sig)$ using a random oracle H. If $\mathcal{A}$ submits $t{x}^{\prime }$, $\mathcal{S}$ computes $h^{\prime }=H\left(t{x}^{\prime }\right)$ and forwards to ${\mathcal{F}}_{\mathrm{PRUNE}}$, which rejects unless $h^{\prime }=h$. Since $\mathcal{S}$ uses the same H, the probability of finding $t{x}^{\prime }$ such that $H\left(t{x}^{\prime }\right)=h$ remains $2^{-256}$.
• Indistinguishability: The environment $\mathcal{Z}$ sees identical outputs in both worlds: valid transactions are stored, and invalid ones rejected. The constraint $S_i\left[n\right]\ge S_H^{min}$ ensures enough data to enforce $2^{256}$ preimage resistance, matching ${\mathcal{F}}_{\mathrm{PRUNE}}$’s rejection of fake inputs.
• Algorithm 1 UC-realizes ${\mathcal{F}}_{\mathrm{PRUNE}}$ for first preimage resistance, as $\mathcal{A}$’s success probability is negligible ($\le 2^{-256}$).

Appendix A.5.2. Proof of Proposition 2

Algorithm 1 sanctions $\mathcal{A}$ cannot find $t{x}^{\prime }\ne tx$ such that $H\left(t{x}^{\prime }\right)=H\left(tx\right)$ with non-negligible probability.

• Ideal World: ${\mathcal{F}}_{\mathrm{PRUNE}}$ rejects $(t{x}^{\prime },h^{\prime })$ if $h^{\prime }=H\left(tx\right)$ but $t{x}^{\prime }\ne tx$. For a random oracle H, $\mathrm{Pr}[H\left(t{x}^{\prime }\right)=H\left(tx\right)\mid t{x}^{\prime }\ne tx]\le 2^{-256}$, ensuring second preimage resistance.
• Real World: Algorithm 1’s constraint $C_1$ (${\displaystyle \frac{2^{256}}{B_{\mathrm{hash}}}}$) ensures pruning retains transactions to verify $H\left(tx\right)$. The hash function’s second preimage resistance guarantees that finding $t{x}^{\prime }\ne tx$ with $H\left(t{x}^{\prime }\right)=H\left(tx\right)$ is computationally infeasible.
• Simulator $\mathcal{S}$: $\mathcal{S}$ simulates $\mathcal{A}$’s view, computing $h=H\left(tx\right)$ for valid transactions. If $\mathcal{A}$ submits $t{x}^{\prime }\ne tx$, $\mathcal{S}$ computes $h^{\prime }=H\left(t{x}^{\prime }\right)$ and forwards to ${\mathcal{F}}_{\mathrm{PRUNE}}$, which rejects unless $h^{\prime }=h$, with probability $2^{-256}$.
• Indistinguishability: $\mathcal{Z}$ cannot distinguish real and ideal worlds, as both reject $t{x}^{\prime }\ne tx$ with $H\left(t{x}^{\prime }\right)=H\left(tx\right)$. Algorithm 1’s storage constraints ensure verification data persists, aligning with ${\mathcal{F}}_{\mathrm{PRUNE}}$.
• Algorithm 1 UC-realizes ${\mathcal{F}}_{\mathrm{PRUNE}}$ for second preimage resistance, with $\mathcal{A}$’s success probability $\le 2^{-256}$.

Appendix A.5.3. Proof of Proposition 3

Algorithm 1 satisfies $\mathcal{A}$ cannot find $t{x}_1\ne t{x}_2$ such that $H\left(t{x}_1\right)=H\left(t{x}_2\right)$ with non-negligible probability.

• Ideal World: ${\mathcal{F}}_{\mathrm{PRUNE}}$ rejects pairs $(t{x}_1,t{x}_2)$ if $H\left(t{x}_1\right)=H\left(t{x}_2\right)$ and $t{x}_1\ne t{x}_2$. For a random oracle, the birthday paradox bounds $\mathrm{Pr}[H\left(t{x}_1\right)=H\left(t{x}_2\right)]\le 2^{-128}$, ensuring collision resistance.
• Real World: Algorithm 1 enforces $C_1$ (${\displaystyle \frac{2^{128}}{B_{\mathrm{hash}}}}$), retaining transactions to detect collisions. The hash function’s collision resistance ensures finding $t{x}_1,t{x}_2$ with $H\left(t{x}_1\right)=H\left(t{x}_2\right)$ requires $\approx 2^{128}$ operations.
• Simulator $\mathcal{S}$: $\mathcal{S}$ generates $h_i=H\left(t{x}_i\right)$ for valid transactions. If $\mathcal{A}$ submits $(t{x}_1,t{x}_2)$, $\mathcal{S}$ computes $h_1=H\left(t{x}_1\right)$, $h_2=H\left(t{x}_2\right)$, forwarding to ${\mathcal{F}}_{\mathrm{PRUNE}}$, which rejects if $h_1=h_2$. The probability of a collision is $2^{-128}$.
• Indistinguishability: Both worlds reject collisions identically, with Algorithm 1’s constraints ensuring sufficient storage for hash verification, matching ${\mathcal{F}}_{\mathrm{PRUNE}}$.
• Algorithm 1 UC-realizes ${\mathcal{F}}_{\mathrm{PRUNE}}$ for collision resistance, with $\mathcal{A}$’s success probability $\le 2^{-128}$.

Appendix A.5.4. Proof of Proposition 4

Algorithm 1 ratifies that $\mathcal{A}$ cannot substitute fake transactions to violate integrity with non-negligible probability.

• Ideal World: ${\mathcal{F}}_{\mathrm{PRUNE}}$ verifies transactions via WOTS signatures ($si{g}_i$) and Merkle roots. If $\mathcal{A}$ submits $(t{x}^{\prime },si{g}^{\prime })$, it’s rejected unless $si{g}^{\prime }$ verifies for $t{x}^{\prime }$ and matches $\mathcal{L}$’s Merkle root. WOTS ensures $\mathrm{Pr}\left[\mathrm{forgery}\right]\le 2^{-256}$.
• Real World: Algorithm 1 uses WOTS signatures and Merkle roots, with constraint $C_1$ (${\displaystyle \frac{2^{256}}{B_{\mathrm{sig}}}}$) ensuring enough transactions remain to verify integrity. Pruning only occurs for settled bills ($C_2$), preserving valid data.
• Simulator $\mathcal{S}$: $\mathcal{S}$ simulates $\mathcal{A}$’s view, generating valid $(tx,sig,\mathrm{merkle})$. If $\mathcal{A}$ submits $(t{x}^{\prime },si{g}^{\prime })$, $\mathcal{S}$ checks $si{g}^{\prime }$ via WOTS and forwards to ${\mathcal{F}}_{\mathrm{PRUNE}}$, which rejects invalid signatures. Forgery probability is $2^{-256}$.
• Indistinguishability: $\mathcal{Z}$ sees identical ledger updates, with both worlds rejecting fake transactions. Algorithm 1’s constraints ensure that signature verification data persists.
• Algorithm 1 UC-realizes ${\mathcal{F}}_{\mathrm{PRUNE}}$ for integrity, with $\mathcal{A}$’s success probability $\le 2^{-256}$.