Fault Recovery Methods for a Converged System Comprised of Power Grids, Transportation Networks and Information Networks

1. Introduction

In recent years, the TNCS has emerged due to the rapid development and deep integration of power grids, transportation networks, and information networks, caused by the widespread adoption of electric vehicles. This convergence system enables efficient coordination among energy flow, traffic flow, and information flow [1,2]. However, the increasing complexity and interdependence of the system has also created favorable conditions for fault propagation, in which local disturbances can easily initiate cascading failures across networks, resulting in severe system risks such as wide-scale power outages [3,4,5,6,7]. Therefore, research on fault recovery method for the TNCS is essential.

The development of a recovery method for the TNCS necessitates a clear understanding of its interaction mechanisms. However, there is a paucity of explicit domestic and international research on modelling the interaction mechanisms of such systems, which adds to the complexity of the study. Furthermore, there is a paucity of research on fault recovery method related to the TNCS. Despite this, a body of research exists on fault recovery for power grids and power cyber-physical systems. In reference [8], for instance, cascading fault recovery model based on the SRG was suggested for power transmission networks and proven by simulations. In reference [9], an optimization model for the entire black start restoration process in gearbox systems was formulated and solved using a linearized hybrid integer linear programming model and the L-shaped algorithm, taking into consideration the uncertainty of wind power output. Reference [10] analyzed the potential delays caused by information system failures on power system generation, transmission line operation, and load restoration. In these studies, however, the impact of generator start-up failures on restoration decisions is not considered. Reference [11] proposed a parallel recovery method for power systems. Although this method considered uncertainties during the restoration process, it did not consider the possibility of new faults occurring in power systems during component restoration.

The recovery of power system components can result in alterations to the power system's topology, power flow, and power levels, which may cause problems such as line overloads, imbalances in energy supply and demand in certain regions, and system instability. However, existing research lacks studies on the risks of new faults or system instability that may arise from component recovery. For instance, in reference [12], a two-stage recovery strategy was proposed for power facility restoration after hurricanes, focusing on the design of distributed generation and maintenance personnel scheduling, but excluding the analysis of new faults that might occur from power system component recovery. References [13,14,15] also ignored the occurrence of new faults.

It is evident that existing research typically establishes optimization models for different fault scenarios and solves them using various optimization algorithms, but lacks consideration of uncertainties that may occur during the actual restoration process, such as generator start-up failures and faults triggered by recovery actions, and their impact on recovery decisions. Consequently, these uncertainties should be considered, when studying fault recovery methods for TNCS. Such research methods would be more applicable to real-world circumstances.

Moreover, due to the high complexity and interactivity of the TNCS, network attacks can exploit intricate network interaction mechanisms and exacerbate the hazards of faults with relative ease. By manipulating power grid data and injecting false information, for instance, the assessment of the system's operational status may deviate, resulting in erroneous decisions by administrators and potentially triggering large-scale cascading failures [16,17,18,19]. By injecting a large number of useless requests and obstructing communication channels, DoS attacks can cause an increase in communication delay and thus initiate or exacerbate power outages [20,21,22,23,24]. An example of this is the DoS attack launched by a hacker group against a power company in the western United States in 2019, which resulted in communication disruption between the control center and various site devices [25]. There is limited research on restoration decisions for power cyber-physical systems that take network attacks into account at present. While the optimization strategy proposed in [26] considered the impact of information system failures caused by DoS attacks on the restoration process, but the solving algorithm lacks flexibility to handle uncertainty factors and becomes less efficient as the system scale increases.

The TD3 algorithm from the discipline of deep reinforcement learning can be used to address this issue [27]. By designing the state space and reward function appropriately, uncertainties can be accounted for, and the neural network can effectively address the curse of dimensionality problem caused by the large scale of the TNCS [28,29]. In addition, compared to DQN, AC, and DDPG algorithms [30,31,32], the TD3 algorithm exhibits better capability to suppress network overestimation and provides more stable network training. Consequently, the utilization of the TD3 algorithm is more suitable for solving the model. However, the optimization objective of the TD3 algorithm is to find actions that maximize the action-value function in different states, without considering the specific role and impact of recovery actions in particular scenarios, such as system security. Therefore, improvements to the TD3 algorithm are necessary to ensure system security.

Based on the above analysis, this study makes the following main contributions to the research on fault recovery methods in TNCS:

• Focusing on the charging behavior of electric vehicles, a TNCS model is established to reveal the underlying interaction mechanisms.
• An efficient fault recovery method for TNCS is proposed, incorporating an improved TD3 algorithm and considering communication delays. By designing and improving the TD3 algorithm, the uncertainties and security issues in the restoration process are considered, leading to the design of an effective recovery algorithm. In addition, the resilience of the algorithm is evaluated by introducing DoS attacks in the context of power grid faults. Lastly, the efficacy of the proposed recovery method is demonstrated through simulation experiments.

2. Methods

2.1. Modeling

2.1.1. Overall Framework

As shown in Figure 1, the TNCS in this study consists of three layers: the execution layer, the coupling layer, and the control layer. The execution layer consists of the actual power grid and transportation network, responsible for the reliable operation of energy interactions. The coupling layer mainly consists of power collection and control devices, as well as communication channels, responsible for information acquisition, execution of control commands, and information data transmission. The control layer, primarily based on the information network, is responsible for state monitoring and dispatch control.

2.1.2. Execution Layer

The power grid topology is a directed graph $G_P=\left(V_P,E_P\right)$, where $V_P$ represents power nodes including generators, loads, and circuit breakers, and $E_P$ represents transmission lines. The topological information of the power grid is represented by a matrix $T^P$ whose elements $T_{ij}^P$ adhere to the following rule: $T_{ij}^P=1$ denotes the presence of a line between nodes i and j, while $T_{ij}^P=0$ denotes the absence of a line. The nodal admittance matrix $B$, nodal injection power vector $P$, line power flow vector $f$, and nodal phase angle vector $\theta$ are established. Based on the DC power flow model, the power flow information matrix $PF$ of the power grid can be obtained from the following equations.

$$PF=diag\left(P\right)+f$$

(1)

$$B\theta =P$$

(2)

$$\left(\theta Q-Q^T{\theta }^T\right)\circ B=f$$

(3)

where the value of $Q$ is $\left[1,1,1,\dots ,1\right]$, and $diag\left(P\right)$ represents a diagonal matrix with vector $P$ as its diagonal elements.

The road topology of the transportation network is an undirected graph $G_T=\left(V_T,E_T\right)$, where $V_T$ represents road nodes (including charging stations) and $E_T$ represents paths between them. A matrix $A_T$ is established to represent the topological information of the transportation network, with matrix elements $A_{ij}^T$ following the following rule: $A_{ij}^T=1$ indicates the presence of a connected road between road nodes i and j, while $A_{ij}^T=0$ indicates the absence of a connected road. A decision vector $SE$ is used to assess whether each road node has a charging station, where $S{E}_i=1$ indicates the presence of a charging station in road node i and $S{E}_i=0$ indicates its absence. A charging station load vector $P^c$ is introduced for the transportation network, where $P_i^c$ represents the load at the charging station located on road node i and can be obtained from the following equations.

$$P_i^{c\text{'}}={\displaystyle \sum }_{j=1}^{N_i^S}CS{O}_{ij}\circ CS{W}_{ij}\circ CS{P}_{ij}$$

(4)

$$P_i^c=S{E}_i\circ P_i^{c\text{'}}$$

(5)

where $N_i^S$ represents the number of charging piles in the charging station located at road node i. $CS{O}_{ij}$ indicates the operational status of the j-th charging pile in the charging station at road node i, with a value of 1 indicating that it is operational and 0 indicating that it is not. $CS{W}_{ij}$ represents the operational status of the j-th charging pile in the charging station at road node i, with 1 indicating operation and 0 indicating inactivity. $CS{P}_{ij}$ represents the charging power of the j-th charging pile at the road node i charging station.

To assure the safety of charging services, the maximum charging power for each charging pile must be specified. Establishing a maximum power vector $CSM$, where $CS{M}_{ij}$ represents the maximum charging power of the j-th charging pile in the charging station at road node i.

2.1.3. Coupling Layer

The matrices $S^P$ and $S^T$ are respectively defined to represent sensor deployment in the power grid and transportation network. In $S^P$, diagonal elements denote sensors on the nodes of the power grid, while off-diagonal elements represent sensors on the power lines. In $S^T$, the diagonal elements represent sensors on the road node charging stations. The elements of both matrices are assigned the value 1 to denote the presence of a sensor and the value 0 to denote the absence of a sensor. The actuator matrices $A^P$ and $A^T$ are defined in the same way as the sensor matrices.

The uplink data communication channel matrix contains $U{C}^P$ for the power grid and $U{C}^T$ for the transportation network. The matrix elements, $U{C}_{ij}$, satisfy $U{C}_{ij}\in \left\{0,1\right\}$. In $U{C}^P$, $U{C}_{ii}^P=1$ indicates the presence of an uplink data communication channel between the sensor on the power grid node and the information network control center, transmitting node power information. $U{C}_{ij}^P=1$ indicates the existence of an uplink data communication channel between sensors on power grid lines i-j and the information network control center, transmitting current flow information and circuit breaker status. In $U{C}^T$, $U{C}_i^T=1$ denotes the presence of an uplink data communication channel between the sensors in the charging station at road node i and the information network control center, transmitting charging station operational information. The downlink data communication channel matrix is defined in the same way as $U{C}^P$ and $U{C}^T$, with different transmitted content. It carries commands for the charging station's circuit breaker open or close, node power adjustment, charging pile open or close, and maximal charging power adjustment.

2.1.4. Control Layer

The information network topology is defined as a directed graph $G_C=\left(V_C,E_C\right)$, where $V_C$ represents the information nodes and $E_C$ represents the communication links between nodes. The connectivity status of the communication links is represented by the link status matrix $C$, with matrix elements $C_{ij}\left(i\ne j\right)=\left\{0,1\right\}$. A value of $C_{ij}=1$ indicates that an effective connection has been established between information nodes i and j. The control center in the information network adjusts the operational status of the power grid and transportation network based on the received data. The storage of received information includes $P{M}^f$ and $P{M}^t$ for power flow information ($PF$) and power grid topology information ($T^P$), respectively, while $T{M}^S$ represents the storage of received charging station information. Utilizing this information, the reduction in output of generator nodes and the reduction in load of load nodes are calculated, thereby calculating the decrease in power at the charging stations in the transportation network, as illustrated below.

$${\displaystyle \sum }_i^N{P}_i^a{\displaystyle \sum }_{j=1}^{N_i^S}CS{O}_{ij}^{\text{'}}={\displaystyle \sum }_{i=1}^N{P}_i^C-P_i^S$$

(6)

$$\left|P^C-P^S\right|\le \left|LS\right|$$

(7)

where $LS$ satisfies equations (1) to (3), it is used to represent the reduction in output of power nodes or the reduction in load of load nodes. $P^S$ is the drop value in charging power at road node i, while $P_i^a$ is the adjusted average charging power of charging piles in charging stations. $CS{O}_{ij}^{\text{'}}$ refers to the adjusted operational state of the j-th charging pile in the i-th road node. N represents the number of road nodes in the transportation network. We assume that the charging stations will bear the maximum extent of power grid load variations.

The control center then generates and distributes decision instructions to the executive layer.

2.2. Fault Recovery Method

2.2.1. Design of Restoration Model

In this section, we define importance indicators for power grid faults in the context of the TNCS, assessing the importance of fault power lines. With the objective of maximizing restoration benefits, a restoration model is established.

We assume that power grid faults in the TNCS result from the overload-induced disconnection of power lines, leading to redistribution of power flow, reduction in output of generator nodes, and load shedding at load nodes. Concurrently, adjustments are made to the charging power of charging stations in the transportation network corresponding to the affected power nodes, which may impact the travel of EV users. The importance of power lines, $\eta$, is determined by the importance of its connected power nodes (${\eta }_P$) and the importance of the charging stations on the corresponding coupled road nodes (${\eta }_S$), as shown below.

$$\eta ={\lambda }_1\cdot \omega {\eta }_P+{\lambda }_2\cdot \left(1-\omega \right){\eta }_S,\eta >0,{\eta }_P>0,{\eta }_S>0,0<\omega <1$$

(8)

where $\omega$ represents the weight coefficient. ${\lambda }_1$ and ${\lambda }_2$ are binary variables that indicate the existence of the respective item, taking a value of 1 if it exists and 0 otherwise. ${\eta }_P$ consists of the degree, generation capacity, and load of the power nodes, and ${\eta }_S$ consists of the degree of the road nodes and the number of electric vehicles affected, as shown below.

$${\eta }_P={\lambda }_3\frac{P_g}{{{\displaystyle \sum }}_{i\in gen}{P}_i^g}+{\lambda }_4\frac{{\gamma }_d}{{{\displaystyle \sum }}_{i\in \left|V_P\right|}{\gamma }_i}+{\lambda }_5\frac{P_l}{{{\displaystyle \sum }}_{i\in load}{P}_i^l}$$

(9)

$${\eta }_S={\lambda }_6\frac{{\gamma }_t}{{{\displaystyle \sum }}_{i\in \left|V_T\right|}{\gamma }_i}+{\lambda }_7\frac{{\chi }_t}{{{\displaystyle \sum }}_{i\in \left|V_T\right|}{\chi }_i^t}$$

(10)

$${\chi }_t={\chi }_{t1}+{\chi }_{t2}$$

(11)

$${\chi }_{t1}={\chi }_{t1}+1,{\displaystyle \sum }_{i=1}^K{\displaystyle \sum }_{j=1}^{\left|V_T\right|}{\displaystyle \sum }_{k=1}^{N^S}\begin{array}{c}IF(\left|T{S}_i^p\left(j,k\right)\right|>0\\ AND CS{N}_i\left(j,k\right)\ne 0,1,0)\end{array}$$

(12)

$${\chi }_{t2}={\chi }_{t2}+1,{\displaystyle \sum }_{i=1}^K{\displaystyle \sum }_{j=1}^{\left|V_T\right|}{\displaystyle \sum }_{k=1}^{N^S}\begin{array}{c}IF(\left|T{S}_i^p\left(j,k\right)\right|>0\\ AND CS{Q}_i\left(j,k\right)\ne 0,1,0)\end{array}$$

(13)

where ${\lambda }_3$, ${\lambda }_4$, ${\lambda }_5$, ${\lambda }_6$, and ${\lambda }_7$ are binary variables indicating the existence of the respective items. $P_g$ represents the power output of generator nodes, and ${{\displaystyle \sum }}_{i\in gen}{P}_i^g$ represents the total power output of generator nodes. ${\gamma }_d$ represents the degree of nodes in the power grid, while ${{\displaystyle \sum }}_{i\in \left|V_P\right|}{\gamma }_i$ represents the sum of degree of power nodes. ${\gamma }_t$ represents the degree of road nodes in the transportation network, while ${{\displaystyle \sum }}_{i\in \left|V_T\right|}{\gamma }_i^t$ is the sum of degree of road nodes. $P_l$ represents the power load of load nodes in the power grid, while ${{\displaystyle \sum }}_{i\in load}^m{P}_i^l$ denotes the total power load of load nodes in the power grid. ${\chi }_t$ obtained by equations (11), (12), and (13) represents the number of EVs impacted during travel, while ${{\displaystyle \sum }}_{i\in \left|V_T\right|}{\chi }_i^t$ represents the total number of EVs impacted during travel. Equation (12) calculates the adjusted number of charging station users ${\chi }_{t1}$, while equation (13) calculates the adjusted number of charging station electric vehicles ${\chi }_{t2}$. $T{S}_i^p$ represents the power adjustment matrix for the i-th charging station, $CS{N}_i$ represents the identifier of the current charging station users, and $CS{Q}_i$ represents the identifier of the current charging station reservation users.

After calculating the importance of each power line, they are sorted in descending order to determine the restoration priority. $\Gamma$ is defined as the quantified priority value, and $e_p$ represents the number of restoration steps. The optimization objective is to maximize the restoration benefit which consists of the recovery power from the power grid and transportation network, based on the importance of power lines.

$$\mathit{max}{\displaystyle \sum }_m^{e_p}\Gamma \left(L{S}_{rm}+P_{rm}^S\right)$$

(14)

$$s.t.\Gamma =1-0.02\left(e_p-1\right),e_p\ge 1$$

(15)

$$\mathit{min}\left(LS\right)\le L{S}_{rm}\le \mathit{max}\left(LS\right)$$

(16)

$$\mathit{min}\left(P^S\right)\le P_{rm}^S\le \mathit{max}\left(P^S\right)$$

(17)

$$P_{min}^G\le P^G\le P_{max}^G$$

(18)

$$V_{min}\le V\le V_{max}$$

(19)

$$f_{min}<f<f_{max}$$

(20)

$$0<T^{up}<T^h$$

(21)

$${\displaystyle \sum }_{i\in gen}{P}_i^g={\displaystyle \sum }_{i\in load}{P}_i^l$$

(22)

where $\Gamma \left(L{S}_{rm}+P_{rm}^S\right)$ represents the restoration benefit obtained from restoring the fault at step m. $P^G$ is the active power of the generator, and $V$ is the voltage at the terminals of the power system. Equation (20) represents the line flow constraint, where power flow varies with the restoration of the faults. Exceeding the limits of line flow, $f_{min}$ or $f_{max}$, may result in overload issues, leading to the disconnection or damage of the concerned line or other lines. In Equation (21), $T^{up}$ represents the unit start-up time, while $T^h$ represents the maximum start-up time of the unit, ensuring the required time for thermal start-up. Equation (22) represents the energy supply-demand balance constraint in the power grid, ensuring the balance between actual supply and demand. Imbalances can lead to voltage fluctuations, frequency deviations, and overloaded operation, causing power equipment failures or system security issues such as power grid collapse. In this study, we assume that energy imbalance can cause line overloads and disconnection so that fault recovery should avoid energy supply-demand imbalances.

2.2.2. Design of Restoration Algorithm

In the context of fault recovery in the power grid of TNCS, the restoration sequence of each faulty line significantly influences the restoration benefit, giving rise to a decision-making problem. In this section, we focus on the design and improvement of the TD3 algorithm in the field of deep reinforcement learning with the aim of determining the optimal restoration strategy. Figure 2 illustrates the restoration algorithm structure based on the TNCS.

Specifically, we consider the control center as the agent and define the action space as the set of all available restoration actions from which the control center can choose in the TNCS environment, denoted as $Ac$, and action $A$ belongs to $Ac$. In the restoration model, the recovery of faults can cause new faults. To address this uncertainty, we employ a state matrix $S$ to represent the state space, reflecting the changes in the state of the TNCS. As shown in Figure 3, each row of the state matrix represents a power node or line, and each column represents a fault type. The arrangement follows the order of power node and line numbering, as well as fault type. Based on the rows and columns, the location and type of faults can be determined. An element value of 1 indicates no fault, while 0 indicates that a fault exists. Consequently, the changes in the state of the TNCS caused by fault recovery and occurrence can be reflected through the variation of element values.

The design of the reward function is as follows.

$$R=\left\{\begin{array}{c}{I}_d+I_b+I_m,\left(\mu =1,\epsilon =1\right)\\ I_d+I_c,\left(\mu =0\right)\\ I_d-1,\left(\mu =1,\epsilon =0\right)\end{array}\right.$$

(23)

$$I_m=\frac{L{S}_{rm}+P_{rm}^S}{\mathit{max}\left(LS\right)+\mathit{max}\left(P^S\right)}$$

(24)

$$I_c=\left(1+\beta \frac{h_t}{H}\right)\Gamma +\alpha \frac{e_c}{E}$$

(25)

The control center conducts a status check on the TNCS at the beginning of each restoration step, and when the check is complete, the reward $I_d$ is obtained. If no data information is received from a node or line in the execution layer within time $T_l$, it is considered communication delays, $\mu =0$ is set, and the reward $I_c$ is obtained. Equation (25) represents the impact of communication delays on restoration decisions, where $\beta$ and $\alpha$ are proportional coefficients with respective ranges of [0,1] and (0,1). $H$ represents the total number of restoration stages, indicating the duration of the entire restoration process, while $h_t$ represents the restoration stage at which the communication delay occurs. The restoration stage $E$ represents the maximum number of restoration steps required to address communication faults (with a single stage consisting of multiple consecutive restoration steps), and $e_c$ represents the actual number of restoration steps required for communication fault recovery. $\mu =1$ denotes normal communication, with $\epsilon =1$ indicating the effectiveness of the restoration action, resulting in the reward $I_b$, and the restoration benefit $I_m$ calculated from equation (24). When $\epsilon$ equals zero, the restoration action is ineffective, resulting in a negative reward of -1.

The effectiveness of a restoration action refers to its objective of targeting the faulty lines without causing any additional faults. Restoring a power node requires at least one connected power line to be operational (excluding black-start nodes). For a power generation node, equation (21) must be satisfied, and if not, it is considered a failed unit start-up, which is uncertain and may be caused by unpredictable factors. In the case of a failed unit start-up, the reward function returns -1, reflecting the impact of this uncertainty on restoration decisions, and we can enhance the practicality of our method by employing this approach.

In response to potential system security issues caused by an imbalance between energy supply and demand during the restoration process, the TD3 algorithm is improved as follows.

$$\rho ={\lambda }_8\eta$$

(26)

$${\left.q=Q_{{\theta }_1}\left(S,A\right)\right|}_{A={\pi }_{\varphi }\left(S\right)}·\left(1-{\rho }_a\right)$$

(27)

Equation (27) adds a multiplicative term, $\left(1-{\rho }_a\right)$ in the original estimated value, $Q_{{\theta }_1}\left(S,A\right){|}_{A={\pi }_{\varnothing }\left(S\right)}$used for updating the Actor network, where $q$ represents the improved estimated value and ${\rho }_a$ belongs to the set $\rho$. $\rho$ represents the set of line importance, as determined by equation (26). ${\lambda }_8$ is a binary variable with values of 0 and 1, and it is set to 1 if the restoration action causes an energy supply-demand imbalance in the system, and 0 otherwise.

The addition of the multiplicative term can affect the original estimated value after the improvement. When high-priority lines are disconnected due to an energy supply-demand imbalance, the multiplicative term becomes smaller, resulting in a decrease in estimated value, and indicating that the restoration actions under the current state pose a security risk to the system. Therefore, the Actor network can conclude that the chosen restoration action is not optimal.

Lastly, the neural networks used in the algorithm consist of an input layer, fully connected hidden layers, and an output layer. The specific parameter setting depends on the simulation experimental scenarios.

3. Experiments and Results

3.1. Experimental Settings

Based on the TNCS model, a simulation experimental scenario is constructed as follows.

The power grid uses the IEEE 30-bus power system [33], as shown in Figure 4. The transportation network has 5000 EVs and 30 nodes with 50 charging piles at each road node. Each charging pile has a maximal output of 20 kW. The nodes of the transportation network and the power grid with the same number are coupled. The information network consists of 101 information nodes, where the first 30 nodes correspond to the power grid nodes, with node 10 serving as the control center. Nodes 31 to 71 correspond to the power lines, while the remaining nodes correspond to the transportation network nodes. The simulation experiments are implemented using MATLAB programming without GPU acceleration techniques.

3.2. Small-scale Power Grid Faults

In this section, we simulated small-scale faults in the power grid of the TNCS. Specifically, we induce faults in the power lines with the line numbers 31, 32, 33, 34, 36, and 37, and these lines are placed in an open-circuit state. The specific experimental parameter settings are as follows.

The Actor network consists of an input layer with 142 neurons, a first hidden layer with 256 neurons, a second hidden layer with 128 neurons, and an output layer with 6 neurons. The target Actor network is updated once after every five updates of the Actor network.

The Critic network consists of an input layer with 142 neurons, a first hidden layer with 256 neurons, a second hidden layer with 256 neurons, and an output layer with 1 neurons. The target Critic network is updated once after every five updates of the Critic network.

The ReLU function is used as the activation function in the hidden layers of all networks. The Actor network is updated once after every three updates of the Critic network. Set the discount factor to 0.99, the exploration noise standard deviation to 0.15, the policy noise standard deviation to 0.3, the batch size to 64, the total number of episodes to 1000, and the $\omega$, $I_d$, $I_b$, and $\tau$ values to 0.5, 0.1, 0.1, and 0.005, respectively. The parameter $\tau$, specifically, is used in the soft update.

The simulation experimental results are as follows.

The result in Figure 5 indicates that the Actor network is well-trained and able to make accurate and effective decisions.

The obtained restoration scheme is shown in Table 1. The restoration sequence is primarily determined by the order of line restoration; once a line is restored, the connected nodes automatically begin their restoration process (except for black-start nodes). The restoration sequence refers specifically to the order of initiating restoration. As a result, the restoration processes for multiple faults can proceed simultaneously. For example, the restoration of line 36 can be initiated during the restoration of node 1.

To validate the optimality of the restoration sequence, we compared it with several backup schemes based on the restoration benefit criterion. As shown in Table 2, the restoration benefit obtained by our proposed scheme, denoted $F_1$, is 15.36% to 25.52% greater than that of the other schemes, indicating that our scheme is optimal in terms of restoration benefit.

Additionally, Figure 6 shows the improvement effect of the TD3 algorithm.

It can be observed that the occurrence number of ${\lambda }_8=1$ is lower in the improved algorithm compared to the unimproved version. This indicates that the evaluation of restoration actions in the improved algorithm is not solely based on maximizing $Q\left(S,A\right)$, but also considers system security issues caused by energy supply-demand imbalances, as represented by the inclusion of a multiplicative term. Using this method, it is possible to lower the original estimated values $Q_{{\theta }_1}\left(S,A\right){|}_{A={\pi }_{\varnothing }\left(S\right)}$, as shown by the lower values of the line in the figure for the improved TD3 algorithm compared to the unimproved TD3 algorithm. When the chosen restoration action leads to an energy supply-demand imbalance, ${\lambda }_8$ equals 1, the estimated value decreases, and the Actor network receives feedback indicating that the chosen restoration action is not optimal. As the number of episodes increases, the Actor network gradually learns to avoid restoration actions that may cause energy supply-demand imbalances. Consequently, in the figure, we can observe that the frequency of ${\lambda }_8=1$ decreases over time, with only three occurrences between episodes 800 and 1000, representing a probability of 1.5%. This is a 66.67% reduction compared to the unimproved TD3 algorithm, and demonstrates the effectiveness of our improvements to the TD3 algorithm in reducing the probability of system security issues caused by energy supply-demand imbalances during the restoration process.

3.3. Large-scale Power Grid Faults

We set all power lines to an open-circuit state, paralyzing the power grid in TNCS. In this scenario, the impact of uncertainties such as failed unit start-up and the occurrence of new faults caused by the restoration process is analyzed, and algorithm proposed in this study is compared with other algorithms to evaluate the performance and efficacy.

Set the batch size to 128 and the total number of episodes to 5000, while the settings of the remaining experimental parameters are identical to those in the small-scale fault scenario. The simulation results are shown below.

We can observe that curve 3 gradually converges to a stable value in Figure 7, which indicates that the Actor network is well-trained and able to make accurate and effective decisions in the scenario of large-scale power grid faults. Furthermore, curve 3 exhibits the lowest average reward, and both curve 3 and curve 1 show significantly higher fluctuations compared to curve 2. This is due to the fact that both generator start-up failures and new faults occurring during the restoration process result in a reward value of -1. Consequently, curve 3 exhibits lower values than the other two curves. The presence of new faults increases the uncertain change of $S$ and the difficulty of convergence, leading to higher fluctuations in curve 3 and curve 1.

To validate the impact of uncertainties on restoration sequence, we take the restoration sequence of power line 42 as an example to observe the change of its restoration sequence, and the results are shown in Figure 8. We can observe that in the case corresponding to curve 3, the restoration sequence of power line 42 in 10 experiments is approximately the 17th step, which differs from the cases corresponding to curves 1 and 2. This indicates that uncertainties during the restoration process can alter the restoration sequence, and should not be ignored in practical restoration process.

At the end of this section, we also compare the proposed algorithm with other intelligent algorithms to verify the superiority of the proposed algorithm in solving the fault recovery problem of the TNCS. The results are shown in Table 3. We can observe that Ours outperforms other algorithms by 0.3% to 20.96% in terms of restoration benefit, and in terms of convergence time, there exists a reduction of 2.82% to 14.39% compared to other algorithms except for the PSO algorithm, with only a marginal increase of 1.2% compared to the PSO algorithm. The aforementioned comparative results demonstrate that ours has distinct advantages in terms of both restoration benefit and convergence time, indicating its superiority in an overall assessment. Moreover, compared to the TD3 algorithm, although the restoration benefit is very close, ours exhibits an 8.93% reduction in time. This is due to improvement made to the TD3 algorithm, which reduces the occurrence of new faults during the restoration process and effectively reduce the uncertain change of $S$, thereby reducing convergence difficulties during training.

3.4. Communication Faults

In this section, we add the simulation of communication faults on the basis of large-scale power grid faults to validate the resilience of algorithm proposed in this study.

Specifically, we assume that DoS attackers send a significant volume of disguised packets to information nodes 31, 32, 33, and 34, resulting in their infection. As a result, the communication channels within the coupling layer connected to the information nodes and the communication links within the information network of the control layer become blocked. When communication delays are detected, the technical staff at the control center suspends the restoration of power grid faults and initiate the deployment of firewalls and intrusion detection systems to restore communication faults. Equation (25) demonstrates that the impact of communication delays on restoration benefit depends on its occurrence timing and restoration speed. To facilitate statistical analysis, the occurrence time of communication delays is divided into six stages: stage 1: 1 to 6 steps, stage 2: 7 to 12 steps, stage 3: 13 to 18 steps, stage 4: 19 to 24 steps, stage 5: 25 to 30 steps, and stage 6: More than 30 steps. We set $H=6$, with $h_t=1$ corresponding to the first stage, $h_t=2$ corresponding to the second stage, etc. There are five levels of restoration speed: level 1: $e_c=1$, level 2: $e_c=2$, level 3: $e_c=3$, level 4: $e_c=4$, and level 5: $e_c=5$. Among them, $e_c=1$ indicates that the restoration of communication delay requires one step, $e_c=2$ requires two steps, etc. We set $E=5$ and $\alpha =0.1$. The remaining experimental parameters are maintained in accordance with the previous section. The simulation results are shown as follows.

As shown in Figure 9, the restoration benefit of the communication delay curves is less than that of the normal communication curve, which is due to communication delay causes a reduction in $\Gamma$ value of certain power lines, illustrated in Figure 10.

Figure 9 also illustrates two additional aspects: first is the impact of the occurrence time of DoS attacks on restoration and second is the impact of restoration speed of communication delay on restoration.

The timing of DoS attacks is characterized by uncertainty, while the reduction in restoration benefit caused by DoS attacks can be mitigated by adjusting the value of $\beta$. According to Table 4, when communication delay occurs in stage 1, stage 2, and stage 3, setting $\beta$ to its maximum value of 1 minimizes the reduction in restoration benefit by 9.23%, 4.22 %, and 0.68%, respectively, compared to the normal communication benefit of 12.3968 MW. However, in stage 4, setting $\beta$ to 1 leads to a restoration benefit of 12.510 MW, which is higher than the normal benefit and unrealistic. Therefore, $\beta$ cannot be set to 1, and $\beta =0.75$ results in a restoration benefit of 12,267 MW, which is marginally less than 12,3968 MW. Thus, the valid range for $\beta$ in stage 4 is $0.75\le \beta <1$. Similarly, in stage 5 and stage 6, the valid range for $\beta$ is $0.75\le \beta <1$ and $0.5\le \beta <0.75$, respectively.

For the second aspect, it can be observed that the restoration benefit of the communication delay curve corresponding to $e_c=1$ is greater than that of the curves corresponding to other restoration speed in all restoration stages. This is due to restoring communication delay leads to obtain the reward $I_c$. If the restoration speed is fast, such as completing the restoration in a single step, one $I_c$ is obtained. Alternatively, multiple $I_c$ can be obtained if the restoration speed is slow and requires multiple stages to complete. However, $I_c$ and $I_b+I_m$ are not equivalent, and a difference exists between them. Obtaining multiple $I_c$ can lead to overestimation or underestimation of $Q\left(S,A\right)$, which may result in policy bias and affect restoration decision-making. To solve this issue, we can adjust the values of $\beta$ and $\alpha$ to reduce the difference between $I_c$ and $I_b+I_m$. It is important to note that we cannot directly control the restoration speed due to its uncertainty and dependence on the defensive deployment level of the information network. When the occurrence time of DoS attacks is fixed, we can adjust the value of $\alpha$ to mitigate the policy bias caused by restoration speed. Figure 11 illustrates the impact of different values of $\alpha$ on policy bias when $\beta =0.5$.

It can be observed from the Figure 11 that as the value of $\alpha$ reduces, the occurrence number of $I_c$ reduces regardless of the restoration speed. First, this indicates that when $\beta =0.5$, the value of $I_c$ is greater than $I_b+I_m$, resulting in a $Q\left(S,A\right)$ overestimation. To solve this, the value of $\beta$ can be appropriately reduced with considering the occurrence time of DoS attacks. Second, the minimum occurrence number of $I_c$ reduces by 60.26% to 80.12% compared to the maximum occurrence number for the five restoration speed by adjusting the value of $\alpha$, demonstrating that adjusting the value of $\alpha$ effectively resolves the policy bias issue caused by restoration speed.

The average occurrence number of $I_c$ for the five restoration speed under each $\alpha$ value in Figure 11 is calculated to represent the actual policy bias under normal restoration speed. The results are shown in Table 5, where the difference ratio indicates the percentage reduction in the average occurrence number of $I_c$ for the next $\alpha$ value compared to the current $\alpha$ value. For instance, when the value of $\alpha$ changes from 1 to 0.9, the average occurrence number of $I_c$ reduces by 11.29%, resulting in a difference ratio of 11.29%. We can observe that as the value of $\alpha$ reduces from 1, the difference ratio increases continuously. When the value of $\alpha$ changes from 0.4 to 0.3, the difference ratio reaches its maximum value of 24.38%, and as the value of $\alpha$ continues to reduce, the difference ratio also reduces. This indicates that under the condition of $\beta =0.5$, the optimal range of $\alpha$ value for minimizing the difference between $I_c$ and $I_b+I_m$ is around 0.3, leading to a more effective improvement in solving the problem of policy bias.

Based on the aforementioned analysis, it is evident that for uncertainties that cannot be directly controlled, such as the occurrence time of DoS attack and the restoration speed of communication delay，we can adjust the value of $\beta$ to mitigate the extent of restoration loss under various occurrence time of DoS attack. Additionally, we can improve the issue of policy bias caused by restoration speed by adjusting the value of $\alpha$. The experimental results demonstrate that the proposed algorithm exhibits strong adaptability and resilience in the presence of communication delay caused by DoS attack.

4. Conclusions

In addressing the issue of fault recovery in TNCS, we propose a recovery method based on improved TD3 algorithm. In the design of this method, we have particularly considered the uncertainties, system security issues and communication faults. Experimental results indicate that the proposed method can obtain the optimal recovery decisions and accomplish the maximum restoration benefit when faults occur in the power grid of the TNCS. Furthermore, the improvement to the TD3 algorithm effectively reduces the occurrence of energy supply-demand imbalance and the uncertainties are able to impact the restoration sequence so that they should not be ignored during actual restoration process. The results also show that communication delay caused by DoS attacks can reduce the restoration benefit and lead to the policy bias. By adjusting the values of $\beta$ and $\alpha$, the proposed algorithm can effectively mitigate the extent of restoration loss under various occurrence time of DoS attack and improve the issue of policy bias caused by restoration speed, thereby demonstrating strong resilience. Lastly, regarding future research directions, several avenues can be explored, such as studying the impact of transportation network faults on restoration process, designing recovery method when confronted with various types of network attacks, and employing other intelligent algorithms to solve the fault recovery problem in TNCS.