Hybrid SAC for Corrective Congestion Management in AC/DC Transmission Networks

1. Introduction

1.1. Computational Hardness of the AC-OPF

The AC optimal power flow (AC-OPF) is a continuous non-convex nonlinear program whose feasible set is generally non-connected because of voltage magnitude and angle constraints [1,2]. Classical interior-point solvers [3] reach local optima in seconds for medium-scale systems, but global optimality guarantees require semidefinite relaxations [4] or branch-and-bound enumeration, both computationally prohibitive in real time. The security-constrained AC-OPF adds N-1 contingency constraints that multiply the problem size by the number of contingency cases [5,6,7,8], which places the problem beyond the redispatch cycle even for medium-sized systems. The ARPA-E Grid Optimization competitions [9,10,11] have benchmarked the state of the art and show that a substantial gap remains between solver runtime and real-time requirements.

1.2. The Hybrid Action Space

Corrective congestion management in a real transmission network involves both continuous and discrete decisions. On the continuous side, generators adjust active output within ramp limits [12] and HVDC links modulate the scheduled power transfer [13]. On the discrete side, network topology is modified by switching lines, transformers, or busbar sections [14,15]. The two interact strongly through the power flow equations: opening a line changes the effective distribution factors [16], which in turn changes the optimal continuous redispatch.

Handling hybrid action spaces in deep reinforcement learning is not straightforward. DQN [17] and its extensions handle only discrete actions; on-policy methods such as PPO [18] can in principle handle both but sacrifice sample efficiency; and standard SAC [19,20] and TD3 [21] handle only continuous actions. Parametrised DQN [22] extends DQN to hybrid spaces by maintaining a separate continuous parameter network, at the cost of a Q-network that must enumerate every discrete choice. The Gumbel-Softmax distribution [23,24] offers a different route: a differentiable continuous relaxation of categorical variables that permits gradient-based optimisation of the discrete decision while preserving the actor-critic structure of SAC. That is the route taken here.

1.3. VSC-HVDC in the Training Environment

VSC-HVDC links are increasingly deployed in transmission grids to improve transfer capacity, controllability and renewable integration [1,13]. Their quasi-static model introduces decoupled active power injections at the rectifier and inverter terminals, with converter losses that grow with the transmitted power [13]. Unlike reactive compensation devices, a VSC-HVDC link controls active power flow directly across long distances, which makes it a first-line instrument for congestion relief rather than an auxiliary one. The companion paper [25] describes the scenario generator that produced the training corpus, including the genotype $\mathcal{G}=({\mathit{P}}_G,s_L,\mathit{o},P_{dc})$ from which each episode is initialised.

1.4. Contributions

The contributions of this paper are:

• a hybrid SAC architecture that unifies continuous redispatch, continuous HVDC control and discrete topology switching under a single maximum-entropy objective;
• the application of the Gumbel-Softmax Straight-Through estimator to topology switching in a full AC 118-bus environment, which makes the discrete heads trainable by backpropagation;
• one entropy temperature per action subspace (${\alpha }_c$, ${\alpha }_g$, ${\alpha }_l$), for subspaces whose entropy scales are not commensurable, together with a measurement of how each of the three behaves over training;
• a 906-dimensional state encoding that exposes full AC observability, including reactive branch flows, bus voltages and angles, and the HVDC set-point; and
• an evaluation on two independent 500-scenario batches and on a 1806-scenario out-of-training corpus, with every reported action verified by an independent AC power flow.

The rest of the paper is organised as follows. Section 2 describes the AC environment and its solver; Section 3 presents the hybrid SAC architecture; Section 4 gives the MDP formulation and the training protocol; Section 5 reports the experimental results; and Section 6 discusses them and concludes.

2. AC Environment and Power Flow Solver

2.1. Newton–Raphson AC Power Flow

The AC power flow equations enforce complex power balance at each bus:

$$P_i=\sum _j{V}_i{V}_j(G_{ij}\cos {\theta }_{ij}+B_{ij}\sin {\theta }_{ij}),Q_i=\sum _j{V}_i{V}_j(G_{ij}\sin {\theta }_{ij}-B_{ij}\cos {\theta }_{ij}),$$

(1)

where $V_i$ and ${\theta }_i$ are the voltage magnitude and angle at bus i and $G_{ij}+\mathrm{j}{B}_{ij}$ is the corresponding entry of the bus admittance matrix [1,2,26,27]. The Newton–Raphson iteration solves the mismatch system $\mathit{g}(\mathit{x})=\mathbf{0}$ with $\mathit{x}={[{\mathit{\theta }}^{\top },{\mathit{V}}^{\top }]}^{\top }$ by iterating

$$\mathit{J}({\mathit{x}}^{(k)})\Delta {\mathit{x}}^{(k)}=-\mathit{g}({\mathit{x}}^{(k)}),{\mathit{x}}^{(k+1)}={\mathit{x}}^{(k)}+\Delta {\mathit{x}}^{(k)},$$

(2)

with the block-structured Jacobian

$$\mathit{J}=\left[\begin{array}{cc}\mathit{H}& \mathit{N}\\ \mathit{M}& \mathit{L}\end{array}\right]=\left[\begin{array}{cc}\partial \Delta P/\partial \mathit{\theta }& \partial \Delta P/\partial \mathit{V}\\ \partial \Delta Q/\partial \mathit{\theta }& \partial \Delta Q/\partial \mathit{V}\end{array}\right].$$

(3)

CasADi [28] with the KLU sparse direct solver [29] evaluates (2) by exploiting the sparsity of the admittance matrix [2,30] and by pre-computing the symbolic Jacobian once at initialisation. Convergence is declared at ${10}^{-6}\mathrm{pu}$ with a limit of 50 iterations.

The environment uses two solver modes by design. At reset() the power flow is solved with the outer control loop enabled, so that on-load tap changers and switched shunts converge to a consistent position, which is then frozen. At every subsequent step() the plain Newton–Raphson solver is used with those devices held fixed. The asymmetry is deliberate: the outer loop is several times more expensive than a plain solve, and running it once per episode preserves the physical coherence of the initial state without paying its cost at every interaction.

2.2. VSC-HVDC Quasi-Static Model

The VSC-HVDC link (bus 89 → bus 12) is modelled as two coupled active power injections [13]:

$$\Delta P_{\mathrm{rect}}=-P_{dc},\Delta P_{\mathrm{inv}}=P_{dc}-P_{\mathrm{loss}}(P_{dc}),$$

(4)

where $P_{\mathrm{loss}}$ is the quadratic converter loss characteristic. Each terminal is rated at 300 MVA, so the set-point is bounded by $P_{dc}\in [0,3.0]\mathrm{pu}$ on a 100 MVA base. The agent does not command $P_{dc}$ directly: it commands an increment $\Delta P_{dc}=a_{\mathrm{hvdc}}·1.5\mathrm{pu}$ with $a_{\mathrm{hvdc}}\in [-1,1]$, and the resulting set-point is clipped to the admissible interval. One unit of action therefore spans half the converter rating, which lets the agent traverse the full range in two steps while retaining fine control.

2.3. Episode Initialisation and the Load-Scaling Defect

Each episode is initialised from a genotype $\mathcal{G}=({\mathit{P}}_G,s_L,\mathit{o},P_{dc})$ [25]. The reset() method applies the generator dispatch ${\mathit{P}}_G$, the load scaling $s_L$, the topology $\mathit{o}$ and the HVDC transfer $P_{dc}$, and then solves the power flow with the outer control loop. An episode is accepted when at least one branch satisfies ${\rho }_l>1.0$ after convergence.

An earlier version of reset() omitted the $s_L$ factor when initialising the demand vectors, a defect documented in [25]. Without $s_L$ every load defaulted to its base-case value, so the training distribution was systematically less loaded than the distribution used for evaluation. The agent trained on that corpus plateaued at 71.0% success rate while its training curves looked healthy; applying $s_L$ before the power flow solve raised the same configuration to 98.6%, an improvement of 27.6 percentage points that exceeds every architectural change reported in this paper. All results below were obtained after the correction.

3. Hybrid SAC Architecture

3.1. Decomposition of the Action Space

The action space $\mathcal{A}={\mathcal{A}}_c\times {\mathcal{A}}_g\times {\mathcal{A}}_l$ is decomposed into three subspaces:

• Continuous (${\mathcal{A}}_c={[-1,1]}^{54}$): 53 per-generator active redispatch increments for the non-slack generators, plus the HVDC set-point increment, parametrised by the standard squashed-Gaussian actor of SAC [20];
• Discrete generator head (${\mathcal{A}}_g\in \{0,\dots ,4\}$): index of the generator to disconnect, with the null action at index 0 and four pre-curated candidates at buses 46, 54, 87 and 111;
• Discrete line head (${\mathcal{A}}_l\in \{0,\dots ,13\}$): index of the line to open, with the null action at index 0 and thirteen candidates formed by five parallel pairs and three single circuits.

The generator at bus 12 and the lines 89–90 and 89–92 are excluded from the candidate sets because they are directly adjacent to the terminals of the VSC-HVDC link (bus 89 and bus 12); switching them would remove the very degree of freedom the agent is meant to exploit. Figure 1 shows the resulting architecture.

3.2. Gumbel-Softmax Straight-Through Estimator

Discrete topology actions are inherently non-differentiable: the $\arg \max$ operation that selects a branch to switch blocks gradient flow through the actor network, which prevents standard backpropagation from training the discrete heads. For a categorical variable $z\in \{1,\dots ,K\}$ with probabilities $\mathit{\pi }=\mathrm{softmax}(\mathit{\varphi })$, the Gumbel-Softmax distribution [23,24] draws

$${\tilde{z}}_k={\displaystyle \frac{\exp \left(({\varphi }_k+g_k)/\tau \right)}{{\sum }_{j=1}^K\exp \left(({\varphi }_j+g_j)/\tau \right)}},g_k=-\log (-\log u_k),u_k\sim \mathcal{U}(0,1),$$

(5)

where $\tau >0$ is the temperature. As $\tau \to 0$ the relaxation concentrates on the $\arg \max$ class and approaches the one-hot representation. The Straight-Through variant [23] uses the hard one-hot vector in the forward pass and the soft vector in the backward pass:

$${\mathit{z}}^{\mathrm{ST}}=\mathrm{onehot}\left(\arg \underset{k}{\max }{\tilde{z}}_k\right)-\mathrm{sg}(\tilde{\mathit{z}})+\tilde{\mathit{z}},$$

(6)

with $\mathrm{sg}(·)$ the stop-gradient operator. The discrete structure is therefore exact at execution time while the gradient still flows during training. The temperature is annealed geometrically towards a floor,

$${\tau }_k=\max \left({\tau }_{\min },{\tau }_0{(1-\kappa )}^k\right)\simeq \max \left({\tau }_{\min },{\tau }_0{e}^{-\kappa k}\right),$$

(7)

with ${\tau }_0=1.0$, ${\tau }_{\min }=0.1$ and a per-update rate $\kappa =5\times {10}^{-6}$, so the floor is reached after roughly $4.6\times {10}^5$ gradient updates.

3.3. Per-Subspace Entropy Temperatures

Standard SAC uses a single temperature $\alpha$ for the entropy of the whole policy. In the hybrid setting the three subspaces have entropy scales that are not commensurable: the entropy of the continuous Gaussian head depends on 54 log standard deviations, whereas the entropy of a categorical head is bounded by $\log 5$ and $\log 14$ respectively. A single temperature therefore produces conflicting gradient signals and leaves at least one head at an inappropriate exploration level.

Three independent temperatures $\{{\alpha }_c,{\alpha }_g,{\alpha }_l\}$ are introduced, each minimising its own loss:

$$\begin{array}{cc}\hfill \mathcal{L}({\alpha }_c)& ={\mathbb{E}}_{a_c\sim {\pi }_c}\left[-{\alpha }_c(\log {\pi }_c(a_c|s)+{\mathcal{H}}_c^{*})\right],\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \mathcal{L}({\alpha }_g)& ={\mathbb{E}}_{a_g\sim {\pi }_g}\left[-{\alpha }_g(\log {\pi }_g(a_g|s)+{\mathcal{H}}_g^{*})\right],\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \mathcal{L}({\alpha }_l)& ={\mathbb{E}}_{a_l\sim {\pi }_l}\left[-{\alpha }_l(\log {\pi }_l(a_l|s)+{\mathcal{H}}_l^{*})\right],\hfill \end{array}$$

(10)

with target entropies ${\mathcal{H}}_c^{*}=-54$, ${\mathcal{H}}_g^{*}=-\log 5$ and ${\mathcal{H}}_l^{*}=-\log 14$: the conventional SAC target for a 54-dimensional continuous action, and one target per categorical head scaled by the size of its catalogue. The coefficients are stored in logarithmic form and clamped independently to $\log {\alpha }_c\in [-3.0,1.0]$ and $\log {\alpha }_g,\log {\alpha }_l\in [-3.0,0.5]$. The total entropy term entering both the critic target and the actor loss is

$$\mathcal{H}(\pi )={\alpha }_c{\mathcal{H}}_c+{\alpha }_g{\mathcal{H}}_g+{\alpha }_l{\mathcal{H}}_l.$$

(11)

What the three coefficients actually do during training is worth reporting, because only one of them adapts. Across both full-length runs, ${\alpha }_g$ and ${\alpha }_l$ leave their initial value within the first few thousand updates and then sit at the lower clamp $e^{-3}=0.0498$ for the remainder of training: their 25th percentile, median and 75th percentile over the run all coincide with that bound. Only ${\alpha }_c$ tracks its target, oscillating between 0.136 and 0.42 with a run mean of 0.197–0.207 depending on the seed. In the configuration reported here, therefore, the mechanism behaves as one adaptive temperature for the continuous head plus two categorical heads held at minimum entropy pressure. Section 5 returns to this when examining which switching actions the agent actually uses.

3.4. State Space: 906-Dimensional AC Observability

The observation is a flat 906-dimensional vector composed of the ten blocks listed in Table 1. Two design choices deserve comment. First, the reactive branch flows $Q_{ij}$ are included alongside the apparent loading, which lets the agent distinguish a branch that is thermally loaded by active transit from one loaded by reactive circulation—a distinction that does not exist in a DC formulation and that determines whether redispatch or topology is the effective remedy. Second, the availability flags of the four candidate generators are included so that the discrete head can observe which of its own options remain, rather than having to infer it from the dispatch vector.

3.5. Networks and Hyperparameters

The actor uses a shared MLP trunk $[512,512]$ that splits into three heads: a continuous head producing $\mathit{\mu },\mathit{\sigma }\in {\mathbb{R}}^{54}$, a generator head with logits ${\mathit{\varphi }}_g\in {\mathbb{R}}^5$, and a line head with logits ${\mathit{\varphi }}_l\in {\mathbb{R}}^{14}$. The critic consists of twin Q-networks with hidden layers $[1024,1024]$, which take the concatenation ${[{\mathit{a}}_c^{\top },\mathrm{onehot}{(a_g)}^{\top },\mathrm{onehot}{(a_l)}^{\top }]}^{\top }$ of dimension 73 alongside the observation; the twin structure mitigates overestimation bias [21]. Table 2 lists the full configuration.

4. MDP Formulation and Training Protocol

4.1. Transition and Reward

The composite action at step t is ${\mathit{a}}_t=[{\mathit{a}}_{c,t},a_{g,t},a_{l,t}]$. The transition applies the redispatch and the HVDC increment, opens the selected generator or line if the corresponding index is non-null, and re-solves the AC power flow on the modified network. The reward combines a quadratic overload penalty, a redispatch cost, a switching cost and a terminal success bonus:

$$\begin{array}{cc}\hfill r_t=& -w_{\rho }{\displaystyle \sum _{l=1}^{186}}{\left[\max (0,{\rho }_l-1)\right]}^2-c_P\sum _g\left|\Delta P_{G,g}\right|\hfill \\ \hfill & -c_G\mathbf{1}[a_{g,t}\ne 0]-c_L\mathbf{1}[a_{l,t}\ne 0]+R_s\mathbf{1}[{\rho }_{\max }\le 1],\hfill \end{array}$$

(12)

with $w_{\rho }=10$, $c_P=0.1$ MW⁻¹, $c_G=c_L=2$ and $R_s=50$. The coefficients are named $w_{\rho }$ and $c_{(·)}$ rather than $\alpha$ and $\gamma$ to avoid collision with the entropy temperatures of Section 3.3 and with the discount factor. The redispatch increments $\Delta P_{G,g}$ are expressed in MW, so $c_P$ converts megawatts into reward units; the expression is a per-step quantity and the episode return is its sum over the steps taken. The quadratic form of the first term makes a severe overload attract far more corrective pressure than a marginal one, which directs the policy towards the critical branch when several are violated at once. The HVDC increment carries no explicit cost, a deliberate design choice: moving power through an existing converter does not incur the redispatch payments that a generator adjustment does.

An episode terminates successfully when every branch satisfies ${\rho }_l\le 1.0$, and is truncated at $T_{\max }=10$ steps. If the Newton–Raphson solver fails to converge after an action, the environment reverts the network to its previous state, returns a penalty of $-50$ and continues the episode rather than ending it. This is a deliberate departure from the more common practice of terminating on divergence: a non-convergent action is an unusable recommendation, not a system collapse, and letting the episode continue allows the agent to propose an alternative within the same decision cycle.

4.2. Scenario Sampling and Training Protocol

Episodes are drawn from the 253,378-scenario AC corpus of [25]. Sampling is stratified over the groups formed by the overload threshold and the outage catalogue of each shard, so that severe and mild congestion, and intact and post-outage topologies, are represented in proportion throughout training rather than in separate phases. 20% of the corpus index is held out at load time and never sampled during training. The runs reported here use the shards whose outage catalogue is either empty or of size five, which keeps the post-contingency population balanced against the intact-grid one.

Training runs for $1\times {10}^6$ environment steps after 2000 warm-up steps with random actions, with one gradient update per environment step and a checkpoint every 50,000 steps. In addition, a separate checkpoint is written whenever the success rate over a rolling window of the last 100 training episodes improves on its previous best; that is the checkpoint evaluated in Section 5. On the multi-core CPU used for the experiments, a full run takes approximately 20 hours, dominated by the Newton–Raphson solve at every step(). No GPU is used.

5. Results

5.1. Evaluation Protocol

Three evaluation sets are used, all disjoint from the training sample. Two batches of 500 scenarios each are converted to full network cases and processed end to end: the agent proposes actions, and an independent AC power flow re-solves the resulting network to confirm that the recommended state is physically attainable. The first batch contains intact-grid scenarios with initial maximum loading between 104% and 130%. The second is deliberately harder: it spans 101% to 199.82% loading and includes up to three simultaneous branch outages in the initial state. The third set is the independently generated 1806-scenario corpus of [25], used to compare the best rolling-window checkpoint against the checkpoint at the end of training. That comparison was run on a different seed from the two 500-scenario batches, and the table below states the provenance of every row.

5.2. Main Results

Table 3 reports the headline figures. The agent clears 493 of the 500 intact-grid scenarios (98.6%) and 487 of the 500 harder scenarios (97.4%), reducing the maximum branch loading by roughly 25 percentage points on average in both cases.

The last two rows, obtained on a different seed, show why the choice of checkpoint matters. On the same 1806-scenario corpus, the parameters at the end of training reach 92.6% while the best rolling-window checkpoint reaches 100%. The difference is not noise: the final-checkpoint policy moves 733 MW per episode on average but 3878 MW on the episodes it fails, which is the signature of a policy that has drifted towards large, indiscriminate redispatch. Reporting the last checkpoint of a long SAC run on a sparse-reward environment is therefore not a neutral choice. Note that the selection criterion is the rolling success rate over the last 100 training episodes, not a held-out score: it is a cheap proxy computed inside the training loop, and the 1806-scenario figures are an out-of-sample check of what that proxy selects, not the criterion itself.

Figure 2 shows the critic and actor losses over the ${10}^6$ training steps, and Figure 3 the episode reward and the redispatched power. The actor loss falls monotonically from its initial value and settles around zero without divergence, which is the practical evidence that the Straight-Through gradient estimator does not destabilise the actor update—the main risk of the approach. The critic loss behaves differently: it falls by roughly an order of magnitude over the first ${10}^5$ steps, reaches its minimum near step $5\times {10}^5$, and then drifts upward for the remainder of the run. That late growth is the quantitative counterpart of the checkpoint effect discussed above, and it is the reason the best rolling-window checkpoint rather than the final one is used throughout this section.

Figure 4 illustrates a representative episode of the harder batch, before and after the corrective action.

5.3. Breakdown by Severity

Table 4 decomposes the harder batch by initial loading and by number of simultaneous outages. Performance degrades smoothly and predictably with the initial loading: the agent attains 100% in the 100–105% and 115–120% bands, and loses about five points on the 130–200% tail. That is the expected shape, since the corrective margin available through redispatch and a single HVDC link shrinks as the initial violation grows.

The decomposition by outages is less intuitive. The failures concentrate on the single-outage population (10 of the 13 total), while the 104 scenarios with two simultaneous outages and the 5 with three are all solved. The explanation is a selection effect rather than a capability one: a scenario that survives two or three outages and still converges is, by construction, one whose remaining topology is robust, whereas the single-outage population includes cases in which the one line lost was precisely the critical one.

5.4. What the Agent Actually Does

Table 5 summarises the policy’s behaviour on the harder batch. Three observations stand out.

First, the policy is decisive rather than incremental: two thirds of the scenarios are cleared in a single step, and only 13 episodes reach the step limit.

Second, the HVDC link is not an auxiliary control. It is modulated in 97.2% of the episodes, its set-point rises from a mean of $0.742\mathrm{pu}$ at initialisation to $1.540\mathrm{pu}$ at the end of the episode, and the set-point moves by between $-1.50\mathrm{pu}$ and $+2.69\mathrm{pu}$ over the course of an episode, against a per-action bound of $\pm 1.5\mathrm{pu}$. The agent treats active power rerouting through the converter as a first-line instrument, which is consistent with the fact that the increment carries no explicit cost in (12).

Third, and least expected, the discrete head is extremely selective. Although thirteen line candidates are available, the agent uses only four of them across the 500 episodes, and a single line (56–59) accounts for 385 of the uses. The categorical head has effectively collapsed onto a small set of structurally useful switching actions rather than exploring the catalogue. Section 3.3 supplies a mechanism: the two categorical temperatures sit at their lower clamp for essentially the whole run, so the discrete heads receive almost no entropy pressure and have little incentive to explore beyond the options that worked early. Whether the outcome is a strength—a compact, interpretable switching policy that an operator could audit—or a symptom of that missing pressure is taken up in Section 6.

5.5. Reproducibility Across Seeds

Table 6 reports the four training runs performed. The three full-length runs are statistically equivalent. Averaged over the whole run, the success rate is 0.9820 for seed 42 and 0.9821 for seed 43, a difference of 0.01%; over the last tenth of each run—some 80,000 episodes—the figures are 0.952 and 0.957, with seed 43 marginally ahead.

The distinction between the two success-rate columns matters, because they are different quantities and can look contradictory when read together. The run mean is taken over every episode of the run. The final window is the mean over the last 100 episodes only, which is the statistic the training loop keeps for checkpoint selection; being a short window, it is noisy, and a value of 0.19 or 0.32 at the last episode reflects a transient dip rather than the state of the policy. Seed 42 shows dips of comparable depth elsewhere in its final 10,000 episodes; they simply did not land on its last episode.

What the runs do share is an end-of-training degradation. The mean success rate falls from 0.990 and 0.988 over the first tenth of each run to 0.952 and 0.957 over the last, and the critic loss drifts upward over the same interval (Figure 2). This is the behaviour that motivates evaluating a checkpoint chosen during training rather than the final parameters, and it is discussed further in Section 6.

Aggregated over the whole of the seed-42 run, the mean episode reward is $-41.63$, the mean episode length 1.248 steps, the mean rolling success rate 0.982 and the mean redispatched power 845 MW per episode, on scenarios whose initial maximum loading averages 119.5%. The gap between the 845 MW seen during training and the 2537 MW of the harder evaluation batch is a direct consequence of the difference in severity between the two populations.

5.6. Ablation

Table 7 reports the ablation carried out in the extended study [32], each row modifying a single element of the reference configuration. Replacing the Gumbel-Softmax Straight-Through estimator with parallel DQN heads and $ϵ$-greedy exploration costs 11.1 percentage points, because the gradient can no longer flow from the critic to the actor through the discrete head, which breaks the unified actor-critic character of the algorithm. Collapsing the three temperatures into one costs 4.6 points. Given that the two categorical coefficients already rest at their lower clamp in the reference configuration (Section 3.3), what the single-temperature variant mainly removes is the ability to regulate the continuous head independently of the discrete ones: one coefficient must then satisfy a target entropy of $-54$ and two targets of order $-\log 5$ and $-\log 14$ at once. Neither is comparable to the 27.6 points recovered by correcting the load-scaling defect, which remains the single most consequential change in the whole pipeline.

5.7. Numerical Robustness

Across the 500 episodes of the harder batch, no episode terminated through Newton–Raphson divergence, and the independent verification power flow converged on 99.6% of the final states. On the intact-grid batch the verification rate is 100%. The agent therefore does not exploit non-convergence as an escape from the reward, which is a real failure mode in environments that revert the state and continue: a policy can in principle learn to trigger divergence in order to avoid the overload penalty. The negative reward attached to divergence and the rollback of the state appear to be sufficient to prevent it here.

6. Discussion and Conclusions

6.1. Comparison with the DC Agent

The hybrid AC agent operates in a substantially harder environment than the DC redispatch agent of [33]: a 906-dimensional observation instead of a few hundred, a mixed discrete-continuous action space instead of a purely continuous one, reactive coupling and voltage constraints, and a Newton–Raphson solve at every interaction instead of two triangular substitutions. Training accordingly takes about 20 hours instead of roughly one. What the AC agent buys with that cost is the validity of its recommendations: an action verified by a full AC power flow is compatible with thermal, voltage and reactive limits simultaneously, whereas a DC-verified action may violate the latter two in practice. For a corrective tool intended to advise an operator, that difference is the point of the exercise.

6.2. Comparison with Related Approaches

PowRL [34] reports 90–95% survival on L2RPN scenarios using topology actions only, without continuous redispatch. AlphaZero-based agents [35] achieve high survival rates through Monte Carlo tree search, at the cost of a search at decision time that the single forward pass of an actor network avoids. MHSAC [36] reaches 96% success on a 14-bus system with a multi-head reward structure, and the RL2Grid benchmark [37] provides a standardised framework for future cross-system comparison. The safe RL approach of [38] adds constraint guarantees through Lagrangian multipliers, which would be directly compatible with the architecture developed here. These figures are not directly comparable with those of Table 3—the systems, the action spaces and the success criteria differ—and they are cited to situate the approach rather than to rank it.

6.3. Limitations

Three limitations bound the results.

The most important is the collapse of the discrete policy onto four of the thirteen line candidates, with a single line accounting for more than three quarters of the switching actions. A policy that always reaches for the same line is easy to audit but fragile, since it has no learned response for the scenarios in which that line is unavailable. Unlike the other limitations, this one has an identified mechanism: the two categorical temperatures rest at their lower clamp throughout training (Section 3.3), so the discrete heads are never pushed to explore. Only three seeds at a single configuration were run, so the reported performance should be read as that of a checkpoint from a converged run rather than as an expected value over seeds.

The second is the quasi-static formulation: every step is a steady-state snapshot, and the transients that follow a switching action are not represented. An action that is valid in steady state may still be unacceptable dynamically.

The third is observability. The 906-dimensional state assumes complete SCADA measurement of branch flows, voltages and angles; a partially observable deployment would require a recurrent or attention-based encoder, and the discrete candidate catalogues, derived by sensitivity analysis on this specific 118-bus topology, would have to be re-derived or learned for another network.

6.4. Conclusions

A hybrid Soft Actor-Critic agent for corrective congestion management in AC transmission networks with VSC-HVDC has been presented. The agent combines the Gumbel-Softmax Straight-Through estimator for differentiable discrete topology actions with a squashed-Gaussian actor for continuous generator redispatch and HVDC control, under a maximum-entropy objective in which each action subspace carries its own entropy temperature.

On the IEEE 118-bus system extended with a VSC-HVDC link, the selected checkpoint clears 98.6% of a 500-scenario intact-grid batch and 97.4% of a 500-scenario batch containing up to three simultaneous outages and initial loadings up to 199.82%, in a mean of 1.5 and 1.7 corrective steps respectively, with independent AC verification of 100% and 99.6% of the resulting states. Performance degrades smoothly with initial loading, from 100% below 105% to 94.4% on the 130–200% tail.

Two findings are worth carrying forward beyond this test system. The HVDC link is used as a primary control rather than an auxiliary one—it is modulated in 97% of episodes and its increments span the whole admissible range—which suggests that embedding converter set-points in the action space of a corrective agent is worth the additional dimension. And the single most consequential change in the whole pipeline was neither architectural nor algorithmic: correcting a load-scaling omission in the environment initialisation recovered 27.6 percentage points of success rate, more than the estimator and the temperature scheme combined.

The principal open problem is exploration in the discrete subspaces. Three observations point the same way: the two categorical temperatures rest at their lower clamp for the whole of training, the switching repertoire collapses to four of thirteen candidates, and the success rate drifts down over the final tenth of each run while the critic loss drifts up. Annealing the discrete entropy targets instead of clamping the temperatures is the most promising route. A second item is to establish the variance of the reported figures over more than the three full-length seeds run here. Beyond that, extension to larger systems, integration of reactive compensation devices as additional discrete actions, and validation against a dynamic simulation are the natural next steps.