Soft Actor-Critic Reinforcement Learning Improves Distillation Column Internals Design Optimization

1. Introduction

Computer-aided chemical process simulation has been an important tool not just in chemical engineering education, but also in commercial platform for the design and analysis of industrial systems that process bulk chemicals, specialty chemicals, pharmaceuticals and many more [1,2]. Among several software packages, the AspenPlus®, which is part of Aspen Tech’s suite of software packages [3,4,5], has been the state-of-the-art due to its wide array of functional capabilities (e.g., property analysis, rigorous models, process flowsheeting, API for external software such as Fortran and Python, etc.), and extensive database of chemical and physical properties [3,6,7,8,9,10]. Amid being the most advanced chemical process modeling software, this software has its inherent limitations [11]. One such limitation is that it still requires users to perform manual iterative optimization of certain designs that can sometimes result in extended periods spent on iterations by the user, or to sub-optimal final design [12]. These challenges can be exacerbated when dealing with a high-dimensional space of design parameters being explored [11,12]. Addressing these challenges and establishing solutions can significantly make the design process via this software tool (and other similar software packages) to be more efficient. This work aimed to demonstrate the use of reinforcement learning (RL), specifically the soft actor-critic (SAC) algorithm [13,14], to improve the design optimization task for the internals of multistage distillation column (Figure 1).

1.1. The Challenge: Distillation Column Internals Design

The specification of column internals in a multistage distillation column significantly dictates the hydraulics involved in the interactions of vapor and liquid streams through the various stages inside the column (Figure 1). This consequently creates a feedback loop in the usual design process – the hydraulics in the column become metrics of the goodness of the column internals design specifications [15]. In this study, the hydraulics parameter called percent (%) approach to flooding, which is denoted here using the notation $f_{\%}$, is used to measure the goodness of distillation column internals design. This parameter $f_{\%}$is not the only metric of good column design [2,15], but it is a significant indicator [16]. The numerous nonlinear relations and models involved in computing the column internal parameters that produce allowable values of $f_{\%}$ make this design task very difficult to automate using traditional programming techniques. Even the AspenPlus® software does not currently have a built-in automated optimization function for this task [3,10]. Hence, this current study demonstrates SAC RL approach to provide this valuable automation. For a seamless integration between the distillation column internal parameters, and the SAC RL model, this study uses the RadFrac module available in AspenPlus®. The RadFrac module is the most rigorous model in AspenPlus® fitting to be used in column internals design because of the stage-by-stage vapor-liquid calculations, hydraulics analysis, and its support for trayed and packed column designs [17,18]. A Python module called ‘pywin32’ [19] serves as API to read data from and to send data to AspenPlus®.

1.2. The Solution: SAC Algorithm-Based Design Optimization

This work evaluates the capability of SAC to automatically learn the best design parameter settings in the RadFrac column. SAC is an algorithm that optimizes a stochastic policy in an off-policy manner, which implements the combined capabilities of stochastic policy optimization and deep deterministic policy gradient (DDPG) approaches [13,14,20]. Because of the blend of stochastic and deterministic capabilities, the prediction of AC model for the next action steps are well within the bounds of the defined action space [13,14]. This is a feature that can be fitting to implement in distillation column design because the column parameters are allowed to be sampled only from a finite range of values due to limitations of fluid dynamics, vapor-liquid equilibrium, energy balance and other constraints imposed by natural law.

Note that a DDPG model was also evaluated during the preliminary works for this project and the DDPG model predictions failed to stay within the bounds of the action variables (column internal design parameters) resulting in input value errors of column design parameters when implemented in the AspenPlus® model file. This prompted the researchers to eliminate DDPG and focus only on SAC as the RL to evaluate for column design optimization in this work.

2. Methodology

2.1. Environment: Multi-Stage Distillation Column as RadFrac in AspenPlus

A model of a tray-type multi-stage distillation for the separation of ethanol (C₂H₅OH) and water (H₂O) was chosen because this binary chemical system is commonly used in many design analysis tasks in chemical engineering. Nonetheless, the methodology of this study should naturally be applied to other chemical systems undergoing distillation in a trayed column. The well-established AspenPlus software was chosen as the modeling platform for the tray-type column using its RadFrac module. Please refer to the accompanying AspenPlus file (.apw and .bkp) for all other details of the model used (see Data Availability section).

The details of the binary mixture to be separated via distillation are based on literature materials [2,15] and are as follows. A mixture of ethanol-water must be distilled in a sieve tray-type distillation column. The feed contains 35% ethanol and 65% water, and the distillate product stream must contain at least 80% ethanol. Note that 87 % ethanol is the azeotropic point of the binary system under atmospheric pressure [15], which is the reason for the target distillate concentration of ethanol. The feed is flowing at 20,000 kmol/hr, is a saturated liquid, and is coming to the column at pressure of 1 bar. The distillate rate must be at 104 kmol/hr and the external reflux ratio must be 2.5. After some preliminary graphical method of estimating the number of stages (using McCabe-Thiele method) [2,15,21], it was determined that the total number of stages must be 20 and the feed tray must be on stage 10. The column must have a total condenser, and a partial reboiler. The task is to design a sieve tray type multistage distillation column (see Figure 2) using the RadFrac module in AspenPlus®. Specifically, the column internals must be optimized with the objective of achieving the approach to flooding $f_{\%}$ levels to within the acceptable range, e.g., 80%-90% (see section 2.2).

To simplify the modeling in AspenPlus® and the implementation of SAC RL, the feed tray was assigned to the TOP section of the column. So, the TOP section covers the Stage 1 to Stage 10 while the BOT section covers the Stage 11 to Stage 20.

2.2. Distillation Column Flooding

The state variable for this study is the column flooding measured as % approach to flooding (denoted here as $f_{\%}$), which is one of the main performance metrics for sizing of distillation column internals [16]. Flooding is the excessive accumulation of liquid in the column resulting in poor equilibrium contact between the vapor and liquid streams. The % approach to flooding must be minimized and a typical range of % approach to flooding is 65% to 90% according to Wankat [15] while Jones and Mellborn [22] suggests a 75% approach to flooding is a good value for various chemical systems. There are four main mechanisms of flooding as shown in Figure 3. Downcomer backup flooding (Figure 3A) occurs when the liquid is backed-up into the downcomer due to tray pressure drop, which is usually caused by short tray spacing [16,22]. Downcomer choke flooding (Figure 3B) occurs when the entrance to the downcomer is too narrow resulting in build-up of excesses friction losses for the liquid to overcome [16]. Spray entrainment flooding (Figure 3C) occurs when there is a relatively low amount of liquid on the tray because of relatively short weir height and narrow downcomer clearance [16]. Froth entrainment flooding (Figure 3D) occurs when significant height of froth reaches the tray above resulting to compromise of the stream concentrations, and this can be a significant effect of settings in the tray spacing, sieve hole diameter, and fraction of the tray allocated for vapor-liquid exchange [15,16].

2.3. Notation

We now present notations to formally define the various components of the task in relation to the established notations in RL. The RL problem at hand is a policy search in a Markov Decision Process (MDP) involving the tuple $\left(S,A,p,r\right)$. The state space $S$ is a continuous space consists of the following column internal design parameters (Figure 1 and 2): top section (TOP) % approach to flooding, and bottom section (BOT) % approach to flooding. Note that the state % approach to flooding can be greater than 100% according to distillation column design principles, so $S\to \left[0,\infty \right)$. The action space $A$ is a continuous space consists of the following column internal design parameters (Figure 1 and 2): tray spacing, weir height, downcomer clearance, weir side length, and hole diameter. The action space is bounded by the allowable values of column design parameters in the AspenPlus software, and the actual values also scale with the units (i.e., ft, cm, or mm; kg or lb; etc.) they are assigned in the software by the user as explained above. The state transition probability $p$ represents the probability density of the next state $s_{t+1}\in S$ given the current state $s_t\in S$ and action $a_t\in A$. The environment (i.e., column internal design) emits a reward $r\in R \to \left[r_{min},r_{max}\right]$ from each transition (i.e., process simulation). Note that this current study also evaluated how various forms of the reward system $R$ may affect the SAC performance.

Table 1 summarizes the variables for the action space and state space with their corresponding description according to the actual settings of the AspenPlus model used in the study. Also refer to Figure 2 for visualization of the action variables defined in Table 1. The action space $A$ is multidimensional consists of the column design parameters denoted as $A_1,A_2,\dots A_{11}$ as shown in Table 1. The state space $S$ is two-dimensional consists of $S_1$ and $S_2$ (Table 1). Note that each section of the column, i.e., TOP and BOT, were represented in the action and state spaces even though the design parameters between the sections were similar.

2.4. SAC RL

We present in this section the key equations of the SAC RL algorithm, but the reader should consult the original works [13,14,20] that established this algorithm if more details are needed.

The goal in SAC RL is to determine a policy $\pi$ which maximizes two components: (1) the expected return $R\left(\tau \right)$ from the rewards $r_t$, where $\tau =\left(s_0,a_0,s_1,a_1,\dots \right)$ is the trajectory, when the agent acts according to the policy, and (2) the entropy of the policy $H\left(\pi \right)$, i.e., entropy of the predicted next actions. To achieve this, the policy model $\pi$ must be estimated by an optimal policy ${\pi }^{*}$ that is calibrated (via its model parameters) using data collected about from environment [14]. Policy in this context is a rule used to decide what next actions to take based on the current state of the environment. Specifically, $\pi$ is a probability distribution over the possible actions $a_t$ given the current state $s_t$ of the environment, i.e., $a_t~\pi \left(·|s_t\right)$. The optimal policy ${\pi }_{\theta }^{*}$ is modeled using deep neural network with network weights $\theta$, and this becomes the Actor network. In essence, the main goal in training a SAC RL model can then be expressed generally as:

$${\pi }^{*}=\arg \underset{\pi }{\max }{\mathsf{{\rm E}}}_{\tau ~\pi }\left[{\displaystyle \sum }_{t=0}^{\infty }{\gamma }^t\left(R\left(s_t,a_t,s_{t+1}\right)+\alpha H\left(\pi \left(·|s_t\right)\right)\right)\right]$$

(1)

where $\mathsf{{\rm E}}$ is the expectation, $\alpha$ is the trade-off coefficient for the entropy $H$ term, and $\gamma$ is the discount factor in the value function [20].

In general, the reward function $R$ depends on the current state $s_t$, the action just taken $a_t$, and the next state $s_{t+1}$ such that $r_t=R\left(s_t,a_t,s_{t+1}\right)$t. The dependence in the function $R$ is frequently simplified to have dependence only on the state, i.e., $r_t=R\left(s_t\right)$, or on the state-action only, i.e., $r_t=R\left(s_t,a_t\right)$. The SAC model used in this study used the former model $r_t=R\left(s_t\right)$. One of the two components of the objective function in the SAC RL agent is to maximize the cumulative reward over a trajectory $\tau$ and this cumulative reward is commonly called the return $R\left(\tau \right)$. It has been shown that a discounted return helps in the convergence of RL models, so this is also adopted in the SAC learning model such that $R\left(\tau \right)={\displaystyle \sum }_{t=0}^{\infty }{\gamma }^t{r}_t$, where $\gamma$ is the discount factor [14].

The maximum entropy term $H\left(\pi \right)$ (second term in Equation 1) is the unique feature of the SAC RL. By design, this term encourages the RL model to still explore as it learns to prioritize actions with projected high rewards [14]. This prevents the model from being focused on sub-optimal actions [23].

The determination of the best actor network ${\pi }_{\theta }^{*}$ is achieved by using value feedback from a system of Critic networks (see Figure 1), which are also deep neural networks, that implement double Q-learning [14]. SAC concurrently learns the policy ${\pi }^{*}$, and two Q-functions $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ of the Critic, where ${\varphi }_1$ and ${\varphi }_2$ are the network weights of the critic networks.

The smoothing constant, $\tau \in \left[0,1\right]$, is a hyperparameter that controls updating the weighs in of the target networks for the Q-functions $Q_{{\varphi }_1\text{'}}$ and $Q_{{\varphi }_2\text{'}}$ [24]. In notation: ${\varphi }_1\text{'}\leftarrow \tau {\varphi }_1+\left(1-\tau \right){\varphi }_1\text{'}$; and ${\varphi }_2\text{'}\leftarrow \tau {\varphi }_2+\left(1-\tau \right){\varphi }_2\text{'}$.

The fourth hyperparameter to be evaluated is the ‘replay buffer length’, which originates from using replay buffer as a solution to extract more information from the history sequential data for RL training [24]. Replay buffer is a component of the computation of the target Q-functions $Q_{{\varphi }_1\text{'}}$ and $Q_{{\varphi }_2\text{'}}$ presented above. The replay buffer is a finite sized (replay buffer length) memory data. The buffer is created by sampling transitions from the environment according to the exploration policy and the tuple $\left(s_t,a_t,r_t,s_{t+1}\right)$ is stored in the replay buffer. The buffer eventually is filled as more tuple entries are added. When the buffer length is reached, the oldest samples are discarded while maintaining the buffer length. At each timestep the actor and critic are updated by sampling a minibatch uniformly from the buffer.

2.5. Implementation: OpenAI Gymnasium, PyTorch

To eliminate the need to create our custom RL environment, we used the prebuilt RL module called “Gymnasium” by OpenAI [25,26]. This module implements the necessary algorithms and functions to define variable spaces and efficient action space-sampling mechanisms to ensure that RL models are exposed to a wide variety of scenarios resulting to improved learning performance [25,27]. The Python codes are based on the prior work by

2.5.1. Action Variables

Since all action variables in $A$ are continuous, these were defined using the ‘Gymnasium.Box()’ function, which is designed to handle tensor of continuous variables. The lower and upper bounds of these action variables were required arguments in the ‘Gymnasium.Box()’ functions.

2.5.2. State Variables

Defining the state space was very similar to how the action space (section 2.5.1) was defined using the ‘Gymnasium.Box()’ function.

2.5.3. Reward Scheme

The reward schemes studied are summarized in Table 2. Reward is based on the flooding level $f_{\%}$ of each column section and the scheme for awarding the reward values. Previous works have demonstrated that reward scheme can significantly affect the performance of RL algorithms [28,29,30,31]. The unique aspect of the reward schemes adopted for this study is the separate computation of reward for each column section $r_k$, and then aggregating these by summation (weighted and unweighted) into a single reward value $r$ (Table 2). Scheme 1 and Scheme 2 are built on a model that computes the penalty for deviating from the target state by taking difference of the target state value(s) and the current value of the state [31]. Scheme 3 and Scheme 4 are based on the concept of dividing the state values according to ranges and assigning the highest score on the target range while assigning decreasing reward score was the binning range moves away from the target range. Scheme 5 is built on top of a binary reward system; hence, it represents a binary scheme. The coding of these reward schemes were done in the environment definition via Python scripting.

2.5.4. Hardware Setup

The computer specifications are the following. Unit: Dell Precision 7670 Mobile Workstation; CPU: Intel Core i7-12850HX, 25MB Cache, 24 Threads, 16 Core, 2.1GHz-4.8GHz; Memory: 64GB DDR5; GPU: NVIDIA GeForce RTX 3080Ti with 16GB GDDR6. All computational work done were implemented in this laptop computer running the AspenTech version 12 software suite containing the AspenPlus®, and Anaconda Navigator version 2.5 software [32] containing Python version 3.10 and all the necessary Python packages such as PyTorch (CUDA-enabled), Gymnasium by OpenAI, and the pywin32 package [19] that interfaces with Windows to access the AspenPlus® file. Note that even though this study used a GPU-enabled computer to accelerate PyTorch computations, a CPU-only implementation of PyTorch can still accomplish the computations but at slower computation speeds. Nonetheless, the computations in this work do not require specialized computer hardware beyond a typical hardware requirement for running the AspenPlus® software.

2.5.5. Code and Documentation of the Work Done

The necessary codes used in this work have been organized and stored in an online repository via GitHub as detailed in the Data Availability section. There are two main code files: (1) a Python class file that defines all functions used to read data from (output data) and write data to (input data) AspenPlus® via Python, and (2) Jupyter Notebook files that document the implementation of the SAC RL.

2.6. SAC RL Runs

Since RL training is very sensitive to randomization of the initial weight in the deep neural networks in the actor and in the critic components of the SAC model, a set of ten runs was implemented for each learning setting. Each run has a unique random number generator (RNG) index value used for all PyTorch computations, which consequently fixed the random number used in initializing the weights of the neural networks. The ten unique RNG indices can be seen in the accompanying Jupyter Notebook files in the project online repository (see Data Availability section). This deliberate randomization using known RNG indices accomplishes two crucial tasks: (1) testing the robustness in SACR RL models, and (2) allowing for the repeatability of the randomized runs and results.

All runs were set to a maximum of 500 iterations per SAC learning run, where each iteration consists of one cycle of SAC model reading the current state and reward values of the environment (column model in AspenPlus®), SAC model making prediction for the next actions, and sending the predicted actions to the environment for simulation (to get the next state and reward for next cycle). Even though the traditional fields implementing RL (e.g., computer science, robotics, etc.) would usually set maximum iterations up to the range of millions [13], distillation column design iteration becomes impractical if the maximum iterations become very high, so our setting of a maximum of 500 iterations per SAC learning run was based on practical basis. The data collected from each run as discussed above would not require extensive downstream processing as these data by themselves are sufficient to evaluate the performance of SAC RL.

Furthermore, each run started with an untrained SAC RL model. This design of the study aimed to demonstrate that even an untrained SAC RL model can learn to optimize within a reasonable length of time (max of 500 iterations).

2.7. Data Collection and Analysis

There were numerous data that could be collected in this study because the nature of the environment is highly data-intensive inherent in computer-based design of distillation columns in AspenPlus®, and because the SAC RL training is data-intensive. This prompted the research team to focus collecting only on key data that would enable evaluation of the performance of SAC RL in column internals design. The data for following variables were saved and used for discussion: levels of the action variables, levels of the state variables, levels of the reward variables, and runtime per iteration. The saving of these data sets were coded in the run codes to eliminate user errors and streamline the analysis.

3. Results and Discussion

The results are presented in the following order: Section 3.1 covers the results to demonstrate the feasibility of using SAC RL for distillation column design; section 3.2 shows the effect of reward scheme on the performance of SAC RL; section 3.3 discusses the effect of column diameter on the performance of SAC; section 3.4 shows the effect of SAC hyperparameters setting on its performance; and section 3.5 covers the comparison of SAC RL with established optimization algorithms.

3.1. Feasibility of Using SAC RL for Distillation Column Internals Design

The first question that must be answered is “Does SAC RL work as automation method in optimizing distillation column internals design?” The answer is yes, it works, but there are pertinent trends in its performance that must be addressed as we now cover as we proceed. The evidence for this claim can be seen in Figure 4 and Figure 5, which were the results when SAC RL was implemented on the column with fixed uniform (in both TOP and BOT sections) diameter of 6-feet. Using the reward Scheme 1, the SAC model learned how to optimize the column design as the training progressed and converging to predicting column designs that maximize the reward (max of 200) as seen in Figure 4A, and selecting actions (column design parameter levels) that resulted to % flooding within the target range of 80-90% (Table 2 Scheme 1) as seen in Figure 4B and 4C. Initially, the SAC model does not have any optimal policy in determining the best next actions because the deep neural networks in the critic and actor components were randomly initialized. This is evident in the very large standard deviation of the reward values and state variable values at the beginning of the training (Figure 4A, 4B, and 4C). However, as the cycle of training-prediction-implementation steps progresses, the model learns the best actions that result to high rewards and the bad actions that result to low rewards, and eventually converges to optimal policies (actor and critic) that favor prediction of actions (Figure 5) that result in favorable $f_{\%}$ values resulting to highest rewards (Figure 4). This trend of convergence is very evident in the last 100 iterations, i.e., 400-500 iteration steps accompanied by smaller (narrower) standard deviation of reward and state variable values (Figure 4).

Another crucial aspect of the design task is the duration of the SAC RL automated optimization because chemical process design can be time sensitive [3]. The summary of runtime (in seconds) as shown in Figure 4D indicates that the average time per design iteration by the SAC model is 1 second (on the laptop computer used in this study – see section 2.5.4 Hardware Setup for details on the computer). This means that a single run with a maximum of 500 iterations (Figure 4) takes only around 10 minutes, which may be significantly lower than the time it takes to manually iterate the design.

With the numerous iterations that resulted in $f_{\%}$ values within the target range [80%, 90%) as shown in Figure 4B and 4C, the SAC-based optimization essentially exhaustively discovered numerous ways on how the column internal parameters can be set while achieving target design specification, i.e., $f_{\%}\in \left[80\%, 90\%\right)$. The action space trends shown in Figure 5 show the corresponding column design parameters for these design iterations that meet design specification.

3.2. Effect of Reward Scheme on the Performance of SAC

There are evidence in the literature of RL algorithms that show the impact of the reward scheme on the performance of RL models [25-27], and this was evaluated in the study. The results in testing the effects of the reward schemes (Table 2) are summarized in Table 3. The data in Table 3 was calculated using the results of the last 100 iteration steps (iteration steps 400 to 500) in each of the ten runs. The idea behind this data analysis is that the learning of the SAC RL model should be improving as it progresses in the sequence of iterations and that the best version of model is achieved towards the end of the learning process. Expectation is based on the proven convergence of SAC RL as more training data is introduced to the model [13,14]. Therefore, computing the fraction of these last steps of iterations that meet specific $f_{\%}$ cut-off would be warranted. Furthermore, the number of steps considered was 100 because the replay buffer length for the SAC model used in these runs is 100.

Looking at the mean and standard deviation results (Table 3), it can be observed that only Scheme 1, 2 and 3 achieved $f_{\%}$ values within the target interval [80%, 90%) while Scheme 4 and 5 produced $f_{\%}$ values above this target range. This supports prior observations in RL models that the reward scheme affects the performance of RL model, even the SAC RL model. An interesting comparison is between Scheme 3 and Scheme 4 because these two schemes are similar in terms of the binning intervals of the $f_{\%}$ values but with Scheme 4 reward values 10 times lower than those of Scheme 3 reward values. This means that the scaling of the reward values significantly affects the performance of SAC RL model.

The consequence of the effects of the reward scheme is also evident in the fraction of the iterations that meet cut-off values of the design metric $f_{\%}$ (Table 3). The best-performing reward schemes Scheme 1, 2 and 3 predicted column design specifications with $f_{\%}<90$ around at least 0.80 fraction of the iterations. The poor-performing schemes predicted column design specifications with $f_{\%}<90$ at low fractions: ~0.50 for Scheme 4, and ~0.35 for Scheme 5. Scheme 5 that represents a ‘binary scheme’ is the worst reward scheme.

3.3. Effect of Column Diameter on the Performance of SAC

With the positive results of implementing SAC RL as shown in the previous section, it was imperative to evaluate the limitation of the approach, and a direct way of accomplishing this is by varying the diameter of the column. This is because column flooding is inversely proportional to column diameter as shown in theory of distillation column design [2,15,16], i.e., $f_{\%}~{\displaystyle \frac{1}{{\left(\mathrm{Column} \mathrm{Diameter}\right)}^2}}$. One set of experiments used fixed column diameter values, 5 ft, 6 ft, and 7 ft; and another set included the column diameter ($A_{11}$) in the action space used by SAC RL model. The results of implementing SAC RL in these experiment settings are shown in Figure 6.

It can be seen in Figure 6 that the SAC RL converges in terms of its reward values and the resulting state variable values as model training progresses across all column diameter settings. In general, the flooding levels $f_{\%}$ decrease as the column diameter increase. This is consistent with the theoretical relation that $f_{\%}$ in inversely proportional to column diameter (relation shown above). When the column diameter is lowest at 5 feet, the converged SAC RL model was just approaching 100% flooding from above (Figure 6A and 6B) with actual values reaching only ~110% flooding as the lower limit. This means that it is impossible to have a 5-ft column that can operate at the target [80%, 90%) range of $f_{\%}$. This impossibility can be quickly checked by implementing the SAC RL as was done here instead of the user spending a very long time manually iterating for something that cannot happen (if column diameter is fixed at 5 ft).

At this point, it is imperative to ask the question “If the flooding $f_{\%}$ is inversely proportional to column diameter, then why not just keep on increasing the diameter?” The answer comes from the limitations of fluid dynamics in the column when diameter is set too large. The issue of ‘weeping’ can occur when there is a large pool of liquid on the tray (due to large column diameter) while vapor pressure is too low due to larger active area of vapor-liquid contact resulting from larger diameter [2,15]. This study does not cover solving the issue of weeping (it can be covered in a future extension of SAC RL in a similar approach). This then begs the question “How was this trade-off in column diameter and flooding included in the SAC RL model?” This trade-off was included by implementing a specialized reward scheme by modifying reward Scheme 1 (Table 2) as follows and let us call this Scheme 6 for quick referencing:

$$r={\displaystyle \frac{1}{3}}{r}_{TOP}+{\displaystyle \frac{1}{3}}{r}_{\mathrm{BOT}}+{\displaystyle \frac{1}{3}}{d}_{dia}$$

(2)

where $d_{dia}={\displaystyle \frac{100}{\max \left(\mathrm{Col}. \mathrm{Diamater}\right)-\min \left(\mathrm{Col}. \mathrm{Diamater}\right)}}\left[\min \left(\mathrm{Col}. \mathrm{Diameter}\right)-A_{11}\right]$,

and $r_k$ is the reward in section $k$ of the column. For each $k$:

$$r_k=\{\begin{array}{c}100, 80\le f_{\%}<90\\ 100-2d_{err}, f_{\%}<80\\ 100-2d_{err}, f_{\%}\ge 90 \end{array}$$

where $d_{err}=\left|f_{\%}-85\right|$.

This reward Scheme 6 uses the maximum and minimum possible values of the column diameter, i.e., [4 ft, 9 ft] to compute a scaling factor adjusted to a magnitude of 100 as maximum but assigned a negative value, i.e., $\left[\min \left(\mathrm{Col}. \mathrm{Diameter}\right)-A_{11}\right]\in \left[0,-\infty \right)$, to penalize the reward $r$ when the predicted action value for column diameter $A_{11}$ gets too large. In essence, $d_{dia}\in \left[-100,0\right]$ is a penalty value that becomes more negative as column diameter increases. This Scheme 6 bounds reward $r$ resulting in the convergence of the SAC RL model as shown in Figure 6G, 6H, and 6I.

3.4. Effect of SAC Hyperparameters on Performance

Inherent in training RL models is the tuning of hyperparameters in order to further refine the models. This section covers the results in evaluating the effect of four hyperparameters: $\alpha$, $\gamma$, $\tau$, and reply buffer length. Of the numerous hyperparameters, these four have direct effects on SAC RL because of their roles in the fundamental equations of SAC. Table 4 summarizes the flooding results for the top section (TOP) of the column set at fixed diameter of 6 feet. Even though data were also collected for the bottom-section (BOT) of the column, space limitations in the manuscript prompted reducing the presented data to half; hence, Table 4 shows data for TOP section.

The effect of hyperparameter $\alpha$ can be seen by first comparing the results between Setting 10 and 11, between Setting 7 and 8 (Table 4). The general trend is that a lower $\alpha$ value of 0.1 will produce higher fractions of runs that fall within the target condition $f_{\%}<90$ compared when $\alpha$ is set higher at 0.5. However, when comparing between Setting 8 and 9, it can be seen that $\alpha =0.1$ has lower fraction $f_{\%}<90$ compared to $\alpha =0.2$. This means that there is an optimal value for $\alpha$ in the range 0.1 to 0.5, and this value may be close to 0.2. Since the value of $\alpha$ is a measure of the fraction of entropy contribution in the loss function (see Equation 1), this observed trend means that the term in the loss function for maximizing entropy should not be zero but it should also be not very large part of the loss function. This is just a fitting observation because this entropy term is a unique feature of the SAC in comparison to other RL algorithms implementing actor-critic networks [13,23].

The effect of hyperparameter $\tau$ can be seen by first comparing results between Setting 3 and 10, and between Setting 4 and 12 (Table 4). Both pairs show that lower value of $\tau$ results to higher fraction $f_{\%}<90$. If there is any optimum value somewhere, these data pairs cannot support such possibility – perhaps a more extensive scan of the hyperparameter space can discover an optimum but this is beyond the scope of this study. Since hyperparameter $\tau$ represents the smoothing coefficient for the target Q-functions, setting its value to zero may not be a good idea because this number controls the update of the Q-function deep neural network weights.

The effect of hyperparameter $\gamma$ can be seen by first comparing results between Setting 1 and 2, between Setting 4 and 5, and between Setting 12 and 13 (Table 4). It can be seen in these pairs that the trends for fraction $f_{\%}<90$ do not follow a clear consistent trend. It is possible that there are confounding effects in the SAC RL training that cannot be easily isolated to be sole effect of $\gamma$. Therefore, there is no conclusive trend for the effect this hyperparameter based on data.

The effect of hyperparameter ‘replay buffer length’ can be seen by first comparing results between Setting 2 and 4, between Setting 6 and 13, and between Setting 8 and 11 (Table 4). It can be seen in these pairs that the trends for fraction $f_{\%}<90$ do not follow a clear consistent trend. Similar to hyperparameter $\gamma$, it is possible that there are confounding effects in the SAC RL training that cannot be easily isolated to be sole effect of ‘replay buffer length’. Therefore, there is no conclusive trend for the effect this hyperparameter based on data.

3.5. Comparison of SAC RL Algorithm with Established Optimization Algorithms

A pertinent question at this point is “How does SAC RL performance compare with established optimization algorithms?” This question is necessary to answer because this study is proposing to use SAC RL in optimization task, which has been heavily studied [33]. The tested algorithms are the following: Nelder-Mead [34], Broyden–Fletcher–Goldfarb–Shanno (BFGS) [35], Sequential Least Squares Programming (SLSQP) [36], Dual Annealing [37], and simplicial homology global optimization (SHGO) [38], which were implemented using their pre-built functions in the Python module SciPy [39]. To simplify the comparative analysis, the target flooding of $f_{\%}=85$ was used, which was based on the reward Scheme 1 used for SAC RL. Each algorithm was implemented ten run times with each run assigned with a unique RNG index. Figure 7 summarizes the results of grouped in two (1) at fixed column diameter of 6 ft as shown in Figure 7A-7F, and (2) at variable column diameter with penalty on the diameter the same as that of Scheme 6 as shown in Figure 7G-7H.

It can be seen that each of the algorithms tested exhibited unique performance in terms of optimizing to 85% flooding. Poor performance can be seen for BFGS and SLSQP that exhibited early stops (Figure 7C and 7D) while dua annealing and SHGO completed the runs but with erratic behavior in various stages of the iterations (Figure 7E and 7F). The Nelder-Mead exhibited a smooth convergence to the target when the column diameter is fixed (Figure 7B). However, Nelder-Mead failed to converge to the target when the column diameter was included in the varied parameters (Figure 7H). We also want to note that Dual Annealing and SHGO were also tested when the column diameter was varied but these algorithms exhibited erratic performance and failed to converge to the target. The SAC-RL algorithm was able to converge to the target in both the fixed column diameter (Figure 7A) and variable column diameter (Figure 7G) design scenarios. These trends show the advantage of SAC RL over established optimization algorithms for the task of optimizing distillation column internals design. Due to SAC RL’s maximum entropy term, it continuous to explore as it learns resulting to prevention of exploitation of actions that may be sub-optimal.

4. Conclusions

This study focused on developing an automated method using soft actor-critic (SAC) reinforcement leaning for optimizing the design of distillation column, and this aim was achieved as supported by the results. The integration of SAC RL into AspenPlus® model optimization was achieved by using Python codes written to allow a cyclic interaction between the distillation column model AspenPlus® and the SAC model in PyTorch. The specific chemical system used as case study is a binary mixture of ethanol-water, and the RadFrac module in AspenPlus® was used to enable rigorous column internal design. The results clearly support the following findings: (1) SAC RL works as automation approach for the design of distillation column internals, (2) the reward scheme in the SAC model significantly affects SAC performance, (3) column diameter is a significant constraint in achieving column internals design specification in flooding, and (4) SAC hyperparameters have varying effect on SAC performance.

This study also demonstrated that an untrained SAC RL model can quickly learn how to optimize the design of a distillation column. This means that the algorithm can be implemented as one-shot RL for column internals design. This has significant implications for possible future implementation of the technique as integral part of computer-based chemical process modeling, e.g., if this technique becomes part of AspenPlus® software (and other similar computer packages). That is, there is no need to install a pre-trained SAC RL model into a computer-based chemical process simulation package that would utilize this technique.