Thrust and Pressure Control in Solid Propulsion System via Reinforcement Learning

1. Introduction

The Attitude and Divert Propulsion System (ADPS) is a propulsion system in aerospace engineering designed to adjust spacecraft attitude, ensuring three-axis stability and enabling orbital maneuvers [1]. ADPS functions by ejecting gas through multiple nozzles: attitude thrust nozzles at the spacecraft's rear control pitch, yaw, and roll, while divert thrust nozzles at the center of mass provide lateral thrust [2]. ADPS types include the liquid attitude and divert propulsion system (LADPS) and the solid attitude and divert propulsion system (SADPS), each with specific advantages and disadvantages tailored to particular spacecraft design objectives and performance requirements [3].The LADPS uses liquid oxidizer and fuel as propellants, allowing for repeated thrust start and stop and providing a higher specific impulse. However, it requires complex fuel storage and delivery systems that need regular maintenance [4]. In contrast, the SADPS uses solid oxidizers and fuels, which eliminates the need for complex pipeline delivery systems, offering ease of maintenance, safety, and storage. Variable thrust in SADPS is achieved by altering the throat area of the nozzle in real-time [5]. Precise control of nozzle openings and the propellant combustion rate enables pressure and thrust regulation in the combustion chamber [6]. However, the shared combustion chamber among multiple nozzles and the pressure-dependent combustion rate result in a gradual increase in free volume, necessitating intricate control algorithms for safe and reliable operation.

The dynamics of the Solid Attitude and Divert Propulsion System (SADPS) are characterized by nonlinear and time-varying behavior due to nozzle control, pressure regulation, and thrust management. These characteristics make traditional control methods like Proportional Integral Derivative (PID) inadequate for meeting performance standards [7]. Sergio's review examines closed-loop pressure control methods in rocket propulsion, typically using linearized models around specific operating points to derive steady-state controllers, predominantly of the PID type. Sergio emphasizes that future control strategies must robustly achieve the desired thrust levels. Potential solutions include enhanced nonlinear approaches and optimal closed-loop control [8]. Ming Zhu designed a pressure control system for pintle solid rocket motors using parameter identification through adaptive control methods, demonstrating significant advantages in terms of stability, robustness, and precision [9]. Bergmans and Di Salvo introduced an adaptive matching technique that adjusts pre-designed controllers based on evolving system dynamics [10], while Joner and Quinquis proposed adaptive controllers to mitigate uncertainty effects [11]. Waxenegger-Wilfing proposed a data-driven method for early warning of thermoacoustic instabilities in rocket engine thrust chambers, successfully using machine learning to predict these instabilities in advance [12]. Previous research extensively investigated spacecraft propulsion, aiming to minimize fuel consumption for effective thrust command allocation and stable attitude control [13,14,15]. Lee and Lim applied adaptive feedback linearization techniques to achieve pressure regulation and conducted extensive work on five-axis thrust distribution [16,17,18].

The integration of artificial intelligence into control systems has led to significant advancements, with Reinforcement Learning (RL) emerging as a prominent approach for addressing sequential decision problems [19]. RL offers advantages such as learning from experience, flexibility, autonomous decision-making, robustness, and resource efficiency [20]. The development of an RL-based mission planning strategy for a Partial Space Elevator (PSE) addresses the challenge of maintaining the main satellite’s orbital stability during climber operations, demonstrating that using a deep Q-network to optimize the waiting intervals between transfer missions can minimize orbital radius changes effectively [21]. Skarka’s [22] review explores the application of Machine Learning (ML) and RL techniques in enhancing the navigation and obstacle avoidance capabilities of Unmanned Aerial Vehicles (UAVs), revealing the effectiveness of various RL algorithms in enabling autonomous navigation through real-time environmental interactions. Additionally, Pham’s [23] research introduces an RL framework for robotic manipulators to adapt to joint malfunctions during tasks, achieving a high success rate of 93.6% in diverse failure conditions. Furthermore, a proposed trajectory tracking control algorithm for Unmanned Surface Vehicles (USVs) employs an improved Deep Deterministic Policy Gradient (DDPG) method [24], demonstrating its effectiveness in dynamic ocean environments through experimental validation.”

In aerospace vehicle control, RL has demonstrated potential in addressing complex and uncertain environments, enhancing autonomous decision-making capabilities, and handling nonlinear control problems. However, there is a lack of literature on the application of RL in aerospace propulsion systems. Leveraging RL, we developed an intelligent control method and architecture for the Solid Attitude and Divert Propulsion System (SADPS). Considering the system's strong non-linearity, multiple inputs and outputs, disturbances, and coupling characteristics, we established a multi-agent architecture for pressure, thrust, and nozzle opening control. We devised an adaptive reward function and trained an expert neural network, demonstrating superior control effects compared to PID. Simulation tests validated the effectiveness of the intelligent control method.

2. Model and Algorithm

2.1. Solid Attitude and Divert Propulsion System (SADPS)

As shown in Figure 1, the four divert thrust nozzles are arranged at the spacecraft's center of mass, generating thrust in the y-axis and z-axis directions. Additionally, six attitude thrust nozzles are positioned at a certain distance from the center of mass, providing pitch, yaw, and roll control along the three axes.

The SADPS operates in three modes: power-saving mode, performance mode, and overload mode.

• Performance mode (Figure 2) requires a medium level of thrust to adjust the spacecraft's orbit and pitch angle.
• Power-saving mode minimizes propellant loss by reducing pressure as much as possible while maintaining flight attitude and compensating for gravity.
• Overload mode necessitates rapid changes in the spacecraft's trajectory and flight attitude.

Typically, chamber pressure levels are set to constant values to ensure stability and predictability in performance.

Figure 2. Three operating states of the solid attitude and divert propulsion system.

Figure 2. Three operating states of the solid attitude and divert propulsion system.

From time t₀ to t₁, the SADPS operates in performance mode, during which the spacecraft adjusts its attitude and changes its orbit to aim at the target. From time t₁ to t₂, the SADPS switches to power-saving mode, maintaining orbital altitude and overcoming Earth's gravity. From time t₂ to t₃, the engine enters overload mode, performing the final rapid maneuvers to correct any deviation from the target.

The control system model is depicted in Figure 3. The gas generator primarily produces the working medium required for thrust. The hot gas can be expelled simultaneously through four divert thrust nozzles and six attitude thrust nozzles. The pressure controller regulates the gas generator’s pressure and provides the total opening command for the nozzles. The thrust controller allocates the opening of different nozzles and, within the constraints of the total nozzle opening, generates the required five-axis thrust. The opening controller drives the motor to rotate according to each valve’s opening instructions, completing the closed loop of nozzle operation. Through the coordinated hierarchical work of the three controllers, the switching of the engine’s operating modes and the output of thrust are achieved.

Each nozzle's aperture size is adjusted by a throat plug-type gas valve. This valve comprises a plug-shaped element located in the center of the valve throat. When this element interacts with the inner wall of the nozzle, it forms an annular gas channel known as the equivalent throat area. By altering the position of the throat plug, the equivalent throat area is adjusted, thereby controlling the nozzle's output thrust [25]. The throat plug regulating valve consists of several components, including a brushless DC motor, a decelerator, a valve body, a nozzle, thermal protection components, and transmission components. The throat plug is driven by the rotary motor through a rotary-to-linear transmission mechanism, causing it to reciprocate horizontally.

2.2. Reinforcement Learning Trainer

The PPO algorithm consists of three neural networks: Actor_learn, Actor_choose,and Critic. It aims to improve strategy actions by computing action mean and variance, then selecting the final action via random sampling. Advantage clipping limits strategy updates to ensure robustness [26]. The basic PPO structure is shown in Figure 4.

The ${\pi }_{{\theta }_{old}}\left(a_t|s_t\right)$ chooses action based on the s_t and receives reward. The $V_{\varphi }\left(s_t\right)$ evaluates state values V_t and output TD error. During updates, $\stackrel{̑}{A_t}$ is backpropagated and clipped within the range (1−ε,1+ε) to refine both the Critic and Actor_learn networks. Periodically, the Actor_learn parameters are copied to Actor_choose for better action selection.

To accelerate training, we applied several techniques to improve the efficiency of PPO:

• Network Simplification: Replaced neural network hidden layers of widths 256 and 128 with widths 64 and 64, reducing parameters by 87% and inference time by over 20% without compromising control effectiveness.
• Generalized Advantage Estimation [27]: Utilized generalized advantage estimation during network weight updates. Calculating the mean and variance of the advantage function across the entire batch made the training process smoother and more stable.
• Learning Rate Adjustment: Set initial learning rates for the Actor and Critic to 1e-4 and 3e-4, respectively. Reduced them exponentially by an order of magnitude over 1000 training epochs for more stable convergence.
• Random Initialization and Gradient Clipping: Initialized network parameters with samples from a standard normal distribution and applied gradient clipping once before updating the network.
• Activation Function Change [28]: Replaced ReLU with Tanh in the hidden layers to avoid the vanishing gradient problem associated with negative values.

2.3. Training Environment

2.3.1. Chamber Pressure Dynamics

To regulate pressure, a combustion chamber pressure dynamics model is established for training the pressure control agent. The pressure-throat area relationship in a solid rocket motor is described by a zero-dimensional internal ballistic equation [29]:

$${\displaystyle \frac{V_c}{\sqrt{k}{\left(\frac{2}{k+1}\right)}^{\frac{k+1}{k-1}}{c^{*}}^2}}{\displaystyle \frac{d{p}_c}{dt}}={\rho }_p{A}_{b}a{p}_c^n-{\displaystyle \frac{p_c{A}_{tsum}}{c^{*}}}$$

(1)

Here, V_c, c^*, and k denote the combustion chamber free volume, characteristic velocity, and heat capacity ratio of the gas, respectively. A_b and ρ_p represent the burning area and density of the propellant, and the burning rate is given by aP_cⁿ, where P_c is the chamber pressure. The term A_tsum denotes the total nozzle opening. Typically, once the propellant grain shape of the engine is confirmed, the thickness of the burned propellant can describe the variation of V_c and A_b. By modifying A_b, it is possible to simulate the dynamic process inside the combustion chamber using numerical solution methods, thereby creating a training environment for the pressure agent. In this study, we employed the fourth-order Runge-Kutta method to solve the equations.

2.3.2. Five-Axis Thrust Calculation

In Figure 5(a), divert nozzles produce lateral acceleration by thrusting at the spacecraft's center of mass, typically in the range of hundreds to thousands of newtons. This allows for rapid orbital adjustments without attitude changes. Nozzles 0 and 2 act in the z-axis, while nozzles 1 and 3 act in the y-axis.

Fig.5(b) illustrates attitude control nozzles at the spacecraft's tail, exerting tens to hundreds of newtons. Nozzles 4 and 7 induce clockwise rolling, while nozzles 5 and 6 induce counterclockwise rolling. Nozzles 4 and 5 pitch up, nozzles 6 and 7 pitch down, nozzle 8 causes left yaw, and nozzle 9 causes right yaw.

The equations for the synthesis of thrust in five-axis direction are as follows:

y-axis thrust:

$$F_y=F_3-F_1$$

(2)

z-axis thrust:

$$F_z=F_2-F_0$$

(3)

yaw-axis thrust:

$$F_{yaw}=F_9-F_8$$

(4)

pitch-axis Thrust:

$$F_{pitch}=(F_6+F_7)-(F_4+F_5)$$

(5)

roll-axis Thrust:

$$F_{roll}=(F_6+F_5)-(F_4+F_7)$$

(6)

The thrust generated by each nozzle can be calculated according to the thrust calculation Equation. The thrust can be expressed as the product of the thrust coefficient C_f , pressure P_c, and the throat area A_t.

$$F=C_f{P}_c{A}_t$$

(7)

2.3.3. Valve Servo Motion

Achieving closed-loop control of nozzle opening by regulating the valve core position is crucial. Modeling the dynamic behavior of the valve accurately is challenging due to the presence of various mechanical components and frictional forces within the gas valve. To address this, we employ system identification techniques to tailor the servo system.

We establish a fourth-order state-space model, achieving a system similarity of 92.1%. The identified observable state space is represented in Equation (8):

$$\begin{array}{c}\dot{x}\left(t\right)=Ax\left(t\right)+B{\omega }_{cmd}\\ y\left(t\right)=Cx\left(t\right)+D{\omega }_{cmd}\end{array}$$

(8)

where x(t) denotes an n-dimensional state vector, A is the system matrix, B represents the control matrix, ${\omega }_{cmd}$ denotes the speed command, C stands for the output matrix, D indicates the direct transfer matrix, and y(t) symbolizes the motor speed. Figure 6 illustrates the system output obtained by applying the same swept frequency signal input to both the actual and identification systems, as referenced in prior work [30].

2.4. Reward Function

2.4.1. Pressure Control Agent’s Reward

The pressure control agent utilizes a reward function to manage the total nozzle opening and stabilize chamber pressure near the command, considering three key aspects:

(1)

Deviation Penalty: When chamber pressure significantly deviates from the command, a penalty term is applied to promptly correct the pressure.

$$R_{Pc\_deviation}=-\left|P_{ccmd}\left(t_n\right)-P_c\left(t_n\right)\right|$$

(9)

(2)

Synchronization Reward: As chamber pressure approaches the command, a synchronization reward stabilizes the pressure, preventing rapid oscillations.

$$R_{Pc\_synchronization}=-\left|\left(P_c\left(t_n\right)-P_c\left(t_{n-1}\right)\right)-\left(P_{ccmd}\left(t_n\right)-P_{ccmd}\left(t_{n-1}\right)\right)\right|$$

(10)

(3)

Steady-State Reward: When chamber pressure consistently tracks the command, a small reward encourages maintaining this state.

$$R_{Pc\_bonus}=1\hspace{1em}if\hspace{1em}\left|R_{Pc\_deviation}\right|<{\epsilon }_1\hspace{1em}and\hspace{1em}\left|R_{Pc\_synchronization}\right|<{\epsilon }_2\hspace{1em}else\hspace{1em}0$$

(11)

(4)

Dynamic Weights: When the pressure deviation is large, the deviation penalty is intensified; when the pressure deviation is small, the synchronization reward is intensified. We aim for the deviation penalty to have at least half of the weight to ensure tracking accuracy.

$$K_{Pc}=0.5tanh(\left|P_{ccmd}\left(t_n\right)-P_c\left(t_n\right)\right|)+0.5$$

(12)

Finally the pressure control agent’s reward function is as shown in Equation (13):

$$R_{P_c}=K_{P_c}{R}_{Pc\_deviation}+\left(1-K_{P_c}\right)R_{Pc\_synchronization}+R_{Pc\_bonus}$$

(13)

2.4.2. Thrust Distribution Agent’s Reward

The thrust distribution agent controls the opening degree of each of the nozzles, with the opening degrees of these nozzles being unknown variables. The known constraints include the 5-axis thrust equation and the total nozzle opening degree command. The reward function design is shown in Equations (14) to (19).

$$R_{F_y}=-\left|F_{ycmd}\left(t_n\right)-F_y\left(t_n\right)\right|$$

(14)

$$R_{F_z}=-\left|F_{zcmd}\left(t_n\right)-F_z\left(t_n\right)\right|$$

(15)

$$R_{F_{roll}}=-\left|F_{rollcmd}\left(t_n\right)-F_{roll}\left(t_n\right)\right|$$

(16)

$$R_{F_{pitch}}=-\left|F_{pitchcmd}\left(t_n\right)-F_{pitch}\left(t_n\right)\right|$$

(17)

$$R_{F_{yaw}}=-\left|F_{yawcmd}\left(t_n\right)-F_{yaw\_target}\left(t_n\right)\right|$$

(18)

$$R_{val}=-\left|A_{tsumcmd}\left(t_n\right)-A_{tsum}\left(t_n\right)\right|$$

(19)

Under normal circumstances, pressure control is prioritized, making the reward item for the total valve opening degree the primary concern. A switch function with a threshold T is implemented; only when this reward item reaches T will other thrust deviation rewards be considered. The thrust distribution agent's reward is finally expressed in Equation (20).

$$R_F=\left\{\begin{array}{c}{R}_{val}+R_{F_y}+R_{F_z}+R_{F_{roll}}+R_{F_{pitch}}+R_{F_{roll}} ,\hspace{1em}if R_{val}>T\\ R_{val} ,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}if R_{val}\le T\end{array}\right.$$

(20)

2.4.3. Opening Closed-loop Agent’s Reward

The opening closed agent is designed to control the motor's displacement, ensuring it quickly approaches and stabilizes near the command, or accurately tracks a continuous curve.

(1)

Displacement Difference: Using the displacement difference as a reward component effectively penalizes significant deviations from the command during initial training, speeding up convergence.

$$R_{opening\_deviation}=-\left|A_{ticmd}\left(t_n\right)-A_{ti}\left(t_n\right)\right|$$

(21)

(2)

Velocity Term: To match the valve core's movement speed with the target curve's slope, a velocity term is added. This requires transforming the observation space, using the displacement difference between new and old states to represent the target curve's slope.

$$R_{opening\_synchronization}=-\left|\left(A_{ticmd}\left(t_n\right)-A_{ticmd}\left(t_{n-1}\right)\right)-\left(A_{ti}\left(t_n\right)-A_{ti}\left(t_{n-1}\right)\right)\right|$$

(22)

(3)

Steady-State Reward: An additional reward is given when both displacement and speed rewards are satisfactory, promoting a state where both closely follow the target curve.

$$R_{opening\_bonus}=1\hspace{1em}if\hspace{1em}{R}_{opening\_deviation}<{\epsilon }_3\hspace{1em}and\hspace{1em}{R}_{opening\_synchronization}<{\epsilon }_4\hspace{1em}else\hspace{1em}0$$

(23)

(4)

Dynamic Weighting: To ensure accuracy, the displacement difference should account for at least half of the weight in the reward function.

$$K_{opening}=0.5tanh(\left|A_{ticmd}\left(t_n\right)-A_{ti}\left(t_n\right)\right|)+0.5$$

(24)

Finally the opening closed-loop agent’s reward function is as shown in Equation (25):

$$R_{opening}=K_{opening}{R}_{opening\_deviation}+\left(1-K_{opening}\right)R_{opening\_synchronization}+R_{opening\_bonus}$$

(25)

3. Simulation and Results

The control process comprises three levels: pressure control, thrust distribution, and opening closed-loop, managed by the pressure control agent, thrust distribution agent, and opening closed-loop agents, respectively. These agents operate in a cascaded manner, as illustrated in Figure 7, which shows the control architecture of the SADPS.

• Pressure Control: The pressure control agent at the top level regulates the total opening of the nozzle based on the combustion chamber pressure command.
• Thrust Distribution: The thrust distribution agent at the middle level allocates the opening commands to each valve according to the thrust command.
• Opening Closed-loop: At the bottom level, ten opening closed-loop agents control their respective motors to adjust the opening of each valve.

The transmission process of the control signal is shown in Figure 8. The upper part represents the four divert control valves, and the lower part represents the six attitude control valves. The outer layer of the flowchart represents the pressure control loop, with the combustion chamber pressure command (P_ccmd) and pressure (P_c) as inputs, and the valve opening area command (A_tsumcmd) as the output. The middle layer represents the thrust control loop, with the throat area command (A_tsumcmd), thrust commands (F_ycmd, F_zcmd, F_pitchcmd, F_yawcmd, F_rollcmd), and pressure (P_c) as inputs, and the valve openings as the output. Finally, the thrust vector synthesis of the thrusts output by each nozzle yields F_y, F_z, F_pitch, F_yaw, and F_roll. As a result, the external system only needs to input the pressure command and the five-axis thrust command into the algorithm, which then outputs the five-axis thrusts, achieving closed-loop engine pressure control and thrust distribution. Due to confidentiality principles regarding the technology, the following simulation results are presented using normalized units.

3.1. Performance of the Pressure Control Agent

The conditions are as follows: the initial combustion chamber pressure is set to 1 standard atmosphere, then 1 for 1 second, 0.22 for 1 second, and 1.33 for 1 second, continuously cycling 7 times. The control effect is shown in Figure 9.

Figure 9 shows that closed-loop pressure regulation using reinforcement learning (RL) is significantly faster and more precise than with PID control. RL adapts its strategy over time as the engine's combustion chamber volume changes, while PID control uses fixed parameters and cannot quickly adjust to the time-varying system. Excluding the initial ignition phase, Table 1 shows the cumulative deviation and average deviation between the pressure command and the actual pressure.

3.2. Performance of the Thrust Distribution Agent

The conditions are as follows: the pressure is set to 1. The divert control thrust commands(F_ycmd, F_zcmd) undergo random step changes with an upper limit of 0.8, and the attitude control thrust commands(F_pitchcmd, F_yawcmd, F_rollcmd) undergo random step changes with an upper limit of 0.1. The results are shown in Figure 10.

Figure 10 shows that the error for orbital control thrust(F_y, F_z) ranges from 2% to 5%, while the error for attitude control thrust(F_pitch, F_yaw, F_roll) ranges from 5% to 10%. Pressure fluctuations due to thrust changes are within ±2%. Excluding initial ignition and propellant depletion phases, the cumulative and step deviations between thrust command and actual thrust are shown in Table 2.

3.3. Performance of the Opening Closed-Loop Age

The control effectiveness of the nozzle opening closed-loop agent was tested by recording displacement tracking, as shown in Figure 11. The closed-loop control showed an error of less than 1% at the step stable position and demonstrated excellent response time and anti-overshoot characteristics.

Adjustment time, amplitude, and phase lag under different valve opening commands were tested, as shown in Table 3 and Table 4.

In step curve tracking, the reinforcement learning algorithm controlled overshoot effectively, keeping it below 1% in all scenarios. The full-stroke step response time was ≤ 40 ms. For continuous curve tracking, the full-stroke 20 Hz amplitude attenuation was ≤ 40%, with a phase lag of 144°.

3.4. Performance of the Muti-Agents

The trained model of the aforementioned multi-agent system was tested under the typical conditions shown in Figure 2, tracking the pressure and thrust commands. The conditions were designed as follows:

a.

Initial 6 seconds:

• Pressure:1
• Divert control thrust command (F_ycmd, F_zcmd): Random steps within the upper limit of 0.8
• Attitude control thrust (F_pitchcmd, F_yawcmd, F_rollcmd): Random steps within the upper limit of 1

b.

Next 8 seconds:

• Pressure: 0.22
• Divert control thrust command (F_ycmd, F_zcmd): Random steps within the upper limit of 0.1
• Attitude control thrust (F_pitchcmd, F_yawcmd, F_rollcmd): Random steps within the upper limit of 0.3

c.

Final 6 seconds:

• Pressure: 1.33
• Divert control thrust command (F_ycmd, F_zcmd): Random steps within the upper limit of 1.2
• Attitude control thrust (F_pitchcmd, F_yawcmd, F_rollcmd): Random steps within the upper limit of 2

The control time step was set to 2.5 ms. The control effect of the multi-agent system is shown in Figure 12.

A statistical analysis was conducted on the joint control effects of multiple intelligent agents. Under these conditions, the hierarchical multi-agent system achieved complete closed-loop control of the solid attitude and orbital control engine, including combustion chamber pressure, thrust, and valve opening.

In terms of error:

• The impact of thrust distribution on pressure closed-loop was within 5%.
• The divert control thrust error did not exceed 5%.
• The attitude control thrust error did not exceed 10%.

In terms of adjustment time:

• The pressure closed-loop adjustment time was within 300 ms.
• The thrust adjustment time was within 100 ms.
• The valve opening adjustment time was within 40 ms.

4. Discussions and Conclusions

The solid attitude and divert propulsion system (SADPS) is characterized by its time-variant, strongly coupled, highly nonlinear, and complex nature. To tackle these challenges, we integrated reinforcement learning (RL) for control system design, adaptive parameter adjustment, and continuous data-driven learning. This study provides several key findings and insights into the effective management of pressure and thrust through a unified RL control architecture.

4.1. Integrated Control Architecture and System Dynamics

Our approach involved designing a comprehensive control architecture to manage the closed-loop regulation of pressure, thrust, and valve positions. The system's dynamics, characterized by nonlinearity and time variance, show that operational duration notably influences the dynamic response of pressure regulation. The RL-based control system adapts to these dynamic changes more effectively than traditional PID control, significantly improving response speed and reducing regulation time.

4.2. Impact of Coupling Effects on Thrust Optimization

The system's design includes shared combustion chambers among multiple valves, causing fluctuations in the total exhaust area. This configuration poses challenges in optimizing thrust distribution across multiple axes, considering constraints imposed by thrust equations for various directions. The coupling effects lead to acceptable pressure fluctuations, demonstrating the system's robustness in handling complex interactions within the propulsion system.

4.3. Enhancing Control Through Reinforcement Learning and Adaptive Strategies

Integrating RL techniques with thrust deviation rewards optimized thrust control. RL enables adaptive adjustment based on real-time engine operating conditions, significantly improving pressure regulation effectiveness compared to traditional methods. The introduction of a thrust deviation reward in RL reduced errors in divert and attitude thrust, showcasing the potential of RL in handling under-constrained optimization problems inherent in thrust distribution.

4.4. Development of Realistic Training Environments and State Space Modeling

A critical aspect of our methodology was creating a highly realistic RL training environment through meticulous state space identification. This ensured accurate initial training of neural networks and subsequent fine-tuning on the actual system. Incorporating engine operating time into the RL observation space significantly improved pressure regulation time compared to fixed-parameter PID control.

4.5. Optimizing Training Efficiency and Technical Enhancements

We implemented technical optimizations such as simplifying neural network architectures, using learning rate decay strategies, and optimizing activation functions. These enhancements collectively contributed to a marked improvement in the overall performance and robustness of the control system. The RL-based multi-agent architecture outperformed traditional PID control by enhancing response speed in pressure closed-loop regulation, achieving under-constrained thrust distribution, and enabling high-speed control with minimal overshoot in valve opening closed-loop scenarios.

4.6. Limitations and Future Work

Despite the significant improvements, the proposed control method has certain limitations. One major limitation is the dependence of agent training on the accuracy of the training environment. The simulated engine model within the training environment may not fully reflect the real operating conditions of the engine, and conducting real engine training is both costly and hazardous. Additionally, the control effect of the thrust distribution agent is challenging to optimize, often failing to achieve perfect coordination between pressure and thrust. The reward function struggles to balance these two aspects effectively.

To overcome these limitations, future work will focus on developing a semi-physical simulation system capable of realistically mimicking SADPS. This system aims to provide a reusable and realistic training environment for the algorithm. For thrust distribution, we plan to introduce global constraints into the RL framework, such as minimizing propellant consumption, maximizing total impulse, ensuring safe control, minimizing valve core oscillations, and maintaining constant openings for orbital and attitude control. These constraints will help the RL agents converge more effectively, achieving better coordination between pressure and thrust.

5. Conclusions

This study showcases the effectiveness of using reinforcement learning (RL) for pressure control and multi-agent systems for thrust and valve management in solid attitude and orbital control engines. The RL-based pressure control agent significantly outperforms traditional PID control, offering faster and more precise regulation. The thrust distribution agent maintains high accuracy and stability in controlling thrust forces with minimal impact on pressure, highlighting the importance of closed-loop control for valve response times. The opening closed-loop agent excels in tracking both step and continuous commands, ensuring low overshoot and rapid response times.

Under typical operating conditions, the hierarchical multi-agent system provides comprehensive closed-loop control, achieving desired performance metrics with minimal errors and swift adjustments. These findings underscore the potential of advanced control strategies in enhancing the performance and reliability of propulsion systems. However, addressing the limitations related to training environment accuracy and thrust coordination remains a priority for future research. Developing a semi-physical simulation system and integrating global constraints into the RL framework will be critical steps toward achieving more robust and reliable control in SADPS.