Autonomous Space Systems via Reinforcement Learning: Review, Challenges, and Future Directions

1. Introduction

1.1. The Autonomy Gap in Space Systems

Space missions increasingly impose requirements that conventional model-based GNC meets only at growing cost. Landing-accuracy requirements have tightened from tens of kilometres to the 10–100 m pinpoint landing now sought for lunar and Mars missions. Small-body missions must operate in gravity fields and rotation states that are unknown at arrival, and currently spend months characterising them before close-proximity operations. Constellations of hundreds to thousands of satellites cannot be flown with single-spacecraft operations models, and on-orbit servicing and debris removal require approaching targets whose inertia and motion are only partly known. Communication delays of up to about 24 minutes one-way to Mars rule out ground-in-the-loop control for many of these tasks.

In each case a decision must be taken onboard, in real time, over a wide range of initial conditions and parameters (a wide “theatre of operations”), with an incomplete model. Trajectory optimisation (indirect, pseudospectral and convex methods) solves the nominal problem accurately but yields a single trajectory, needs a tracking controller, and has run times that are hard to bound onboard. Lyapunov feedback laws such as the Q-law are cheap and globally defined but sub-optimal and sensitive to their weights. Model predictive control (MPC) handles constraints well, but its cost grows with model fidelity and horizon, and it is only as accurate as its prediction model; multiple-horizon multiple-model formulations reduce this cost and have been applied to soft landing and hovering at irregular asteroids [1], but still require a predictive model fixed before flight. Robust and adaptive nonlinear control offers stability guarantees, but for restricted uncertainty classes and with some conservatism.

RL offers a different trade. Design effort shifts from deriving a control law to specifying a reward and a simulator, computation shifts from online optimisation to offline training, and the result is a closed-loop policy, defined over the sampled region of state space, that executes in a small and fixed number of operations. The annual volume of RL-for-space GNC publications has grown roughly fivefold since 2019. Similar approaches have been tested in robotic domains that share the partial observability, tight resource budgets and safety-critical nature of space autonomy, for example UAV path planning in which an MPC-structured policy inside the actor-critic learner handles collision avoidance under uncertainty [2], a structure that reappears in the MPC-shaped docking rewards of Section 10.3.

1.2. What Has Changed Since the Last Comprehensive Surveys

The most complete previous treatment devoted exclusively to RL in spacecraft control [3] surveyed six application fields and closed with a set of open issues: availability of high-fidelity simulators, stability and constraint guarantees, verification, hyperparameter sensitivity, training cost, the simulation-to-reality gap, and deployment onto space-grade processors. Complementary surveys addressed artificial intelligence in spacecraft GNC more broadly [4,5], machine learning in spacecraft control design [6], and AI in space missions generally [7].

Four developments since 2022 justify a new and differently organised review.

First, the field acquired hardware evidence. In 2025 the InnoCube 3U CubeSat flew the first in-orbit demonstration of a deep-RL attitude controller [8], and NASA’s Astrobee free-flyers executed the first RL-commanded manoeuvres aboard the ISS [9]. Air-bearing and robotic facilities, including the University of Luxembourg’s Zero-G Lab [10], drone-based cyber-physical emulators and reaction-wheel hardware-in-the-loop benches have become routine in the stronger papers, so sim-to-real transfer can now be assessed from data.

Second, the range of algorithms widened. Proximal policy optimisation (PPO) remains dominant, but recurrent meta-RL has become a standard tool for adaptive guidance; transformers and, more recently, selective state-space (Mamba) models have entered spacecraft guidance; model-based methods have begun to address sample efficiency; multi-agent RL has become the natural formulation for constellations and swarms; and large language models have appeared as high-level spacecraft operators.

Third, safety became a subfield. Run-time assurance, control-barrier-function safety filters, shielding and formal neural-network verification are now common in the proximity-operations literature, driven in particular by the AFRL Safe Trusted Autonomy for Responsible Space (STARS) programme [11].

Fourth, the evidence base is now large enough to compare. In 2022 most application areas had too few quantitative studies for cross-comparison; by 2026 several categories contain a dozen or more works reporting comparable metrics on comparable problems, and placing them side by side yields conclusions that no single paper states.

1.3. Scope and Exclusions

This review covers RL and closely coupled learning-based control of space systems, understood as the closed-loop determination of actuator commands and of the discrete decisions that govern them. The scope spans attitude control; orbit control, including station-keeping, phasing and formation flight; low-thrust transfer; cislunar and interplanetary guidance; entry, descent and landing; small-body proximity operations; rendezvous, docking, inspection and servicing; space robotics, including tethered, continuum and soft systems; planetary surface mobility; and mission-level autonomy where the learned decisions drive spacecraft actuation or resource state (observation scheduling, debris-removal sequencing, sensor tasking, power and data management).

We include, where they serve as the comparison baseline or the hybrid partner, non-RL learning methods that occupy the same design niche: behavioural cloning and imitation learning, guidance-and-control networks trained by supervised learning on optimal trajectories, physics-informed and Pontryagin neural networks, and neuro-evolutionary controllers. We treat classical optimal control, MPC and Lyapunov control as baselines rather than subjects.

Out of scope are: RL for satellite communication networks, routing and resource allocation where no spacecraft actuation is involved; onboard perception and image classification not embedded in a control loop; RL for launch-vehicle ascent trajectory design except where it forms part of a multi-phase control study; and terrestrial robotics except where a specific result is imported into a space application.

Table 1.1 compares this review with the existing survey literature. Its distinguishing features are the systematic extraction of quantitative results with their baselines, the uniform grading of validation fidelity, and the explicit reporting of comparisons in which learned control is outperformed.

1.4. Review Methodology

The corpus was assembled in three stages, described below. Table 1.2 gives its distribution across the eleven categories, with the number of works in each for which a numerical result, an explicit baseline and a validation level of L2 or above could be recovered. The counts are the authors’ classification; because some works span two categories and are assigned to their dominant one, they should be read as indicative.

Counts sum to slightly more than the number of distinct publications because a few works contribute to both an application category and the cross-cutting set.

Stage 1: seed and backward snowballing. The reference list of the most complete previous RL-specific survey [3] was used as the seed set, covering the foundational pre-2022 literature.

Stage 2: systematic search. Structured queries combined RL terms (reinforcement learning and its deep, meta-, multi-agent and inverse variants) with platform terms (spacecraft, satellite, lander, rover, space robot, CubeSat, constellation) and with application and method terms covering every category of Table 2.3, together with sim-to-real transfer, run-time assurance and verification. The search covered 2018–2026, with emphasis on 2022–2026, and was repeated against the principal journals (including Acta Astronautica, Advances in Space Research, Aerospace Science and Technology, Journal of Guidance, Control, and Dynamics, Journal of Spacecraft and Rockets, IEEE Transactions on Aerospace and Electronic Systems and Aerospace), the AIAA, AAS/AIAA, IAC and IEEE Aerospace conference series, and the arXiv and NTRS preprint records. The search closed in July 2026.

Stage 3: forward and backward snowballing. Reference lists of the retrieved reviews and of the most-cited primary works were mined for missed items, and forward citation tracking was applied to the seminal works of each category.

Inclusion criteria. A work was included if (i) it applies RL, or a learning method directly compared against RL, to a space-system control or control-adjacent decision problem; (ii) it reports at least one performance measure; and (iii) it is available as a peer-reviewed article, conference paper, technical report or archival preprint. Duplicate conference/journal pairs were merged and cited by the journal version.

The tables distinguish not reported (the authors did not state the quantity, a limitation of the study) from not recoverable (the quantity exists but could not be extracted from the accessible text, a limitation of this review).

For each work we recorded, where available: category; dynamical model; observation and action spaces; reward; algorithm and architecture; training budget; baselines; headline results with units; uncertainty model and Monte Carlo sample size; inference cost; and validation level. No number was imputed, rescaled or estimated, and qualitative comparisons are reported as qualitative. This constraint is deliberate, because the frequent absence of the quantities needed for comparison is itself one of the main findings.

Coverage of non-English, particularly Chinese-language, literature is partial and under-represents a large community. Some results appear only in figures behind paywalls and could not be extracted. Publication bias is likely to be strong, since negative results are rarely published, so aggregate statements that RL outperforms a baseline should be read as upper bounds.

1.5. Contributions

• A category-by-category synthesis of more than two hundred works, with the baseline and validation fidelity attached to each quantitative claim.
• A common comparison framework (Section 2.5) comprising seven metric families and a five-level validation-fidelity ladder, applied uniformly across categories.
• Comparison tables for each of eleven control categories, reporting algorithm, environment, baseline, headline figures and fidelity level.
• A cross-cutting analysis (Section 14) of where RL wins, where it loses and what it costs: the optimality-robustness trade-off, training cost, onboard inference cost and the distribution of works across the fidelity ladder.
• A critique of benchmarking and reporting practice, with concrete, adoptable reporting standards.
• A twelve-item gap analysis and a prioritised research agenda, including four benchmark problems and a staged flight-qualification pathway.

1.6. Organisation

Figure 1 shows the organisation. Section 2 establishes the formal background and the comparison framework. Section 3, Section 4, Section 5, Section 6, Section 7, Section 8, Section 9, Section 10, Section 11, Section 12 and Section 13 treat the eleven application categories. Section 14 performs the cross-cutting comparative analysis. Section 15 states the gaps, Section 16 the prospective directions, and Section 17 concludes.

2. Background, Formalism and the Comparison Framework

2.1. Markov and Partially Observed Decision Processes

Space control problems are cast as a Markov decision process (MDP) (S, A, P, R, γ) with state space S, action space A, transition kernel P(s′ | s, a), reward R(s, a) and discount factor γ ∈ [0, 1). A policy π(a | s) induces the state-value function V(s) = 𝔼[∑_t γᵗ r_t+1 | s₀ = s, π] and the action-value function Q(s, a), which the optimal policy maximises. In most spacecraft applications the agent observes a navigation estimate or raw sensor data rather than the true state, so the correct formalism is a partially observed MDP (POMDP) (S, A, O, P, Z, R, γ) with observation space O and observation model Z(o | s). This matters in practice: several of the strongest results in the field (adaptive landing under engine failure, hovering over unmapped asteroids, docking with rotating targets under delayed observations) rely on inferring hidden quantities such as unmodelled acceleration, body mass or target inertia from the history of observations, which requires a policy with memory.

Two structural features shape algorithm choice. First, episodes terminate on constraint violation as well as on success, so constraints enter through terminal penalties and episode truncation rather than as explicit feasible sets. This is a genuine weakness relative to optimal control and MPC, and the source of much of the reward-engineering burden discussed in Section 14.10. Second, the reward is almost always dense-plus-terminal: a progress-shaping term, a control-effort penalty, a constraint penalty and a large terminal bonus for reaching a success box, with remarkably little variation across landing, rendezvous, transfer and attitude control (Figure 2).

2.2. Algorithm Families in Use

Value-based methods (Q-learning, SARSA, DQN) require enumerable actions and appear where the action is naturally discrete: thruster on/off, target or task selection, manoeuvre menus and grid-based motion, as in discretised powered descent, orbit raising over weight sets, rover navigation, debris-removal target selection and hopping-rover planning.

Policy-gradient and actor-critic methods dominate continuous control. PPO is the primary algorithm in close to half of the corpus (Section 14.1), and DDPG, TD3, SAC and A2C/A3C account for most of the rest. PPO’s popularity reflects its tolerance of long episodes with delayed terminal rewards, its stability under reward terms that differ by orders of magnitude, and easy parallelisation across randomised environments. Off-policy methods (SAC, TD3) are preferred where samples are expensive and episodes short, notably in attitude control and manipulator planning.

Meta-reinforcement learning is the dominant adaptive paradigm. A recurrent policy is trained by PPO across a distribution of tasks (mass, inertia, gravity, thruster health, disturbances); at deployment the weights are frozen and the hidden state adapts the policy to the realised task without gradient updates, at the cost of one forward pass. This architecture underlies adaptive Mars and asteroid landing, hovering over unmapped bodies, image-based lunar landing, adaptive rendezvous and transformer-based rocket-landing guidance.

Model-based RL and world models remain rare but are increasing, through learned dynamics for short-horizon lookahead, latent world models and model-based representation learning; they are the natural answer to the sample-cost problem of Section 14.4 and are under-exploited.

Inverse RL and imitation have been used to recover the reward implicit in expert manoeuvre plans and rover paths; behavioural cloning serves both as a baseline and as an initialiser for RL.

Multi-agent RL, usually with centralised training and decentralised execution (commonly MAPPO), is applied to constellation phasing and scheduling, multi-inspector missions, swarm reconfiguration and pursuit-evasion.

Hierarchical RL is used where a mission decomposes into waypoint selection and low-level control, notably in multi-agent inspection and rover exploration, where it also improves sim-to-real robustness.

Tree search with learned networks (AlphaZero-style) has been applied to debris-removal sequencing and to hybrid discrete-continuous trajectory design; it retains online replanning, which purely reactive policies lack.

2.3. Function Approximation and Architectures

Fully connected networks with two to four hidden layers of 64–400 units remain the default and are what has flown. Recurrent layers are standard in meta-RL and partially observed problems, and convolutional encoders wherever the observation is an image. Transformers entered the field around 2024 for multi-phase trajectory optimisation, landing and rendezvous, and state-space (Mamba) models appeared in 2026 for memory-efficient safety-critical proximity control. The trend is towards sequence models that carry task inference in an internal state.

A recurring finding is that state representation matters more than architecture: direction-cosine-matrix attitude representations outperform quaternions because of the double-cover ambiguity; normalised relative (error) states train far better than absolute ones; a consistent reference frame matters more than which frame is chosen; and adding angular velocity to an attitude observation reduces tracking error by more than an order of magnitude (Section 3.2).

2.4. Reward Design Patterns

Four patterns account for nearly all reward functions in the surveyed corpus.

• Shaping-plus-terminal with constraint penalties, combining a tracking or velocity-field error term, a control-effort penalty, a success bonus and a violation penalty; universal in landing, rendezvous and transfer.
• Error-difference shaping, rewarding the reduction of a scalar error between steps; common in attitude control and well behaved because it is approximately potential-based and hence approximately policy-invariant.
• Multi-objective weighted sums of propellant, time and terminal error, with fixed weights or, in multi-reward PPO, several policies trained simultaneously on shared data to produce a Pareto set in one run.
• Learned or theory-derived rewards, recovered by inverse RL from expert demonstrations or derived from a Lyapunov function so that reward increase implies stability-consistent behaviour, which has been shown to shorten training.

2.5. The Comparison Framework Used in this Review

To compare heterogeneous studies we extract seven metric families and one fidelity classification.

• M1: Terminal accuracy. Position, velocity, attitude and rate errors at episode termination, with the success box that defines them.
• M2: Cost and optimality gap. Propellant mass, Δv or final mass, and where a reference optimiser is available the relative gap against an indirect, pseudospectral, collocation or convex reference.
• M3: Robustness. Monte Carlo success rate and error percentiles under a stated uncertainty model (navigation error, unmodelled acceleration, actuator failure, mass and inertia dispersion, missed thrust), distinguishing uncertainty inside the training distribution from uncertainty outside it.
• M4: Sample and training cost. Environment steps or episodes to convergence, wall-clock time and hardware.
• M5: Inference cost. Policy evaluation time, memory footprint, and the processor class on which it was measured.
• M6: Safety. Constraint-violation rate, number and type of run-time-assurance interventions, and whether any formal guarantee is claimed and over what set.
• M7: Statistical hygiene. Number of random seeds and Monte Carlo samples, and whether dispersion is reported; a property of the study rather than of the method.
• Figure conventions. In every quantitative figure in this review, blue denotes a learned (reinforcement-learned) controller, neutral dark grey a classical or optimisation-based baseline, orange a search or heuristic method, and violet an imitation-learned or hybrid method. Error bars are the dispersion reported by the source (one standard deviation unless stated). Logarithmic axes are labelled as such.

Table 2.1 defines the five levels used to grade every result.

Studies between two levels (for example, a Monte Carlo campaign over a narrow dispersion in an otherwise idealised model) are marked L0/L1. The ladder is attached to every table row to prevent the most common error in reading this literature: treating an L0 or L1 result as evidence about flight performance.

The ladder is not a strict ranking. A planar air-bearing table (L3) is three-degree-of-freedom and gravity-offloaded, and cannot represent orbital mechanics, illumination or a navigation filter, whereas an independent six-degree-of-freedom ephemeris Monte Carlo with a real estimator (L2) represents all of these and omits only hardware effects. Zero-shot transfer to an air-bearing table [12] is therefore strong evidence about actuator, friction and sensor discrepancies but none about orbital dynamics. Table 2.2 separates the two axes.

On this classification, air-bearing transfer is (E1, P2), surrogate-to-propagator evaluation (E2, P0), processor-in-the-loop guidance (E1, P1), and the InnoCube and Astrobee flights (E3, P3). The cells the corpus almost never occupies, a physical testbed driven by independently validated high-fidelity dynamics (E2, P2) and processor-in-the-loop testing with a real navigation filter (E2, P1), are those that would most change what the field can claim.

2.6. Taxonomy of Applications

Table 2.3 defines the eleven application categories. Cross-cutting themes (algorithms and architectures, safety and verification, simulators and benchmarks, sim-to-real transfer and flight demonstration, and computational deployment) are treated in Section 14 because they cut across all eleven.

3. Attitude Determination and Control (C1)

3.1. Problem Statement and Why RL Was Proposed

Attitude control is the most mature spacecraft control discipline, so a learned controller faces the highest bar. Quaternion feedback, PD/PID, LQR, sliding-mode and adaptive controllers have decades of stability theory behind them, are cheap, and are what flies. The case for RL therefore rests not on nominal accuracy but on four limitations of classical design: gains must be retuned when inertia changes (appendage deployment, propellant depletion, capture or docking); hard constraints such as keep-out cones and rate limits are awkward for linear feedback and expensive for online optimisation; actuator faults require reconfiguration logic designed in advance for each failure mode; and actuator nonlinearities (deadbands, friction, magnetorquer coupling, CMG singularities) are linearised away or handled by bespoke logic.

Formulations are uniform: the observation is an attitude-error representation with body rates (in the better studies also wheel speeds, magnetic field and a goal encoding), the action is body torque or per-actuator command, and the reward is error-difference shaping plus a pointing-box bonus and constraint penalties.

3.2. Regulation and Slew: Does RL Beat Classical Control?

On deterministic, well-modelled slew and tracking, classical control wins. The most careful head-to-head study compares PPO agents, with and without rate observation and recurrence, against a tuned quaternion-rate-feedback (QRF) controller on two reference trajectories [13]. On the smooth trajectory QRF is about eight times more accurate on the mean, and its worst case is better than the best RL case; on the harder trajectory the mean gap narrows and the RL agents achieve a smaller worst-case error than QRF (Figure 3). This is the first instance of a pattern that recurs throughout the review: RL trades nominal accuracy for better tail behaviour. The same study shows that representation matters more than algorithm: adding angular velocity to the observation reduces mean error by a factor of about 17, and direction-cosine matrices outperform quaternions.

With uncertain or varying inertia, RL wins. In a post-capture scenario with mass varying between 10 and 1000 kg, SAC and PPO agents with stacked observations were compared against a PID controller tuned at 300 kg [14]. The mean improvement is modest, but the dispersion of the final error is about fourteen times smaller, because the PID controller fails at the low-mass end. Observation stacking roughly doubles the return of both algorithms, showing that the benefit comes from online system identification from the recent history rather than from function approximation.

Under actuator faults RL again outperforms fixed-gain control. After a reaction-wheel fault, a PD controller shows large deviations and persistent oscillation, whereas PPO, A2C and TD3 agents recover, with TD3 variants giving the fastest recovery and near-zero steady-state error [15]. Fault-tolerant formulations based on Q-learning and on fixed-time optimal control reach similar conclusions, with learning used to adapt control allocation rather than to replace the controller.

Under hard pointing constraints RL is competitive with, and cheaper than, constrained optimisation. A critic-only approximate optimal controller with barrier functions for forbidden zones and rate limits has been demonstrated on a microcontroller-and-turntable hardware-in-the-loop testbed at much lower computational cost than MPC [16], and encoding a keep-out cone in the SAC observation, combined with curriculum learning, makes the constrained problem trainable [17].

Whether a learned attitude policy can carry a stability argument remains open. Two works make stability an explicit training objective: a Lyapunov-decrease condition shaped with sliding-mode concepts and placed in the reward [18] (Section 3.3), which promotes practical stability during training without certifying the trained network, and a stability-augmented RL method for attitude control in the poorly known environment of an asteroid [19], where a model-based stability argument is hardest to obtain.

3.3. Momentum Management: Wheels, CMGs and Magnetorquers

Momentum-exchange devices add an allocation and desaturation problem traditionally handled by heuristic steering laws. TD3 has been applied to angular-momentum planning for a control-moment-gyro array, achieving large slews within a minute and fine attitude hold under nonlinear disturbance [20]. For magnetically actuated CubeSats, where controllability is instantaneously rank-deficient and B-dot and projection controllers are sub-optimal, RL has been applied to detumbling and coarse pointing. For the purely magnetorquer-actuated case, a Lyapunov-decrease reward term shaped with sliding-mode concepts promotes practical stability under uncertainty, with evaluation against traditional controllers in simulation and hardware-in-the-loop experiments [18].

3.4. Flexible Structures, slosh and ATMOSPHERIC Re-Entry

RL for flexible spacecraft is thinner than its classical counterpart and is usually an add-on to a model-based inner loop. Launch-vehicle slosh is better developed, with DRL controllers shown to be stable and robust. In on-orbit refuelling, propellant slosh couples into rotation and translation; the docking studies of Section 10.3 include volume-of-fluid slosh torques in the dynamics seen by the learned controller [21].

The most demanding attitude problem attacked so far is hypersonic re-entry, where the aerodynamics vary from Mach 26.8 down to Mach 0.5 [22]. The study compares an industry-standard gain-scheduled PID with pure RL and with two hybrids (residual RL on top of the PID, and RL that outputs the PID gains), and finds that the hybrids track angle of attack better than either pure approach and are more robust to variations in mass, inertia and actuator bandwidth. A model-based method (MR.Q) was competitive without task-specific tuning. Where a good classical controller already exists, pure end-to-end RL is not the winning architecture.

3.5. The First In-Orbit Demonstration

The LeLaR experiment on the InnoCube 3U CubeSat [8] is, to the authors’ knowledge, the first in-orbit demonstration of an AI-based attitude controller and the only L4 result in this category. Its details define what “flight-ready” currently means.

The 4.2 kg CubeSat uses three reaction wheels and magnetorquers. The controller is a PPO variant with small separate sub-networks for the wheels and magnetorquers, trained for 5×10⁸ steps per sub-network in the Basilisk framework with domain randomisation of inertia, orbit and sensor noise, the noise statistics being calibrated from thermal-vacuum tests. The deployed policy occupies about 105 kB and runs in about 38 ms on a 40 MHz microcontroller inside a 1 Hz loop, with negligible extra power.

In orbit, the learned controller kept steady-state pointing errors below 1° on all axes over repeated manoeuvres, with settling times comparable to simulation. The comparison with the satellite’s own PD controller on the same platform is the most valuable part of the experiment: the PD controller met the 1° per-axis criterion in fewer than half of its manoeuvres and needed the threshold relaxed to about 2°, whereas the learned controller met it every time (Figure 4).

The policy also absorbed sim-to-real discrepancies that were not modelled in the form encountered: reaction-wheel dead time of up to about 20 s, wheel-speed measurement spikes, a residual magnetic dipole, gyroscope bias, an unresponsive wheel-speed band and the loss of one of two planned magnetorquer sets. This tolerance is attributable to broad domain randomisation and to a safety cage that limited commanded wheel acceleration and body rates.

LeLaR is not the first machine-learning experiment in orbit (ESA’s OPS-SAT flew onboard AI experiments, including attitude-control algorithms), but it is the first closed-loop AI attitude controller to command a spacecraft in orbit and to be benchmarked in flight against the vehicle’s conventional controller.

3.6. Comparative Synthesis for Attitude Control

Table 3.1 collects the quantitative record.

The only flight result in Table 3.1 also has the largest training budget and the most explicit domain randomisation, and it beats the flight PD controller on consistency across manoeuvres rather than on best-case accuracy, which is what randomisation buys. Where the plant is well known and the reference smooth, classical feedback still wins on mean error. The reliable RL advantage lies in the second moment: lower dispersion, a better worst case and graceful degradation under faults and unknown inertia. Hybrids that adapt gains or add a residual to a classical law beat both pure alternatives on the hardest plant. Inference cost is small enough that the barrier to deployment is verification and qualification, not computation.

No study reports a formal stability certificate for a trained attitude policy on a nonlinear plant with actuator dynamics; barrier-function and Lyapunov-shaped formulations [18] constrain the training objective rather than certify the network. Flexible-structure attitude control by RL is under-developed, and the empirical base for flight claims is a single 3U CubeSat.

4. Orbit Control: Station-Keeping, PHASING, Formation Flight and Constellations (C2)

4.1. Problem Statement

Orbit control maintains or reconfigures the translational state against perturbations: drag and J2 in low Earth orbit (LEO), triaxiality and luni-solar effects in geostationary orbit (GEO), and the unstable dynamics of libration-point orbits. Classical solutions are mature: impulsive targeting, Lyapunov and integer-programming station-keeping, x-axis-crossing control for near-rectilinear halo orbits (NRHOs), and convex or MPC formation reconfiguration. What changes the case is scale and autonomy: large constellations cannot be maintained by ground-planned manoeuvres at acceptable cost, and cislunar assets will need onboard station-keeping when ground contact is intermittent.

Three RL formulations appear: sequences of impulsive or finite manoeuvres (station-keeping); continuous thrust or discrete configuration choices over days to weeks (phasing and propellantless control); and multi-agent problems in which each satellite observes its neighbours (formation and swarm control).

4.2. Libration-Point and Halo-Orbit Station-Keeping

The best-instrumented study is the Sun-Earth L2 halo station-keeping work of Bonasera et al. [28], in which a PPO policy plans impulsive manoeuvre sequences, first in the CR3BP and then, by transfer learning, in a full ephemeris model. The policy succeeds from all tested initial conditions and, over a decade-long campaign, uses about 3% more Δv than a constrained-optimisation reference; in a direct head-to-head it is about 10% worse in both accuracy and cost, but it returns its answer in a single forward pass. Transfer learning from the CR3BP halves the training time for the same final reward, the cheapest training accelerator documented in the corpus.

For NRHOs the most rigorous recent benchmark is non-RL: a full-state-targeting MPC reduces perilune position and epoch errors about fourfold relative to the x-axis-crossing control used on CAPSTONE and baselined for Gateway, at about 15% higher annual Δv [29]. This is the standard against which learned NRHO station-keeping should be measured, and it is not clear that any published RL controller meets it, because the closest studies report different metrics. This benchmarking gap is cheap to close, since the baseline, the dynamical model and the metrics are all public.

4.3. Low-Earth-Orbit and Geostationary Station-Keeping

For LEO tight station-keeping with low thrust, a SAC policy has been reported to improve manoeuvre efficiency and computational load relative to linear MPC [30], although the accessible text lacks the Δv figures needed for a like-for-like comparison; formulations based on relative orbital elements and on neuro-evolution report similar robustness to perturbation modelling error. In GEO the case for learning is weaker: manoeuvres are infrequent, ground computation is available, and any optimality gap translates directly into lost propellant and lifetime.

Learning has also been applied to CubeSat orbit correction, where thrust rather than computation is the binding constraint. A J2-exploiting manoeuvre sequence [31] has been combined with a gated recurrent unit (GRU) that adjusts each burn in real time and a Kalman filter that estimates thruster pointing accuracy [32]. The J2 sequence needs roughly a quarter of the Δv of a classical sequence when the correction window is long enough, and the learned layer keeps the miss distance small under large pointing errors, where the uncorrected sequence diverges. The method is supervised rather than RL, but it reports Δv against a classical baseline, which most learned orbit-control studies do not.

4.4. Propellantless and Drag-Based Phasing

Smith et al. [33] address propellantless phasing of three satellites by differential drag with a discrete action space, comparing A2C and PPO. The agents are trained against a surrogate four orders of magnitude faster than the high-fidelity propagator and then tested on the propagator itself, an early and clean L2 protocol; A2C reaches the phasing criterion in two-thirds of the time needed by PPO. The authors conclude that propagator latency, not algorithm choice, is the binding constraint on RL for constellation control. A propellant-reducing alternative exploits the J2 drift of node and perigee when sequencing CubeSat orbit-insertion manoeuvres, saving up to 85% of the Δv of a conventional sequence when the window is long enough [31] (Section 4.3).

4.5. Formation Flight, Swarms and Collision Avoidance

Formation control by RL uses either a centralised policy or decentralised per-satellite policies: multi-agent Q-learning for reconfiguration trajectories [34], policy-gradient methods for swarm reconfiguration with collision avoidance, networks embedding SE(3) dynamics, coupled attitude-orbit formulations, and a safe-RL optimal formation controller with constraint-satisfaction guarantees [35].

A notable hybrid is the neural predictive control of Silvestrini and Lavagna [36], in which each satellite runs a local MPC while an LSTM (short horizon) and an inverse-RL network (long horizon) predict its neighbours’ trajectories. Learning predicts other agents rather than controlling the spacecraft, which sidesteps the stability question, a template that recurs in the safest parts of this literature.

Debris collision avoidance has been posed as safe RL with a variable number of objects, as multi-agent conjunction resolution, and within servicing mission planning. The multi-agent environment OrbitZoo [37] is valuable as infrastructure: it validates its propagation against Starlink ephemerides to within 0.2% mean absolute percentage error and demonstrates avoidance policies that drive collision probability below 10⁻⁶ with small semi-major-axis changes.

4.6. Comparative Synthesis for Orbit Control

As Table 4.1 shows, orbit control is the category in which the optimality gap is best quantified and smallest: a few per cent over a decade of station-keeping, and about 10% head-to-head. The argument for RL here is therefore weak on optimality and strong on autonomy, since the policy runs onboard without a solver. The propellantless phasing study shows most clearly that simulator throughput, not algorithm design, is often the binding constraint, and its surrogate-then-verify protocol should be standard practice.

The NRHO benchmark disconnect; the absence of RL station-keeping results in Δv per year, the unit operators use; little work on GEO electric-propulsion station-keeping, where the economic stakes are highest; and little on the interaction between station-keeping and collision avoidance, which in dense constellations share the same actuator.

5. Low-Thrust Transfer and Many-Revolution Orbit Raising (C3)

5.1. Problem Statement

Electric propulsion makes GTO-to-GEO orbit raising and many science transfers efficient at the cost of hundreds of revolutions and months of near-continuous thrust. The long, oscillatory optimal-control problem defeats full-fidelity direct transcription and makes indirect methods sensitive to initial costates. The pragmatic classical answer is the Q-law, a Lyapunov feedback law whose weights set the priority of each orbital element; it is globally defined, cheap and robust but sub-optimal, and its performance depends strongly on weights usually tuned offline by heuristic global optimisation.

This structure suits RL well, and the field converged on a two-level scheme in which a learned policy sets the parameters of the classical law. In our assessment this is the most successful design pattern in the RL-for-space literature: learning is confined to a low-dimensional, bounded and interpretable decision, while the classical law keeps its global structure and its Lyapunov stability argument.

5.2. RL-Parameterised Q-Laws

Holt et al. [38,39] learn an actor that maps osculating elements to state-dependent Q-law weights, with an extreme-learning-machine critic in the earlier work and PPO in the later one, and report better optimality and robustness than fixed or time-scheduled weights tuned by particle-swarm optimisation. Because the low-level law is a Lyapunov controller for any admissible weight vector, the closed loop inherits its stability whatever the network outputs, provided the output is constrained to the admissible set [39]. This is the cleanest safety argument in the corpus, and it exists precisely because learning was confined.

Arora and Dutta [40] decompose the transfer into per-revolution sub-problems and use DQN to select objective weights from a discrete set, rewarding transfer-time reduction, while Zaidi et al. [41] use a cascade of DRL agents for successive phases of the transfer.

5.3. What the Numbers Say, and What Is Missing

The design pattern is convincing, but the quantitative record is weak (Table 5.1). Comparisons are mostly against Lyapunov and heuristic baselines rather than converged optimal solutions, and transfer time and propellant are reported in problem-specific units that cannot be compared across studies. There is no shared GTO-to-GEO benchmark with reference solutions, although one would be easy to define and would make a decade of work comparable (Section 16.1).

The generic multi-phase study of Jain et al. [42] indicates what is achievable when a converged reference exists: a PPO agent with a gated transformer-XL policy comes within 1–8% of the cost of LQR and collocation references, and inserts close to the target orbit in a multi-phase rocket ascent. The ascent figures are terminal-state errors, not optimality gaps. The transferable figure of merit is the cost ratio, a 1–8% penalty against a converged optimiser for a single-forward-pass closed-loop policy, and RL-Q-law studies should report against it.

6. Cislunar and Multi-Body Guidance (C4)

6.1. Problem Statement

The circular restricted three-body problem and its ephemeris refinements support families of periodic and quasi-periodic orbits (Lyapunov, halo, near-rectilinear halo, distant retrograde) that are central to lunar infrastructure planning. Transfers between them exploit invariant manifolds, are extremely sensitive to initial conditions, and have solution spaces riddled with local minima. Onboard autonomy is attractive because ground-based redesign of a low-thrust transfer is slow and because the dynamical sensitivity means that navigation error propagates rapidly.

6.2. Transfer Guidance Between Libration-Point Orbits

The most quantitatively complete study [43] trains PPO for fuel-optimal low-thrust transfer between planar Lyapunov orbits and compares it with the GPOPS-II pseudospectral optimiser. The learned policy reaches the target faster but uses two to five times more propellant, and it terminates several orders of magnitude further from the target state, though within mission tolerance. Under Monte Carlo navigation errors ten times larger than nominal it still converges with almost unchanged propellant use, and inference takes milliseconds.

This is the clearest illustration in the corpus of the optimality-robustness trade. An open-loop solution five times cheaper in propellant needs a tracking controller and replanning to be usable, whereas a closed-loop policy five times more expensive converges under large navigation errors within milliseconds. Reporting only one of these numbers, as much of the literature does, misrepresents the method in either direction.

6.3. Multi-Objective Formulations

Sullivan et al. [44] address the propellant-time trade with multi-reward PPO, training four policies with different mass penalties on shared transitions for halo-to-halo transfers near Earth-Moon L2 for a 180 kg smallsat. The policies recover transfers of distinct geometry and cost, and when used as initial guesses for sequential quadratic programming they converge to nearby locally optimal transfers. Generating a Pareto set of feasible transfers in one training run is arguably more useful operationally than any single optimal trajectory, because initial-guess generation is the bottleneck of multi-body trajectory design.

Comparing three reward correspondences for low-thrust approach to a periodic orbit, Sullivan and Bosanac [45] find that rewarding proximity to the nearest reference state minimises tracking error, while penalising thrust-direction changes yields more implementable manoeuvres at some cost in accuracy. LaFarge et al. [46] build the corresponding closed-loop guidance with a KD-tree for fast nearest-state lookup, and Federici et al. [47] apply multi-objective RL with a delayed terminal reward combining distance and fuel.

6.4. Learning from Experts, and Attention

Elliott et al. [48] use maximum-margin apprenticeship learning to recover the reward implicit in LQR-designed station-keeping and rendezvous manoeuvres near an L2 NRHO, with the aim of capturing the tacit criteria of flight-dynamics engineers. The contribution is methodological rather than a performance gain: the reward function is where mission-specific engineering judgement enters an RL design, and inverse RL is the only systematic way the field has to elicit it.

Attention-augmented RL has been applied to low-thrust cislunar trajectory optimisation [49], continuing the trend towards sequence models, and meta-RL to cislunar proximity operations.

Table 5.1 and Table 6.1 contain the field’s best-documented optimality gaps, large in propellant (two- to fivefold in cislunar transfer) but small in insertion accuracy relative to mission tolerances. These categories also contain the field’s best structural idea, confining learning to the parameters of a classical law, which provides a stability argument at no extra cost, and its most operationally useful capability, the generation of Pareto sets and initial guesses.

7. Interplanetary Trajectory Design and Robust Guidance (C5)

7.1. Problem Statement

Interplanetary low-thrust design must handle thrust arcs lasting months, dynamical and navigation uncertainty, execution errors and, operationally most important, missed-thrust events (MTEs) that remove thrust for days to weeks. Classical practice designs a nominal optimal trajectory and reserves propellant margin for the worst case. A closed-loop policy trained across the disturbance distribution, including MTEs, promises to turn worst-case margin into expected-case performance.

7.2. Robust Design Under Structured and Unstructured Uncertainty

Zavoli and Federici [50] formulate the Earth-Mars low-thrust transfer as a Sims-Flanagan sequence of impulsive decisions and train a PPO policy to minimise propellant, with penalties on Δv limits and terminal error, under unmodelled accelerations, parameter uncertainty, observation noise, execution errors and a randomly timed MTE. This is the reference formulation for the category, and its central claim, that a policy trained across the disturbance distribution achieves a higher expected final mass than a nominal-optimal trajectory flown with margin, is the right way to frame the value of RL here.

Federici and Zavoli [51] extend this to recurrent meta-RL, improving robustness under several simultaneous uncertainty sources, and the approach has been applied to gravity-assist trajectories [52]. One of the earliest PPO studies in this area [53] candidly states that its results were not yet comparable with a dedicated trajectory-optimisation tool, a useful marker of how far the category has since moved.

7.3. Guidance-and-Control Networks: Imitation Versus Reinforcement

The most methodologically valuable study in this category, and one of the most valuable in the review, compares behavioural cloning (BC) with RL for guidance-and-control networks (G&CNETs) on four problems spanning low and high control authority: a GTOC-11 transfer, an Earth-Mars transfer, and landings on Psyche and on comet 67P [54]. The network class, problems and evaluation are held fixed; only the training signal changes.

Nominally, cloning is more optimal, as expected of a method that regresses onto optimal trajectories, although both methods stay within a few per cent of the reference.

Under perturbation the picture reverses. The study evaluates four distinct conditions, each in a 200-sample Monte Carlo campaign: initial-condition error, zero-order-hold control discretisation, orbit-determination error and control-execution error. Averaged over the sixteen problem-condition cells, full-state convergence is about 49% for BC and 96% for RL; BC falls below 25% in half of the cells, whereas RL never falls below 74%.

The failures depend strongly on the type of perturbation. Under initial-condition error alone the two methods are identical on two of the four problems, whereas BC collapses, often to near zero, under execution error, navigation error or discretisation. Behavioural cloning therefore degrades under small departures accumulated along the trajectory far more than under a single displacement at the start. A policy displaced once may return close to the demonstrated states, but a policy perturbed at every step drifts away from them into regions where it has no data; RL, which visits off-nominal states during training, degrades gracefully. The weakest RL case, under navigation error on the high-authority 67P problem, is consistent with this mechanism, because there observation error maps most directly into control error.

Two negative optimality residuals, where a learned network slightly outperforms the reference, should not be read as learning beating optimal control: they arise because the indirect solver converged to a local optimum for those instances. RL is also not uniformly more sample-hungry than cloning; it needed fewer samples in three of the four cases, because cloning must first generate the optimal-trajectory dataset.

A related line trains the same networks by direct optimisation against the optimal-control problem, for example through neural ordinary differential equations, and has established conditions under which networks trained on optimal trajectories inherit the optimality structure of the underlying problem. Physics-informed and Pontryagin neural networks instead embed the necessary conditions in the loss. These methods are not RL, but they are the right comparison class for any RL result that claims near-optimality and should be used as baselines far more often.

7.4. Solar Sailing and Propellantless Interplanetary Transfer

Solar sails have no propellant to optimise but pose an attitude-constrained, weakly actuated, long-horizon control problem that suits policy learning. Robust sail trajectories under uncertainty have been designed with PPO and related methods [55,56], and neural and neuro-fuzzy sail controllers complete a small but coherent sub-literature.

7.5. Hybrid Search-and-Learning for Mission-Scale Combinatorics

For mission-scale combinatorial problems of the GTOC class, which combine body selection with continuous transfer parameters, the successful architecture combines tree search with learned policy and value networks. AstroZero [57], derived from AlphaZero, handles hybrid discrete-continuous actions without discretisation and achieves competitive, in places state-of-the-art, GTOC XI results on a single workstation, showing that learning can contribute search guidance rather than control while preserving the completeness of search.

7.6. Comparative Synthesis for Interplanetary Guidance

Table 7.1 shows that this category holds the strongest evidence for the field’s central claim, and that the evidence comes from a controlled experiment: with network, problem and evaluation fixed and only the training signal changed, imitation buys nominal optimality and reinforcement buys robustness. Future claims that RL is robust should be tested against this template.

8. Entry, Descent and Landing (C6)

8.1. Problem Statement

Powered descent and landing is where RL for spacecraft control began, and it remains the most populated category. Three features suit learning: the handover state lies anywhere in a deployment ellipse that may be tens of kilometres across, so the controller must be globally defined; the dominant uncertainties (mass, environmental accelerations, atmosphere, thruster performance and failures, navigation bias) are exactly those a policy can be trained across; and the terminal requirement is a box that maps naturally onto a terminal reward.

The classical alternatives are strong: convex programming for fuel-optimal powered descent with guaranteed convergence, pseudospectral open-loop optimisation, and analytic feedback laws such as energy-optimal DR/DV guidance and zero-effort-miss/zero-effort-velocity (ZEM/ZEV) guidance.

8.2. From 3-DOF to 6-DOF Integrated Guidance and Control

Gaudet and Furfaro [58] established the template for adaptive, fuel-efficient Mars landing by RL, and Gaudet et al. [59] extended it to integrated guidance and control, mapping navigation estimates directly to thrust commands without a separately computed reference trajectory.

A six-degree-of-freedom extension maps the lander state to individual engine commands [60]. Compared with an open-loop GPOPS solution and the DR/DV law, the learned policy is less fuel-efficient than the optimum but more accurate than both baselines across the deployment ellipse, a pattern that later work has repeatedly reproduced. Policy evaluation takes under 1 ms on a desktop processor, or roughly 23 ms when scaled to a 100 MHz flight processor.

Gaudet et al. [61] introduced the recurrent meta-RL formulation that has since become standard, with Mars and asteroid landing scenarios that randomise lander mass, environmental disturbances, body rotation, radiation pressure and asteroid mass over a tenfold range. The meta-RL policy outperforms DR/DV and adapts to engine failure, larger mass variation and higher navigation bias, under which the fixed law degrades.

Federici and Furfaro [62] provide the controlled comparison the category needed: on the same landing benchmark and protocol, recurrent meta-RL outperforms standard RL with feedforward networks in optimality, constraint handling and robustness to unmodelled dynamics, navigation error and engine malfunction, establishing that recurrence, rather than capacity, delivers adaptation.

8.3. Image-Based and Hazard-Aware Landing

Image-based meta-RL maps simulated descent-camera images and ranging data directly to thruster commands for lunar pinpoint landing, with metre-level terminal errors under parameter uncertainty and actuator failure [63]. Its extension to safe-site selection combines a seeker guidance law with convolutional hazard segmentation trained on synthetic terrain imagery; it is robust to seeker and actuator lag and degradation and supports multiple divert manoeuvres [64].

This is where the structural advantage of RL is clearest, because a sensor-to-actuator policy has no analytic counterpart: the classical architecture needs terrain-relative navigation, hazard mapping, site selection and retargeting as separately validated subsystems. JAXA’s SLIM mission flew a camera-in-the-loop precision lunar landing with non-learned guidance, showing that the architectural precondition for such policies can be flight-qualified.

8.4. Hybrid and Constrained Formulations

Furfaro et al. [65] learn the gains of the ZEM/ZEV law as functions of state while keeping its analytic structure, obtaining quasi-fuel-optimal closed-loop trajectories that satisfy thrust constraints. This is the landing-domain instance of the pattern of Section 5: learn the parameters and keep the law.

Jiang et al. [66] integrate entry guidance with powered descent by combining RL with a pseudospectral method. Wilson and Riccardi [67] reduce training time by discretising the action space, compare hyperparameter-selection criteria, and report the important negative result that policies are very poor when the state representation and the reward are not strongly coupled. Ciabatti et al. [68] use transfer learning to reduce retraining across landing scenarios.

RL-enhanced MPC, in which learning adjusts the MPC formulation rather than replacing it, retains constraint handling and feasibility while improving adaptivity [69].

8.5. Atmospheric Entry and Propulsive Recovery of Launch Vehicles

Reusable launch-vehicle recovery resembles planetary powered descent with added aerodynamics and severe actuator constraints. DRL has been applied to propulsive first-stage landing [70], to genetic-algorithm-augmented landing guidance, and to controllers synthesised by neural architecture search, whose landing accuracy under untrained parameter deviations and wind is reported to match that under trained conditions [71]. Transformer-based meta-RL is the current architectural state of the art for atmospheric rocket landing [72].

8.6. Atmospheric Entry Guidance and Aerocapture

Hypersonic entry guidance, the bank-angle-modulated phase that sets where the vehicle arrives in the deployment ellipse, has received less RL attention than powered descent, even though entry dispersion dominates landing error; the exceptions are integrated entry-descent formulations, reinforcement meta-learning for hypersonic glider approach guidance, and the re-entry attitude work of Section 3.4. For aerocapture, our searches found no peer-reviewed RL guidance study, a statement limited by the search protocol rather than a proof of absence. The problem is a natural fit, with a single scalar control, a large and poorly modelled density uncertainty and a terminal box in apoapsis and periapsis, and classical predictor-corrector guidance supplies both the baseline and the reference solution. It is one of the clearest unexploited opportunities in the corpus.

8.7. Comparative Synthesis for EDL

EDL has the most consistent qualitative finding in the corpus (Table 8.1), reported by independent groups over a decade: RL trades a few per cent of fuel optimality for large gains in terminal accuracy across the dispersion, and for adaptation to failures that fixed laws cannot handle. It also has the strongest structural argument for RL, image-to-thrust policies with no classical counterpart, and one of the weakest fidelity records: nearly all of it is L1, with no hardware-in-the-loop or flight demonstration of RL landing guidance, a striking gap for the field’s oldest application.

No common powered-descent benchmark with reference solutions, although the Gaudet formulation is a de facto standard that could be published as one; no study of learned guidance with a realistic navigation filter rather than additive noise; and no comparison against modern convex powered-descent guidance, the actual flight state of the art and a far stronger baseline than DR/DV.

9. Small-Body Proximity Operations (C7)

9.1. Problem Statement

Asteroids and comets are where the model-free argument for RL is strongest, because the model does not exist before arrival. The gravity field is irregular and poorly known, rotation may be complex, solar radiation pressure is comparable to gravity at operating distances, and the shape model on which classical guidance relies takes months of proximity operations to build.

The RL proposition is to train across a distribution of plausible bodies so that, on arrival, the policy adapts online from sensor data without a shape model. The payoff would be months of mission time and much of the ground-operations budget, not a few per cent of propellant.

9.2. Body-Fixed Hovering and Terminal Guidance

Gaudet et al. [73] achieve six-degree-of-freedom hovering over unmapped asteroids with reinforcement meta-learning, mapping LIDAR altimetry and estimated state to thrust without a shape model, and extend the framework to terminal adaptive guidance for close-proximity operations [74]. Both papers argue that such policies could operate from a large deployment region in unknown dynamics without the multi-month characterisation campaign.

The asteroid scenario used in this work has become the de facto test case, randomising body mass over a factor of ten together with rotation rate, radiation pressure and lander mass. A distribution this wide requires a recurrent policy: a feedforward policy cannot distinguish a light, fast-rotating body from a heavy, slow one, whereas a recurrent policy infers it from the response to its own thrust. Attitude stabilisation in the same environment has been treated with stability-augmented RL [19] (Section 3.2).

9.3. Reconnaissance, Approach and Impactors

Reconnaissance guidance at small near-Earth asteroids has been formulated with rewards for imaging quality and coverage under safety and illumination constraints [75], and robust approach guidance addresses uncertainty in the body’s gravitational parameter and ephemeris [76]. Chen et al. [77] combine meta-learning with decision-time search, the hybridisation that also succeeds in mission planning (Section 13).

For planetary-defence impactors, image-based meta-RL maps camera images to control in the terminal phase, where light-time precludes ground control, including target discrimination in a binary system [78]. The same line has reached optical-bench validation against real imaging hardware [79], one of the few image-based guidance results with hardware evidence (L3).

RL guidance for autonomous shape reconstruction of an uncooperative target [80] closes the loop between control and mapping: the objective is information gain, so the controller performs experiment design rather than tracking.

9.4. Science Operations and Mapping

Chan and Agha-mohammadi [81] formulate imaging and mapping of small bodies as a POMDP covering spacecraft state, memory, camera and downlink modes and illumination, with rewards for image quality and penalties on thruster use; REINFORCE outperforms three benchmark algorithms on real asteroid models.

The architecture matters more than the score. Instead of letting the policy actuate the spacecraft, the authors embed it in the MEXEC flight planning and execution system: MEXEC supplies the trajectory, and the agent only selects imaging times along it. This turns an unverifiable control policy into a verifiable scheduling recommendation, the most readily deployable pattern in the review. Herrmann and Schaub [82] extend this line.

9.5. Comparative Synthesis for Small-Body Operations

Table 9.1 shows that the qualitative case for RL is strongest here and the quantitative record thinnest, because the missing model that motivates the approach also removes the reference solution against which results could be scored. The main contributions of the category are architectural: meta-RL without a shape model, optical-bench validation and the MEXEC decoupling pattern. Future work should report against a model-informed upper bound, the performance of a classical law given the true body parameters. Multiple-horizon multiple-model predictive control, developed for soft landing on irregular asteroids [1] and, with overlapping horizons, for finite-time hovering [83], is a natural candidate. It is also the natural model-based comparator for recurrent meta-RL, because its bank of prediction models performs explicitly the model selection that the recurrent policy performs implicitly.

10. Rendezvous, Docking, Proximity Operations and Inspection (C8)

10.1. Problem Statement

Rendezvous and proximity operations (RPO) is the category with the most hardware evidence, the most safety machinery and the most adversarial variants. On-orbit servicing, refuelling, inspection, assembly and debris removal all require approaching a target that may be non-cooperative, tumbling, of uncertain inertia or evasive, under severe and heterogeneous constraints: approach corridors, line-of-sight cones, keep-out zones, range-dependent velocity limits, plume and passive-safety requirements, illumination and propellant budgets.

Classical RPO guidance (Clohessy-Wiltshire-Hill targeting, glideslope and V-bar/R-bar approaches, convex and MPC formulations, artificial potential functions) handles these constraints explicitly, and RL must match it. This is why, more than any other category, RPO builds safety architectures around the learned policy instead of trying to learn safety.

10.2. Guidance-Level Learning with Classical Inner Loops

Hovell and Ulrich [12] introduced the widely adopted deep guidance concept, in which the policy outputs a desired velocity that a conventional inner-loop controller tracks. Learning is placed where it adds value, in high-level guidance, while classical control provides robustness to model discrepancy.

Policies trained on a 10 kg simulated platform were deployed without fine-tuning on Carleton’s SPOT air-bearing facility, with 16.9 kg platforms, on-off thrusters, sensor noise, friction and table slope, and reproduced all simulated scenarios, including approach to a rotating target. This zero-shot transfer is the empirical foundation for guidance-level learning.

Federici et al. [84] address terminal rendezvous with a non-cooperative target under corridor and visibility constraints and compare PPO with behavioural cloning, anticipating the G&CNET result of Section 7.3: RL is more robust but slower to train and more sensitive to hyperparameters, and the authors recommend initialising RL by cloning.

10.3. Six-Degree-of-Freedom Docking

For six-degree-of-freedom docking with rotating targets, Oestreich et al. [85] train PPO against an offline LQR reference acceleration that imposes a hard time constraint on the trajectory. Hu et al. [86] formulate constrained 6-DOF proximity control in dual quaternions, with an actor-critic value approximation initialised from a PD-like policy, and, unusually for this literature, prove closed-loop asymptotic stability.

Recent work has moved to physics-informed transformers for 6-DOF rendezvous, meta-RL with transformers under delayed observations, multi-agent cooperative pose control and imitation learning from expert demonstrations, while Wang et al. [87] pursue stability-certified learned guidance.

The most quantitatively complete docking study isolates the effect of formulation. Arora et al. [88] train four PPO configurations for 6-DOF docking of JAXA’s Int-Ball2 free-flyer inside the ISS and evaluate them over more than 900 attempts. The best achieves sub-centimetre accuracy and over 99% stable docking; the worst succeeds in 0.1% of attempts. The same algorithm on the same problem thus spans three orders of magnitude in success rate depending on observation content, reward and action parameterisation (Figure 5), and papers that report only their best configuration hide the variance a practitioner will meet.

Propellant slosh, unavoidable in on-orbit refuelling, is the docking disturbance that the orbital RL literature has most neglected. Ramezani et al. [90] address it with an MPC-structured reward: a convex MPC computes a collision-free reference from slosh-free dynamics, and the agent, acting on the full plant with volume-of-fluid slosh dynamics over a wide range of fill fractions, is rewarded for tracking it, so that MPC shapes the learning signal without issuing commands. The MPC-shaped reward raises docking success by 14–15 points for both PPO and SAC while reducing position error and control effort, whereas the two algorithms differ by only three to four points under either reward; as in the Int-Ball2 study, reward design matters more than the choice of algorithm. A further study [21] carries the framework to fuel-aware refuelling with collision avoidance and reports it ahead of conventional RL and MPC-based methods in success, precision and control effort.

Both studies include physical-testbed evidence from the University of Luxembourg’s Zero-G Lab [10], which provides air-bearing platforms on a precision-flat floor, robotic arms and motion capture. There, a PPO policy trained with the MPC-reference reward kept position and attitude errors small after repeated manual disturbances, whereas the same policy trained without the MPC term settled at errors about three times larger and had not converged after 20,000 episodes [91]; the slosh-docking study repeats the comparison on the same platforms without propellant [21]. These are single-trial planar demonstrations without fluid, but they place MPC-shaped rewards among the few RPO formulations evaluated at L3.

10.4. Inspection, and the Safety Architecture Around it

Spacecraft inspection has become the canonical benchmark for safe RL in space, largely through the STARS programme [11], which defines planar and 3-D docking and single- and multi-agent inspection challenge problems on Clohessy-Wiltshire-Hill dynamics in a Gym-compatible environment, and enforces fourteen safety constraints covering collision avoidance, range-dependent speed limits, keep-in zones, passive safety, sensor keep-out, and thermal, power and propellant limits.

Two run-time assurance (RTA) architectures are used: switching (simplex) RTA, which hands control to a verified backup controller, and optimisation-based RTA implemented as a control-barrier-function quadratic program. The programme’s ablations produced three results of general importance:

• RTA during training helps only if intervention is penalised in the reward; otherwise the agent learns to rely on the filter and is unsafe without it.
• Action-space discretisation should follow the task. Discrete thrust levels are best for inspection, where adjustments are sparse; continuous actions are best for docking, where fine terminal control dominates.
• Reference-frame consistency matters more than reference-frame choice, with Hill’s frame producing the best overall behaviour.

STARS also supplies the field’s best deployment figures: neural controllers executed in under 2 ms on commercial, radiation-tolerant and radiation-hardened processors, supporting 10 Hz loops with the RTA optimisation also in the loop. Formal verification established liveness and safety properties for 2-D docking, with the explicit caveat that the method does not scale to larger observation and action spaces or more complex dynamics.

Dunlap et al. [92] address scalability with an observation of fixed dimension, independent of the number of agents. A policy trained with three agents succeeds in every episode with two or more, covers almost the entire target surface with small Δv and finishes close to the theoretical minimum time, while a single agent succeeds in about 90% of episodes. Related work covers inspection with discrete control barrier functions, RTA for 6-DOF inspection and docking, and multi-objective RL that saves fuel without losing coverage.

Sim-to-real evidence for inspection comes from the LINCS laboratory, where motion-captured quadrotors emulate satellites under two-body and J2 dynamics. Lei et al. [93] run one hierarchical PPO inspection policy at three fidelity levels: all targets are reached at every level, but on the physical testbed mission time increases by about a third and distance travelled by about a fifth relative to simulation (Figure 6). This efficiency penalty with no loss of task completion is the best-documented quantification of the sim-to-real gap for an RL space controller.

Inspection with RL and RTA has also been demonstrated on unmanned aerial surrogates, and Georgia Tech’s ASTROS five-degree-of-freedom testbed has validated attitude RTA that maintains sensor keep-out cones under human joystick input.

10.5. Free-Flyers in Space: The First Flight Results

RPO also contains the corpus’s only orbital demonstrations of RL control of a free-flyer. In the APIARY experiment [9], a PPO policy trained in NVIDIA Isaac Lab commanded a NASA Astrobee aboard the ISS on 27 May 2025, executing eight manoeuvres, including undocking, rotations and a redocking that succeeded on the second attempt.

The companion paper [89] documents the methodology: a small actor-critic network trained across 10,000 parallel environments under a 22-level curriculum over position, orientation and mass. Without the curriculum success is below 10%; with it, success exceeds 90%, and wider mass randomisation adds a few further points. In orbit, the undocking manoeuvre undershot its target but stayed within the accepted margin.

Together with LeLaR (Section 3.5), these results define the current recipe for flying an RL controller: train entirely in simulation, randomise aggressively over parameters that cannot be measured, use a curriculum, keep the network small, cage actuator commands and body rates, and expect hardware discrepancies that were not modelled.

10.6. Adversarial and Game-Theoretic Proximity Operations

When the target actively evades, the problem becomes a differential game. Deep RL has been applied to multi-player orbital pursuit-evasion, pursuit-evasion in elliptical orbits, interception of randomly manoeuvring targets, diverse strategy generation, delayed-reaction games and multi-satellite interception. MIT Lincoln Laboratory’s Kerbal Space Program Differential Game challenge (KSPDG) has provided a public benchmark with an annual competition since 2024, one of very few shared benchmarks in the field, and has also become the venue for evaluating large language models as spacecraft operators (Section 13.5). Posadas-Nava et al. [94] combine the adversarial setting with input-constrained control barrier functions whose class-K parameters are learned, comparing LSTM, GRU and Mamba backbones under PPO and SAC.

10.7. Comparative Synthesis for RPO

Table 10.1 shows that RPO has the most complete evidence chain, from L0 to L4, and that the pattern along it is consistent. Both flight demonstrations used simulation-trained policies with heavy randomisation, small networks and actuator caging; the best ground evidence puts the reality gap at an efficiency penalty of about a third with no loss of task completion; the best formulation study shows a three-order-of-magnitude spread from observation and reward design alone; and the most mature safety work wraps the policy in run-time assurance rather than certifying it, because formal verification currently reaches only 2-D docking.

11. Space Robotics and Manipulation (C9)

11.1. Problem Statement

A manipulator on a free-floating base is dynamically coupled to it: every joint torque disturbs the base attitude, and the generalised Jacobian depends on the inertia distribution, which changes when a target is grasped. Classical methods (generalised-Jacobian kinematics, reaction-null-space and coupled attitude-manipulator control) are mature but need accurate inertia knowledge, which is precisely what is lost when the target’s mass properties are unknown. Contact dynamics, joint limits, obstacle avoidance around tumbling targets and noisy vision-based pose estimates strengthen the case for learning.

The category divides into rigid manipulators (the bulk of the literature), capture and contact, and compliant systems (tether-nets, continuum and soft robots) where the configuration space is effectively infinite-dimensional and model-based control is hardest.

11.2. Free-Floating Manipulators: Path Planning and Trajectory Generation

Al Ali and Zhu [97] report the most complete quantitative study, using DDPG on a six-degree-of-freedom free-floating manipulator without base attitude control. Adding joint-velocity constraints halves the joint-velocity cost at the price of moderately larger end-effector errors, an explicit statement of the accuracy-actuation trade that most papers leave implicit. With observation noise, an agent trained with noise completes the task, while an agent trained without noise fails outright.

This finding recurs throughout the review: robustness comes from exposure to the disturbance during training, and a policy that excels with clean observations can fail completely with realistic ones, the manipulator analogue of the behavioural-cloning failure of Section 7.3.

D’Ambrosio et al. [98] apply PPO to a seven-degree-of-freedom redundant manipulator during motion synchronisation with a tumbling target, delegating redundancy resolution to the policy; validation covers 500 test episodes but reports error histories rather than summary statistics. Related work covers continuous-pose trajectory planning, dual-arm planning, simulation-efficient training and image-based capture under unknown dynamics.

11.3. Capture of Tumbling and Non-Cooperative Targets

Capture of non-cooperative tumbling targets by a free-floating robot has been addressed with deep RL trajectory planning [99] and with vision-based guidance. Combining deep RL with tactile feedback improves the soft-capture phase of debris removal [100], one of very few studies to use contact sensing, the modality that classical capture control handles least well.

Continuum and compliant systems form a small but distinct sub-literature: RL tracking control of a continuum capture robot, robust policies for deploying and closing tether-nets whose dynamics resist closed-form modelling [101], and multi-debris capture by a tethered net robot. The terrestrial soft- and continuum-robotics RL literature transfers directly but is little used by the space community; on the evidence of this review, this is where RL has the largest untapped advantage, because the classical alternative is weakest there.

11.4. Comparative Synthesis for Space Robotics

Table 11.1 summarises the category, whose reporting is weaker than in RPO or EDL: several strong papers give trajectory plots instead of error statistics over a test set, and base-attitude disturbance, the defining quantity of free-floating manipulation, is rarely reported as a metric. Contact and post-capture dynamics, where inertia changes discontinuously and adaptation would be most valuable, are almost untouched, and no free-floating-manipulator policy has been validated on orbital hardware, although the Astrobee results of Section 10.5 show that the platform now exists.

12. Planetary Surface Mobility (C10)

12.1. Problem Statement

Surface mobility divides by timescale and sensing horizon. Long-range planning uses orbital imagery and is done on the ground with round-trip latencies; short-range navigation uses onboard cameras and LIDAR in partially unknown terrain; and on low-gravity bodies wheels become ineffective, so hopping or legged locomotion, with uncontrolled ballistic phases, is the practical mode.

12.2. Long-Range Planning from Orbital Imagery

Pflueger et al. [102] use inverse RL with soft value-iteration networks to learn, from a single expert path, the reward implicit in rover route choices over orbital imagery and elevation data. The learned value function gives paths from any location on the map and can also score landing sites, but it must be retrained when the map changes, so it reduces rather than removes ground involvement.

12.3. Short-Range Navigation and Exploration

Deep Q-learning has been used for cooperative detection by two LIDAR- and camera-equipped vehicles, with rewards for coverage, collision avoidance and communication range [103]. For large map-free environments, a goal-driven hierarchical DQN with hindsight replay improves success rate and path length over a non-goal-driven hierarchy [104]. Hierarchical decomposition also improves sim-to-real robustness in multi-agent inspection (Section 10.4), which suggests a structural rather than incidental benefit.

12.4. Low-Gravity Mobility: Hopping and Legged Locomotion

Hockman and Pavone [105] plan hopping-rover motion on small bodies under gravity, control and position uncertainty with off-policy least-squares policy iteration, so that the same simulated data can re-derive the policy when scientific priorities, and hence rewards, change in flight. Jiang et al. [106] use DQN on a height-level terrain map with rolling, walking and hopping moves of different energy cost, finding minimum-energy paths that are robust to small terrain changes but do not generalise beyond the training map.

Recent work extends this to legged systems, with a single RL policy for walking, jumping and in-flight attitude control in low gravity [107]. Because the policy simply observes contact state, it handles the hybrid stance-flight dynamics that classical control must decompose into separate problems.

12.5. Operational Context

In 2026 NASA’s Perseverance rover completed its first AI-planned drive on Mars, moving route planning onboard. The system is not an RL controller and should not be cited as one, but it shows institutional willingness to delegate route planning to onboard autonomy, which lowers the barrier for learned planners. Table 12.1 summarises the category.

13. Mission-Level Autonomy and Resource Management (C11)

13.1. Problem Statement

Above guidance and control sit decisions about what the spacecraft does: which targets to image and when, which debris to remove in which order, where to point sensors, and how to allocate energy and storage. These problems are combinatorial, often NP-hard, and usually solved on the ground by heuristics, local search or integer programming. They suit RL because a trained policy decides in microseconds where search takes minutes, and because visibility windows, memory, energy and slew-time constraints map naturally onto episode dynamics.

They are also where the main weakness of RL is most exposed: search replans online and a learned policy does not. Within training conditions the policy wins on speed and often on quality; outside them, search wins on quality by a wide margin (Section 13.3).

13.2. Earth-Observation Scheduling

Earth-observation scheduling allocates observations within visibility windows under memory, energy and slew constraints to maximise priority-weighted profit. The RL literature has moved from single-satellite, fixed-target formulations to agile, dynamic, multi-satellite and multi-objective ones.

Huang et al. [108] cluster nearby targets before scheduling with DDPG and outperform heuristics, but their batch scheduling needs retraining whenever the target list changes. Wang et al. [109] remove this limitation with a dynamic stochastic knapsack formulation in which arriving tasks are accepted or rejected, improving profit and execution time with A3C. For agile satellites, two-phase approaches trade profit against image quality [110], and decomposition with parameter transfer speeds up bi-objective scheduling [111]; an early multi-satellite formulation used genetic-hybrid learning [112].

The current state of the art is a two-phase hybrid in which an encoder-decoder network sequences tasks and a heuristic allocates resources [113]. Against adaptive large-neighbourhood search on scenarios of up to 1200 tasks and four satellites, it delivers more than 16% higher profit in the hardest scenario using under 3% of the computation time. This, the strongest scheduling result in the corpus, again comes from hybridising learning with a heuristic rather than from learning end to end.

Herrmann and Schaub [114] compare PPO, shielded PPO, A2C, DQN and a tree-search method against a genetic-algorithm upper bound. On short horizons the algorithms perform similarly; on longer horizons the policy-gradient methods hold up while tree search and DQN degrade; and on the harder agile-satellite problem the ranking inverts and tree search leads. PPO and shielded PPO show the smallest variance across seeds, and shielding is the cheapest because it avoids costly resets after resource violations. The resulting guidance, policy gradients for long horizons and tree search for complex short-horizon resource management, is one of the few actionable algorithm-selection results in the field.

Multi-agent formulations extend scheduling to constellations, including realistic multi-satellite Earth-observation case studies, sequential satellite assignment, multi-target replanning and disaster-monitoring planning, while training-environment enhancement targets the generalisation problem directly.

Hierarchical decompositions help when onboard resources, rather than geometry, are binding. Ramezani et al. [115] split CubeSat-constellation scheduling into a MADDPG level for global assignment and a multi-agent DQN level that acts as a safety layer, reassigning tasks when a satellite has failed or lacks the energy or time to complete them, with attention-based prioritisation and a learned energy forecaster. In simulation with up to five CubeSats it shortens the makespan by at least 10% relative to flat MADDPG. The margins are modest and simulation-only, but the structure, learned allocation above a safety-preserving layer, is the one that wins elsewhere in this section.

The enabling infrastructure is BSK-RL [116], which couples the Basilisk high-fidelity simulator to the Gymnasium API, runs hundreds of times faster than real time, and supports heterogeneous multi-agent constellations with communication models. It is the closest thing the field has to a standard benchmark suite, and adopting it is the cheapest available improvement in reproducibility.

13.3. Active Debris Removal Mission Planning

Debris-removal sequencing is a time-dependent travelling-salesman problem under propellant and duration budgets, whose complexity grows explosively with the number of objects. Yang et al. [117] gave the first RL treatment, a modified AlphaGo Zero scheme with tree search guided by a deep network, which converged quickly to the optimal sequence on real Iridium 33 debris data but left scaling open.

The most instructive study in this category [118] compares masked PPO, trained nominally or with domain randomisation over Δv and mission duration, against Monte Carlo tree search (MCTS). In the training regime the learned policies visit slightly more debris than MCTS and decide about 240 times faster. When the Δv budget is cut to a third of the training value, the nominal policy collapses to about a tenth of its nominal performance, domain randomisation recovers only part of the loss, and MCTS visits nearly twice as many objects as the best learned policy (Figure 7).

This experiment captures the field’s central unresolved weakness. Domain randomisation, at 5.5 times the training cost, recovered part of the loss but still fell 47% short of online search. Robustness within the training distribution is not generalisation beyond it, and describing the former as the latter is the most consequential terminological error in the field. Where the operating envelope may shift, a learned policy should guide an online search rather than replace it, as the AlphaGo Zero-derived planners do.

Related work covers multi-debris rendezvous with RL-guided planning, optimal multi-debris planning with co-elliptic transfers and refuelling, mission planning with refuelling and adaptive collision avoidance, machine-learning-based ADR planning, and design of multiple debris-removal missions with machine learning.

13.4. Sensor Tasking and Space Situational Awareness

Sensor tasking for space situational awareness (SSA) is a partially observed allocation problem whose reward is information gain about a catalogue. Deep RL has been compared with classical optimisation for ground-based tasking and extended to space-based sensors, cislunar SSA and multi-agent, multi-target tasking; the area is methodologically close to Earth-observation scheduling and would benefit from the same benchmarking discipline.

RL has also been applied to navigation itself. Getzandanner and Martin [119] pose joint navigation and environment characterisation in the planar two-body problem as an MDP that rewards reduction of the formal uncertainty in the gravitational parameter, with ground-station measurements and manoeuvres as actions. The PPO policy outperforms heuristic and random baselines and learns to reshape the orbit to improve observability, although the authors stress that substantial development is needed before flight-like use. Onboard optical position estimation for cislunar navigation extends the idea to another sensing modality.

13.5. Onboard Resource Management and Language-Model Agents

Deep RL has also been applied to energy management of LEO power systems, to the synthesis of operations procedures and, with safety shielding, to complex spacecraft tasking.

Since 2024, large language models (LLMs) have been evaluated as spacecraft operators in Kerbal Space Program, as goal-oriented exploration agents and in the KSPDG benchmark. The most developed of these, GUIDE [120], pairs a lightweight acting model, conditioned on an evolving natural-language playbook of rules, with an offline frontier model that revises the playbook by reflection between episodes. On four KSPDG scenarios GUIDE improves the benchmark’s composite score over a static LLM by factors of about six to several hundred and outperforms LQR, yet a simple prograde-alignment heuristic still beats it in two of the four.

LLM agents are therefore not yet competitive with well-tuned classical guidance or trained RL policies on control quality, and these papers say so. Their interest lies in adaptation without gradient updates, human-readable rationales and natural-language operator intent, properties that matter for human-autonomy teaming and are independent of control performance.

13.6. Comparative Synthesis for mission-Level Autonomy

Table 13.1 shows the largest measured speed advantage of RL (thirtyfold in scheduling, more than two-hundredfold in debris planning) and its most clearly documented generalisation failure. These are the same finding seen from two sides: a policy is fast because it does not search, and brittle for the same reason. The architectures that resolve the tension, learning that guides search and learning that sequences while a heuristic allocates, produce the best results in this section.

14. Cross-Cutting Comparative Analysis

This section aggregates across categories to answer five questions: which methods are used; whether learned controllers beat classical ones, and on what; what they cost to train and to run; how far up the validation ladder the field has climbed; and whether the evidence is reported well enough to be relied on.

14.1. Algorithm Usage Across The corpus

Table 14.1 summarises the dominant algorithm families in each category, indicating the modal choice and notable exceptions rather than giving an exhaustive count.

Counted by primary algorithm, PPO or a direct derivative (shielded, masked, multi-reward, SkipPPO, or recurrent meta-RL trained by PPO) is used in 121 of the 253 retained works (47.8%), followed by DDPG and TD3 (34), the DQN family (26), SAC (19), A2C/A3C (14) and tree-search hybrids with learned networks (11); the remainder use inverse RL, imitation, REINFORCE, policy iteration, evolutionary or physics-informed methods, or are frameworks rather than algorithms.

Three observations follow. First, PPO’s dominance reflects convenience (tolerance of long episodes and badly scaled rewards, and easy parallelisation) rather than demonstrated superiority: where algorithms have been compared head to head, the ranking is task-dependent (Section 13.2) and the margins are small relative to the effect of formulation (Section 10.3). Second, the two-level pattern of Figure 8 recurs independently in five categories (Q-law parameterisation, ZEM/ZEV gain learning, deep guidance over a conventional inner loop, a DRL planner over an MPC tracker, and learned sequencing with heuristic allocation) and gives the best results in each. Third, nearly every claim of online adaptation is implemented as a recurrent policy trained across a task distribution rather than by online gradient updates.

14.2. Does RL Beat Classical Control? A Structured Answer

Table 14.2 lists every head-to-head comparison in the corpus with numbers on both sides, and classifies the outcome.

Of the seventeen comparisons, the learned controller wins six, loses six and splits five. The outcome is organised not by application domain but by the type of metric (Figure 9). On distributional and off-nominal metrics (success under dispersion, consistency, adaptation to faults) RL wins four of five, the exception being the out-of-distribution debris-planning case. On nominal optimality against a converged optimal-control reference it loses five of five. Against combinatorial search inside the training distribution it wins both comparisons, and every multi-metric comparison returns a split.

This evidence has limits. Seventeen comparisons drawn from about twelve studies is a small sample, and because several studies contribute two rows, the consistency of the partition is partly inflated; several rows come from the same groups; classifying a verdict as split is a judgement; and publication bias makes unfavourable comparisons less likely to appear. The pattern is consistent enough across domains to state, but it is a hypothesis supported by the evidence rather than a measured fact.

A corollary follows: reporting only nominal optimality understates RL, and reporting only robustness overstates it. A comparison is complete only when both are reported for both methods, which fewer than a dozen studies in the corpus do.

14.3. The Optimality-Robustness Trade, Quantified

Where both sides are reported, the trade has a consistent shape (Table 14.3, Figure 10).

The rows use different objectives (quadratic cost, time or final mass, Δv and propellant mass), so a given percentage on one is not equivalent to the same percentage on another. The summary “1–10% versus two- to fivefold” describes the order of the penalty in well-conditioned versus chaotic problems, not a single normalised quantity.

For practitioners, three expectations follow: a penalty of 1–10% where the problem is well conditioned and the reference converges reliably; a two- to fivefold penalty where the dynamics are chaotic and the reference exploits global structure that the policy cannot see; and a reversal of the comparison once operating conditions leave the training distribution.

14.4. Training Economics

Four conclusions follow from Table 14.4 and Figure 11. (i) Sample budgets span more than two orders of magnitude, from 10⁶ steps for a discrete planning problem to 5×10⁸ for the flight attitude controller, and the largest budgets accompany the highest-fidelity outcomes. (ii) Transfer learning and curricula are the cheapest accelerators: transfer across dynamical models of increasing fidelity halved training time in halo station-keeping, and a curriculum raised Astrobee docking success roughly tenfold. (iii) RL is not always more sample-hungry than imitation, which must first generate optimal trajectories. (iv) Reporting is poor: fewer than half of the studies in Table 14.4 give both a sample count and a wall-clock time, and few give the hardware needed to interpret it.

14.5. Onboard Computational Cost

This is the dimension on which RL is clearly superior to online optimisation, and independent sources agree (Table 14.5, Figure 12).

Two implications follow. First, computation is no longer the main barrier for small networks in slow loops: a 105 kB policy running in 38 ms on a 40 MHz microcontroller, and sub-2 ms execution on radiation-hardened processors, settle the question that dominated the 2018–2022 literature. Second, the relevant comparison is now with MPC and online optimisation, where RL is orders of magnitude cheaper; the counter-argument is that the cost of MPC buys constraint guarantees that the policy lacks, which is a verification argument rather than a computational one.

14.5.1. What the Timing Figures do not Settle: Implementation Assurance

Three qualifications apply, none of which is addressed in the surveyed corpus.

Duty cycle. The flight figure of 38 ms at 1 Hz is a 3.8% duty cycle; the same network on the same processor would occupy 38% of a 10 Hz loop and could not run at the 10–50 Hz rates typical of agile pointing or proximity-operations inner loops. Inference is cheap for small networks in slow loops, which is the regime in which RL has flown.

Provenance. The sub-2 ms figure is a maximum observed across several processor classes, reported without the specific processor or the network size. Timing results should always be reported together with the architecture, parameter count and device.

Implementation assurance. The questions that gate flight software are absent from this literature:

• Radiation effects: a single-event upset in a weight, or silent corruption of an activation buffer, produces a wrong control output without raising an exception, yet no study addresses upset rates in weight memory, the sensitivity of closed-loop behaviour to bit flips, or scrubbing, error-correcting-code and redundancy strategies for network parameters.
• Quantisation: flight deployment will use fixed-point or int8 arithmetic, but no work reports the closed-loop effect of quantising a trained policy, and any verification result would have to be redone on the quantised network.
• Worst-case execution time: all reported timings are measured maxima, whereas schedulability analysis needs a bounded worst case; fixed-topology feedforward networks are well suited to static bounding, so this gap is tractable.

Gaps G3 and G4 concern verification of the policy; there is no corresponding work on verification of its implementation, recorded as gap G12 in Section 15.

14.6. Distribution Across the Validation-Fidelity Ladder

Each work was assigned to the highest ladder level at which it reports evidence (Section 2.5). Of the 253 retained works, 218 could be classified from the accessible text. Assignment is a judgement, especially at the L0/L1 and L1/L2 boundaries, and stricter criteria would move works down rather than up, so the distribution in Figure 13 and Table 14.6 is an upper bound on the evidential maturity of the field.

In total, 31 works (14.2% of those classified) reach L2 or above, the classification also used in Table 1.2.

About six works in seven have never been evaluated outside the simulator that trained them. This is not a criticism of individual papers, since L1 is appropriate for algorithm research, but it means that the field’s claim to flight readiness rests on three works, two of them from the same team on the same platform. The L2 tier offers the cheapest progress: evaluating a policy in a simulator it was not trained in needs no hardware and exposes the failure modes that domain randomisation was meant to cover.

14.7. Navigation Fidelity: What Estimator is The guidance Policy Actually Flying with?

Table 14.7 tests the claim that navigation is idealised in this literature (gap G8), using the observation model recorded for each work at L1 or above, where it could be determined.

Only three works propagate an estimator with its own covariance inside the training or evaluation loop. Elsewhere, navigation error is absent or modelled as additive noise, which lacks what makes real navigation hard: temporally correlated error, transients after manoeuvres, degradation under poor geometry, and divergence. The works that use real sensors are, with one exception, those that reach L3 or L4, so navigation realism and physical realism are currently acquired together, which is why the (E2, P1) cell of Table 2.2 is nearly empty.

The consequence is specific. A policy trained against additive Gaussian noise learns to average it out, whereas behind a real filter the error is predictable over short horizons and biased after manoeuvres, in ways that averaging makes worse and that current training environments cannot reveal.

14.8. Safety and Constraint Reporting: The M6 Metric Family

The safety metrics (M6) receive no treatment elsewhere in this section because almost no work reports them. Table 14.8 collects what exists; its sparseness is itself the finding.

Intervention frequency is the most informative safety quantity that a run-time-assured controller can report, and no work reports it. It distinguishes a learned controller with an occasional guard from a classical backup controller with a learned suggestion attached, which require different certification arguments: a controller whose filter intervenes on 40% of steps has not been shown to work, only not to break the filter.

14.9. Benchmarking and Reproducibility

Four assets in the corpus function, or could function, as shared benchmarks: BSK-RL, the STARS docking and inspection challenge problems with their fourteen safety constraints, KSPDG with its annual competition, and OrbitZoo with its ephemeris-validated multi-agent dynamics. The four categories in which most work is done (powered descent, cislunar transfer, low-thrust orbit raising and attitude control) have none.

The consequences are visible throughout Section 3, Section 4, Section 5, Section 6, Section 7, Section 8, Section 9, Section 10, Section 11, Section 12 and Section 13: studies of the same problem use incommensurable metrics, a decade of orbit-raising work cannot be ranked, no RL controller can be compared with the best NRHO station-keeping figures, and the landing literature, the oldest in the field, has no common initial-condition distribution. Section 16.1 proposes an inexpensive remedy.

14.10. The Hidden Cost: Reward Engineering and Formulation Sensitivity

Every RL paper reports its algorithm and most report hyperparameters, but almost none report how many reward functions were tried, even though formulation, not algorithm, is the dominant source of performance variance:

• Four PPO configurations on the same Int-Ball2 docking problem span 0.1% to 99.3% stable-docking success [88].
• Adding angular velocity to an attitude observation reduces tracking error about seventeenfold, and replacing quaternions with direction cosine matrices improves it further [13].
• Observation stacking roughly doubles the return for both SAC and PPO on varying-mass attitude control [14].
• Curriculum learning raises Astrobee docking success from about 9% to over 90% [89].
• Three reward correspondences for the same cislunar approach problem produce materially different tracking error and manoeuvre feasibility [45].
• Run-time assurance during training degrades the resulting policy unless RTA intervention is penalised in the reward [11].
• Weak coupling between state representation and reward yields very poor policies in powered descent [67].

The fix is simple: report the ablation, not only the winner. The STARS ablations are the model to follow, and they are also the source of the most transferable design guidance in the corpus.

14.11. Statistical Reporting Practice

Assessed against metric family M7, the literature is weak.

• Random seeds. Few studies report results across training seeds; the exceptions (ten seeds in the scalable-inspection study, five in the scheduling comparison, and several attitude studies) are among the most convincing papers in their categories. Given the large seed-to-seed variance of deep RL, a single-seed result is closer to an anecdote than to a measurement.
• Confidence intervals. Monte Carlo sample sizes are usually reported, but intervals on success rates almost never are. A 96% success rate from 200 samples has a 95% Wilson interval of 92.3–98.0%, so differences of a few points between methods are not resolvable at typical sample sizes, and several published comparisons fall within that band. The same applies to this review: only the large, order-of-magnitude contrasts in Table 14.2 survive an interval-aware reading, and the conclusions rest on those; smaller contrasts are directional.
• Baseline tuning effort is essentially never reported, so readers cannot tell whether a PID or PD baseline received comparable effort; since the most carefully tuned classical baseline in the attitude literature beat every RL agent, this matters.
• Few unfavourable results. Only eleven of the 253 works (4.3%) report a comparison in which the learned method is outperformed on some metric, and these eleven supply every losing or split row of Table 14.2. This proportion does not measure publication bias directly, but it is consistent with strong bias, and the six-win, six-loss partition of Section 14.2 should be read as a property of the few studies willing to report both outcomes.
• Negative results, such as that RL was not yet comparable with a dedicated optimiser, that quaternion feedback beat every agent, or that domain randomisation recovered only part of the gap to tree search, are rare and disproportionately informative.

15. Gaps and Open Problems

The twelve gaps below are ordered by the extent to which closing them would change what the field can claim, not by difficulty.

G1: No shared benchmarks where they matter most. Powered descent, low-thrust orbit raising, cislunar transfer and attitude control make up most of the corpus but have no common problem definition, initial-condition distribution, disturbance model or metric set. Where shared assets exist, the associated literature is more comparable and more self-critical.

G2: Robustness is conflated with generalisation. Monte Carlo campaigns sample the training distribution. The one study that systematically evaluates outside it finds a learned planner collapsing and falling 47% short of online search (Section 13.3). Three independent findings point the same way: cloned networks fail under perturbations accumulated along the trajectory while remaining robust to initial displacement (Section 7.3); a manipulator policy trained without observation noise fails outright under noise (Section 11.2); and batch-trained scheduling policies need retraining when the task set changes (Section 13.2). Until out-of-distribution evaluation is routine, robustness claims should be read as in-distribution claims.

G3: Verification does not scale. The only formal verification result, liveness and safety for 2-D docking, explicitly does not extend to larger spaces or richer dynamics. Elsewhere the field relies on run-time assurance, which is sound engineering but moves the certification object from the policy to the filter, and leaves open what performance survives frequent intervention.

G4: Stability guarantees exist only where learning was confined. The two clean stability arguments, Lyapunov Q-laws that are stable for any admissible learned weights and a dual-quaternion proximity controller with an explicit Lyapunov analysis, both restrict what the network decides. No stability certificate exists for an end-to-end deep policy on a nonlinear spacecraft plant. Lyapunov-decrease reward shaping based on a sliding surface [18] narrows the gap empirically and has reached hardware-in-the-loop testing, but it promotes rather than certifies stability.

G5: Reward engineering is an unreported cost. Formulation choices change outcomes by up to three orders of magnitude, yet the search over formulations is invisible, so a competent engineer given a paper cannot reliably reproduce its result.

G6: Statistical practice lags adjacent fields. Single-seed results, missing confidence intervals and baselines of unreported tuning effort are common, so many published differences are not statistically resolvable.

G7: The fidelity ladder is bottom-heavy. About six works in seven are evaluated only in their training simulator and three report flight results; the nearly free L2 tier is almost unused.

G8: Navigation is idealised. Of the 218 classified works, only three propagate an estimator with its own covariance in the loop, among them the joint navigation-and-characterisation formulation [119], against 41 using additive noise and 34 assuming perfect state (Section 14.7), so the interaction of learned guidance with a real estimator is essentially unstudied.

G9: Contact, capture and post-capture dynamics are under-served. The phase in which a servicer’s inertia changes discontinuously, where adaptation would be most valuable, is the least addressed, and only one work uses tactile or force feedback.

G10: Soft, continuum and tethered space systems are an open field. Classical control is weakest exactly where learning is strongest, and the terrestrial literature transfers directly, yet fewer than ten space-specific works exist.

G11: Human-autonomy teaming is barely addressed. STARS is essentially alone in treating directability, bounded autonomy, calibrated trust and situational awareness as requirements, and its operator studies remain future work. A controller that cannot be reviewed, bounded or overridden intelligibly will not fly, whatever its performance.

G12: Implementation assurance is absent. No work addresses single-event upsets and silent data corruption in network memory, redundancy strategies for parameters, the closed-loop and verification consequences of quantisation, or bounded worst-case execution time (Section 14.5.1). The gap is tractable, since fixed-topology networks suit static timing analysis and radiation-effects methods exist, but no learned controller can be qualified for a high-consequence function until it is closed.

Two category-specific gaps complete the list: no RL NRHO station-keeping result in Δv per year, where a strong MPC benchmark exists (Section 4.2), and no comparison of RL landing guidance with modern convex powered-descent guidance (Section 8.7).

16. Prospective Research Directions

16.1. Benchmarks, and a Reporting Standard

The highest-value, lowest-cost action available to the community is to define four reference problems with published reference solutions: Mars 6-DOF powered descent (B1), GTO-to-GEO orbit raising (B2), NRHO station-keeping (B3), and attitude slew and tracking (B4). Table 16.1 specifies each in the form a benchmark requires.

Alongside these, a minimal nine-item reporting standard would greatly improve comparability at almost no cost:

• report results over at least five training seeds, with dispersion;
• report confidence intervals on Monte Carlo success rates;
• report both nominal optimality against a reference optimiser and robustness under a stated dispersion;
• report at least one out-of-distribution evaluation with the shift stated explicitly;
• report the ablation over formulation choices, not only the winning configuration;
• report sample count, wall-clock and hardware;
• report the baseline’s tuning effort;
• state the validation-fidelity level explicitly;
• release the environment.

Benchmarks fail more often for organisational than for technical reasons, and the assets that already work show what is needed. BSK-RL is maintained and versioned by a university laboratory alongside the simulator it wraps; the STARS problems have a government host and a public-release process; the annual KSPDG competition, with a fixed submission interface, puts every entrant on the same evaluation harness; and OrbitZoo validates itself against external data.

A workable governance model for B1–B4 therefore needs a named institutional host with a succession plan; a permissive licence for environments and reference solutions; semantic versioning with a frozen evaluation seed set per version; reference solutions from at least two independent solvers, published with their convergence tolerances; and an annual evaluation round with a fixed submission interface, ideally attached to an existing conference. None of this is research, but it is the difference between a benchmark that is cited and one that is used.

16.2. Verification and Run-Time Assurance that Scales

Two lines of work are needed. The first is scalable formal methods (abstraction, reachability analysis and semidefinite relaxations for neural controllers) applied to six-degree-of-freedom problems rather than 2-D docking. The second is assurance-case-shaped RTA: because RTA moves certification from the policy to the filter, the field needs a principled account of what performance survives frequent intervention, how to bound intervention frequency, and how to design rewards that keep the policy independent of the filter, which the STARS ablations show is not automatic. The immediate step is to report, for every run-time-assured result, the intervention frequency, its distribution over the episode, and performance with the filter disabled; no work currently reports any of these (Section 14.8).

A promising middle route is certificate-carrying policy structures, such as the RL-parameterised Lyapunov law, in which the network’s output range is constrained so that the closed-loop guarantee is structural. This should be treated as a design methodology rather than a device specific to Q-laws, asking for each application which classical law can be parameterised.

16.3. Hybrid Learning-and-Optimisation Architectures

The evidence of Section 14.1 indicates clearly that the two-level pattern wins. The most promising directions are:

• Learned warm starts and initial guesses for convex and pseudospectral solvers, exploiting the demonstrated ability of multi-reward RL to produce Pareto sets of dynamically feasible trajectories that converge under SQP.
• Learned planners over classical trackers, extending the DRL-planner-plus-MPC-tracker architecture, with processor-in-the-loop validation as standard.
• MPC-shaped rewards, in which an MPC reference enters the reward rather than the command path, which in slosh-affected docking improved both success and control effort for PPO and SAC [21].
• Learning as search guidance, extending AlphaZero-derived planners, which retain online replanning and therefore do not exhibit the out-of-distribution collapse of purely reactive policies.
• Residual and gain-scheduling RL over an existing certified controller, which was the winning architecture on the hardest plant in the corpus and which has the practical merit that the fallback is the current flight controller.

16.4. Sample Efficiency, Model-Based RL, and Offline RL

Sample budgets of 5×10⁸ steps are affordable for a research group with GPUs but not for a mission that relies on a slow, non-differentiable industrial simulator. Three routes matter: model-based RL and world models, which the one re-entry study to evaluate them found competitive without tuning; offline RL on the large archives of telemetry and operational simulation that agencies already hold, for which the off-policy hopping-rover formulation is an early precedent; and surrogate-then-verify pipelines that train on a fast surrogate and verify, and report results, on the high-fidelity propagator.

16.5. Sequence Models, Foundation Models and Multi-Task Policies

The architectural trend runs from MLPs through recurrent meta-RL to transformers and state-space models. The likely next step is multi-task, cross-phase policies, for example a single policy for approach, inspection, docking and departure, or for entry through touchdown, which multi-phase transformer results suggest is feasible. Because verification becomes harder as a policy’s scope grows, this direction should be paired with that of Section 16.2.

Language-model agents should be pursued where their evidence is strong (explanation, interpretation of operator intent, procedure synthesis and adaptation without retraining) rather than for closed-loop control, where a simple heuristic remains competitive with the best current LLM agent.

16.6. Sim-to-Real as a Discipline

The flight and testbed results share a recipe (simulation training, aggressive randomisation over unmeasurable parameters, curricula, small networks and actuator caging) that the field should formalise. The priorities are to characterise what must be randomised, for which the LeLaR discrepancy list is a better starting point than a generic inertia dispersion; to report the reality gap quantitatively, following the three-fidelity LINCS protocol; and to make safety cages a standard component, since every flight demonstration used one.

16.7. Navigation-in-the-Loop and Integrated GNC

Replacing additive noise with an actual estimator and, more ambitiously, co-designing policy and estimator so that manoeuvres improve observability, is both an obvious gap and an area of natural advantage for RL, because information gain is easy to express as a reward and hard to express in a classical guidance law. The joint navigation-and-characterisation and shape-reconstruction studies are the seeds of this direction.

16.8. Contact, Capture and Morphologically Complex Systems

Priorities: post-capture adaptation to a discontinuous inertia change, which is the canonical meta-RL problem and is almost untouched; tactile and force-feedback policies for soft capture; and systematic transfer of the terrestrial soft- and continuum-robotics RL literature to tether-nets, deployable structures and continuum manipulators, where the classical alternative is weakest.

16.9. Multi-agent constellation control at operational scale

Existing MARL work rarely exceeds a handful of agents, whereas mega-constellation station-keeping, collision avoidance and scheduling involve thousands. Fixed-dimension observations independent of agent count, as demonstrated for multi-agent inspection, are the right primitive, and combining them with the validated propagation of OrbitZoo and the throughput of BSK-RL is a concrete and achievable programme.

16.10. Human-Autonomy Teaming and the Operational Pathway

The qualification pathway should build on existing learning-assurance frameworks rather than start afresh. Aviation has built this apparatus in recent years, and the space community has so far engaged little with it. The relevant precedents are EASA’s machine-learning guidance and its W-shaped learning-assurance process, which extends the DO-178C V-model with a learning-management branch covering data management, learning-process verification, model implementation and inference-model verification; the joint EUROCAE ED-324/SAE ARP6983 process standard for certifying aeronautical products that implement AI; DO-178C’s treatment of non-deterministic software, which is why every deployed policy in this corpus has frozen weights; and, for space, ECSS-E-ST-10-02C on verification, ECSS-Q-ST-80C on software product assurance and NASA-STD-8739.8 on software assurance and safety.

Mapping L0–L4 onto the W-shaped process is straightforward: L0–L1 correspond to the descending branch (requirements, data management, learning-process verification), L2 to inference-model verification and implementation, L3 to integration and hardware/software verification, and L4 to in-service monitoring. The space community would have to add two elements that aviation does not need: radiation and single-event-effect assurance for the implementation (G12), and a treatment of environments that cannot be flight-tested before use, such as asteroid proximity operations, which is why the surrogate-then-verify and independent high-fidelity tiers matter more here than in aviation.

Finally, the field needs a staged flight-qualification pathway, analogous to the technology-readiness scale and mapped onto these frameworks, that defines the evidence needed to move a learned controller from L1 to L4: independent high-fidelity evaluation, processor-in-the-loop timing on target hardware, hardware-in-the-loop testing with realistic actuator pathologies, run-time assurance with a certified fallback, an operator interface that exposes bounded autonomy and intervention history, and a monitored in-orbit experiment on a low-consequence platform before operational use. Both flight campaigns to date followed a version of this sequence. Figure 14 sets out the pathway; its adoption by agencies would do more for the deployment of RL in space than any algorithmic advance currently in prospect.

17. Conclusions

This review examined what learned controllers achieve, relative to which baselines, and under what validation conditions across more than 200 studies in eleven space-control categories. The evidence shows that reinforcement learning (RL) has progressed beyond simulation: three studies report in-flight demonstrations, including CubeSat attitude control and Astrobee experiments aboard the ISS. These results used compact policies trained in simulation with domain randomisation and transferred without hardware-specific retraining. Small networks also meet representative flight-computing constraints, although radiation effects, quantisation, and worst-case execution-time guarantees remain largely unstudied.

Performance relative to classical control depends strongly on the metric and operating regime. Learned controllers tend to perform well on robustness, adaptation, consistency, and decision speed, but generally retain a modest optimality gap when reliable numerical optimisation is available. Robustness within the training distribution should not be interpreted as out-of-distribution generalisation, which remains weakly tested. Across studies, controller formulation, including observations, rewards, curricula, and architecture, often affects performance more than the choice among commonly used RL algorithms.

The strongest results frequently use hybrid architectures in which learning modifies, selects, or augments established guidance and control methods rather than replacing them entirely. Despite these advances, the evidence base remains dominated by training-simulator evaluations, with few independent high-fidelity, hardware, or flight tests. Reporting of random seeds, uncertainty, baseline tuning, and common benchmarks is also limited.

RL is therefore technically feasible for selected space-control functions, but broader adoption is constrained less by computational capability than by validation and assurance. Shared benchmarks, stronger reporting standards, independent simulator evaluation, scalable verification methods, and staged qualification procedures are the main requirements for establishing whether learned controllers can be accepted for higher-consequence mission functions.