Submitted:
07 September 2026
Posted:
09 September 2026
You are already at the latest version
Abstract
Persistent custody of moving ground targets is comparatively easy when every satellite has clean sensors and global team state. It is not easy under range-limited inter-satellite links (ISL), packet dropout, and slot-level false-positive clutter. In a Keplerian+J2 environment with an honest pointing rule (a tip holds only if the reported contact is on the true target), a memoryless local greedy policy achieves 42.4% mean custody on the HARD preset, and shared-policy PPO saturates near 49%— close to the local true-detection ceiling, not a failed optimizer. A graph-conditioned controller that fuses agreeing neighbor contacts and coasts the last confirmed track recovers 92.2% (heuristic) / 92.4% (hierarchical PPO), matching the privileged geometry reference (91.3%). On the EASY preset (no clutter, global links) local greedy already reaches 93.1%, so the performance gap is a HARDonly phenomenon driven by information and track persistence rather than by the inability of PPO to optimize. The contribution is a cluttered, communication-constrained custody benchmark, anempirical demonstration that local policies collapse to the true-detection ceiling, and a graph-conditioned hierarchical policy (heuristic and learned) that recovers the visibility ceiling.
Keywords:
multi-agent reinforcement learning
; satellite custody
; inter-satellite links
; sensor clutter
; hierarchical PPO
; space domain awareness
; tip-and-cue
1. Introduction
Maintaining persistent custody of moving ground targets from low Earth orbit (LEO) remains a core challenge for space-based intelligence, surveillance, and reconnaissance (ISR). Ground targets move, LEO satellites follow fixed orbital geometry with limited fields of view, and continuous tracking requires timely handoffs. Classical scheduling and local heuristics struggle when visibility windows are intermittent and observations are noisy.
Prior multi-agent reinforcement learning (MARL) work on satellite tasking and sensor coordination has shown that shared-policy or centralized-training methods can coordinate constellations when agents receive clean observations and, often, global team state [1,2,3,4,5,6,7]. Our own earlier shared-policy PPO study on a simplified 4-satellite / 2-target custody task with Skyfield kinematics, exclusive capture, and a re-tip-while-visible rule achieved roughly 67–74% mean custody under clean sensors and full team cues [8]. That setting is the unconstrained special case: perfect visibility bits, no false positives, and no communication limits.
Operational constellations do not enjoy those conditions. Inter-satellite links are range-limited and lossy; sensors report slotted contacts that can be false positives; a tip pointed at a ghost does not hold the true target. Under those constraints a memoryless local policy—whether hand-coded greedy or shared-policy PPO—cannot reach the geometric visibility ceiling, because the action that would sustain custody is no longer labeled by a reliable “target visible” bit.
This paper studies that failure mode and a graph-conditioned remedy. The contributions are:
- A multi-agent custody environment with Keplerian+J2 motion, range-limited ISL with packet dropout, slot-level false-positive clutter, false negatives, sensor noise, and an honest pointing rule (non-privileged tip credit requires a true detection).
- Empirical evidence that, on a HARD preset, local greedy and shared-policy PPO remain near the local true-detection ceiling (≈42–49% custody), while a privileged geometry reference sits near 91%.
- A graph-conditioned controller that (i) fuses agreeing neighbor contacts on a target slot and (ii) coasts the last confirmed track through subsequent ghost frames, recovering ≈92% custody as both a heuristic and a hierarchical PPO with masked attention over currently reachable neighbors.
The gap appears only when clutter is present: on an EASY preset (global links, no false positives) local greedy already matches the visibility ceiling. The lesson is about information and short-term track persistence, not about PPO failing to learn.
2. Related Work
Reinforcement learning has been applied to agile Earth observation scheduling [1], multi-satellite coordination [4,5,9], and space situational awareness sensor tasking [2,3]. Hierarchical and communication-aware MARL methods appear in multi-agent control more broadly, including attention over neighbors and centralized training with decentralized execution. Multi-target tracking literature contributes the classical idea of a track file that persists through missed detections.
Our prior shared-policy custody environment [8] (and the open baseline implementation marl_custody_model_new_5.py) assumed clean sensors and constellation-wide cues. That work remains a useful unconstrained baseline. The present study keeps the same operational tip rule (re-tip every step while visible, exclusive capture) but replaces clean visibility bits with slotted, cluttered reports and adds range-limited ISL. Graph fusion and short track coast are classical tracking mechanisms exposed to the agents through observations; hierarchical PPO learns to act on that representation rather than inventing memory from a purely local Markov observation.
3. Problem Formulation
We consider N LEO satellites and T moving ground targets. Time is discrete with step s. Each satellite maintains a local report for each target slot . A report may be a noisy measurement of the true target, a miss (false negative), or a false-positive contact placed uniformly in the field of view. True angular visibility is determined by sub-satellite geometry and a fixed FOV (default ).
Custody of target t is held by at most one satellite. A non-privileged tip sustains or acquires custody only if the satellite is in true FOV and the contact it is pointing at (after any fusion/coast) is near the true target location. Privileged credit uses true FOV only and serves as a geometry reference, not as an ISL policy.
Inter-satellite communication forms an undirected graph : an edge exists if inter-satellite range is within and the packet is not dropped. Local policies receive only own reports. Graph policies receive neighbor messages (fuel, assignment, per-target offsets and confidence) and a reachability mask. The environment then applies:
- 1.
- Fusion. Within a connected component, the largest Earth-fixed cluster of agreeing contacts on slot t becomes the network track; members that can see the target adopt that pointing.
- 2.
- Coast. If the current frame is not a true detection but a recent good track exists and the target remains in true FOV, the last good pointing is retained for a limited age and preferred over fresh clutter.
Fuel and power budgets constrain tips and small maneuvers. The objective is mean custody fraction over targets and timesteps.
Three ceilings matter on HARD (): any-satellite true visibility ; local true-detection rate ; graph true-detection after fusion and coast . Privileged heuristic custody is .
4. Methods
4.1. Environment
The environment is a PettingZoo ParallelEnv with Keplerian+J2 propagation. Default constellation: Walker-style , RAAN spacing , inclination , altitude 550 km, targets, episode length 240 steps. Targets begin near distinct subpoints and execute slow random walks. Custody is exclusive; a holder that fails to re-tip while the target remains visible loses the hold.
Presets:
- EASY: effectively global links, no dropout, false-positive rate 0, small noise.
- MEDIUM: ISL range km, dropout , false-positive rate , false-negative rate .
- HARD: ISL range km (adjacent planes km), dropout , sensor noise , false-positive rate , false-negative rate , tight fuel (initial , cost ).
Observation modes: local, graph, privileged. Action modes: flat discrete tip/idle, or hierarchical tip plus .
4.2. Controllers
Heuristics (not RL): random; local greedy; graph auction / hierarchical heuristic (prefer confirmed or coasted tracks; optional small ); privileged geometry oracle.
Trained RL (RLlib PPO): shared-policy PPO on local observations (MLP); hierarchical communication-graph PPO (hier_comm) on graph observations with masked attention over currently reachable neighbors. Iteration 0 is untrained weights. Heuristic and RL evaluation paths are separate.
4.3. Evaluation Integrity
An earlier internal curve that showed for every method by iteration 1 was invalid (slots behaved like labeled visibility bits). In the current environment, non-privileged tip credit requires a true detection, local observations omit consensus confidence, heuristic and RL names are never aliased, and iteration 0 is reported as untrained. Metric: mean custody fraction. Seeds for headline comparisons: 11, 22, 33.
5. Results
5.1. EASY Versus HARD
On EASY (), local greedy already reaches 93.1% mean custody, matching hierarchical heuristic (93.1%) and sitting near the privileged reference (91.3%). With clean sensors and global connectivity there is no operational gap for local policies to close.
The local true-detection ceiling () explains why memoryless local policies cannot approach 90%. Graph fusion and coast raise the usable detection rate to .
5.2. Trained Policies on HARD
Shared-policy PPO (local observations, 16 iterations) and hierarchical communication-graph PPO (20 iterations) are summarized in Table 2Table 3 and Figure 2. Untrained hierarchical PPO already reaches custody because graph-mode observations include fused and coasted contacts, so even random tips benefit from the environment-side track layer; local untrained PPO does not.
Return increases for both methods. Shared-policy PPO saturates near the greedy heuristic (–49%). Hierarchical PPO reaches the privileged/visibility ceiling by iteration 2 and then improves return through fuel/maneuver discipline. Matching the hierarchical heuristic is the success criterion; exceeding physics is not claimed.
5.3. Sensitivity and Ablation
5.4. Qualitative Rollout
Figure 7.
Example hierarchical-heuristic rollout (rollout_tracks.png): ground tracks of satellites and targets (top) and custody intervals per target (bottom).
Figure 7.
Example hierarchical-heuristic rollout (rollout_tracks.png): ground tracks of satellites and targets (top) and custody intervals per target (bottom).

6. Discussion
When reports are clean and links are global, local policies already sit near the visibility ceiling (EASY). When reports are cluttered and links are intermittent, local policies are limited by the local true-detection rate. Graph fusion supplies multi-satellite agreement when connectivity exists; short track coast preserves pointing through unique-FOV and ghost frames. Hierarchical PPO with masked attention learns to use that graph observation and matches the hand-designed hierarchical heuristic.
Limitations: the false-positive model is simulated; coast state lives in the environment and is reflected in observations (this paper does not claim the network invented a track file from a purely local Markov stream); is small; in the reported sweeps. The unconstrained shared-policy baseline [8] remains the appropriate comparison for the clean-sensor regime.
7. Conclusions
Under range-limited ISL, packet dropout, and slot-level false-positive clutter, memoryless local policies for satellite custody remain near the local true-detection ceiling. A graph-conditioned controller that fuses agreeing neighbor contacts and coasts confirmed tracks recovers custody at the privileged geometry level, both as a heuristic and as hierarchical PPO. On an EASY preset without clutter the gap disappears. The contribution is a realistic failure mode, a measurable information ceiling, and a communication-aware hierarchical remedy that closes the gap without claiming to exceed physics.
Data Availability Statement
Code for the communication-aware hierarchical environment and trainers is provided as an isolated package (custody_hier) requiring Python 3.10+, PettingZoo, Ray RLlib, and PyTorch. Public release via GitHub is planned upon acceptance; contact the author for details.
Conflicts of Interest
This work was supported by the University of Alaska Fairbanks Geophysical Institute. The author has no conflicts of interest to report.
References
- Liu, X. Mission schedule of agile satellites based on Proximal Policy Optimization Algorithm. arXiv 2020, arXiv:2007.02352. [Google Scholar] [CrossRef]
- Siew, P.M.; Jang, D.; Roberts, T.G.; Linares, R. Space-based sensor tasking using deep reinforcement learning. J. Astronaut. Sci. 2022, 69, 1855–1892. [Google Scholar] [CrossRef]
- Siew, P.M.; Smith, T.; Ponmalai, R.; Linares, R. Scalable multi-agent sensor tasking using deep reinforcement learning. In Proceedings of the Proceedings of the Advanced Maui Optical and Space Surveillance (AMOS) Technologies Conference, 2023; p. 3. [Google Scholar]
- Zhang, B.; Wu, Y.; Zhao, B.; Chanussot, J.; Hong, D.; Yao, J.; Gao, L. Progress and challenges in intelligent remote sensing satellite systems. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2022, 15, 1814–1822. [Google Scholar] [CrossRef]
- Zhang, G.; Li, X.; Hu, G.; Li, Y.; Wang, X.; Zhang, Z. MARL-based multi-satellite intelligent task planning method. Ieee Access 2023, 11, 135517–135528. [Google Scholar] [CrossRef]
- Holder, J.; Jaques, N.; Mesbahi, M. Multi agent reinforcement learning for sequential satellite assignment problems. Proc. Proc. AAAI Conf. Artif. Intell. 2025, Vol. 39, 26516–26524. [Google Scholar] [CrossRef]
- Zheng, Y.; He, T.; Ma, Y.; Liu, X. Improving Arctic surface radiation estimation using a nonlinear perturbation model with a fused multi-satellite cloud fraction dataset. Atmos. Chem. Phys. 2026, 26, 3321–3338. [Google Scholar] [CrossRef]
- Schuster, D. Multi-Satellite Custody of Moving Ground Targets. J. Comput. Sci. 2026, 22, 1139–1144. [Google Scholar] [CrossRef]
- Hady, M.A.; Hu, S.; Pratama, M.; Cao, Z.; Kowalczyk, R. Multi-agent reinforcement learning for resources allocation optimization: a survey. Artif. Intell. Rev. 2025, 58, 354. [Google Scholar] [CrossRef]
Figure 1.
HARD preset mean custody for the main controllers produced by the experiment harness (hard_comparison.png). Local methods remain near the true-detection ceiling; graph-conditioned and privileged controllers recover the visibility ceiling. Trained PPO bars appear when checkpoints are available.
Figure 1.
HARD preset mean custody for the main controllers produced by the experiment harness (hard_comparison.png). Local methods remain near the true-detection ceiling; graph-conditioned and privileged controllers recover the visibility ceiling. Trained PPO bars appear when checkpoints are available.

Figure 2.
Training progress on HARD (training_curves.png). Left: mean custody versus PPO iteration for shared-policy PPO (local) and hierarchical communication-graph PPO. Right: episode return. Horizontal references indicate local greedy and hierarchical-heuristic levels. Iteration 0 is untrained weights.
Figure 2.
Training progress on HARD (training_curves.png). Left: mean custody versus PPO iteration for shared-policy PPO (local) and hierarchical communication-graph PPO. Right: episode return. Horizontal references indicate local greedy and hierarchical-heuristic levels. Iteration 0 is untrained weights.

Figure 3.
Custody versus ISL range on HARD (custody_vs_comm_range.png). Heuristic controllers only. Track coast is local once a track exists, so unique-FOV custody does not require continuous connectivity at every step.
Figure 3.
Custody versus ISL range on HARD (custody_vs_comm_range.png). Heuristic controllers only. Track coast is local once a track exists, so unique-FOV custody does not require continuous connectivity at every step.

Figure 4.
Custody versus slot-level false-positive rate on HARD (custody_vs_noise.png). Local greedy falls with clutter; graph+coast remains high until false positives become severe enough to prevent clean track bootstrap.
Figure 4.
Custody versus slot-level false-positive rate on HARD (custody_vs_noise.png). Local greedy falls with clutter; graph+coast remains high until false positives become severe enough to prevent clean track bootstrap.

Figure 5.
Scalability on HARD for (scalability.png). Heuristic controllers only. Mean custody becomes harder as the target count grows with constellation size.
Figure 5.
Scalability on HARD for (scalability.png). Heuristic controllers only. Mean custody becomes harder as the target count grows with constellation size.

Figure 6.
Ablation on HARD (ablation_bars.png): default controllers plus hierarchical heuristic with global ISL and with clutter removed, and greedy with clutter removed. Removing clutter lifts local greedy substantially but not all the way to the EASY ceiling while false negatives remain.
Figure 6.
Ablation on HARD (ablation_bars.png): default controllers plus hierarchical heuristic with global ISL and with clutter removed, and greedy with clutter removed. Removing clutter lifts local greedy substantially but not all the way to the EASY ceiling while false negatives remain.

Table 1.
Heuristic custody on HARD (, seeds 11/22/33). All rows are non-learned controllers.
| Method | Mean custody | Notes |
|---|---|---|
| Random | heuristic | |
| Greedy (local) | heuristic | |
| Hierarchical heuristic | graph fusion + coast | |
| Privileged | geometry oracle |
Table 2.
Shared-policy PPO on HARD (local observations). Iteration 0 is untrained weights.
| Iteration | Custody | Episode return |
|---|---|---|
| 0 (untrained) | — | |
| 1 | ||
| 2 | ||
| 6 | ||
| 16 |
Table 3.
Hierarchical communication-graph PPO on HARD. Iteration 0 is untrained weights.
| Iteration | Custody | Episode return |
|---|---|---|
| 0 (untrained) | — | |
| 1 | ||
| 2 | ||
| 4 | ||
| 7 | ||
| 20 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.