Directional Pheromone Gradient Observations for Decentralized Multi-Agent Reinforcement Learning in Swarm Drone Search and Rescue

1. Introduction

Natural and man-made catastrophes result in hazardous or impenetrable terrain for humans to conduct rescue missions. Quick localization of survivors is key, as research shows less than 50% of rescue probability within 24 hours after a structural collapse [1]. Unmanned aerial vehicles (UAVs) can swiftly deploy, capture thermal images, and enable communication despite broken ground-level facilities [2].

One drone alone would be incapable of handling a vast area due to battery limits, sensing range constraints, and the possibility of broken communication. Multiple drones can share tasks [3,4]; however, a fundamental question emerges in such an application scenario: how can many drones coordinate themselves effectively without having a delicate central controller when communication is broken?

This problem has been previously solved in one of two ways, namely, stigmergic principles [5], which enable indirect communication through leaving environmental traces similar to those of social insects, and multi-agent reinforcement learning (MARL) [6]. Stigmergy is efficient in communication costs but not adaptive since it requires fixed rules. MARL is capable of learning policies that can adapt to changing circumstances, yet is burdened by non-stationarity and sample inefficiency [7,8].

The critical idea presented by this paper is that by giving directional pheromone gradients as inputs to the RL observation, we can connect both worlds: stigmergy allows for the encoding of structured low-bandwidth coordination memory, whereas RL learns how to interpret and make use of this memory flexibly. Instead of considering the pheromone values as environmental variables, our design of the observation allows us to encode the direction of the victim information and unexplored areas.

The main research question is:How do directional pheromone gradient observations and neighbor density awareness improve decentralized MARL coordination for multi-drone SAR in uncertain disaster environments?.

The manuscript is organized as follows: Section 2 reviews related work. Section 3 presents the mathematical background. Section 4 describes the system and methodology. Section 5 reports results including ablation and communication loss experiments. Section 6 concludes.

The main contributions are:

• The decentralized multi-agent reinforcement learning pheromone gradient observation framework, where each drone observes pheromone fields indicating victim-likelihood and coverage. Agents do not need a global map, they coordinate their search/exploration according to the field patterns in the local area.
• A multiple-layer virtual pheromone model. These layers represent specific environmental factors, such as searched areas, possible victim locations, hazard zones, and communication quality. The layers cannot be maintained if they exist alone and provide a common environmental memory for the swarm without a centralized mapping. This is appropriate for the situation in CTDE (Training with richer information and decentralized execution).
• A hybrid stigmergy-MARL (indirect coordination from biological systems and reinforcement learning). Agents learn to improve their behavior using reward signals and simple low-bandwidth environmental cues, while several baselines are used to benchmark against, feature ablations are used to evaluate the framework, and communication failures are used to test it.
• The purpose of these experiments is to evaluate the effectiveness and robustness of the observation design under disaster search and rescue environment.

2. Literature Review

2.1. Swarm Robotics

In swarm robotics, the behavior of large numbers of simple robots is studied as a useful collective behavior, arising from local rules and decentralized interaction [9]. The field is inspired by swarm intelligence proposed by Beni and Wang [10].These systems are attractive because they’re easy to use: They scale, they’re resilient to failure, and they can handle several actions simultaneously. That last is more important than usual during disaster response. If one drone is down then the mission should not be stopped [11].

2.2. Execution and Swarm: Lightweight Models for Real-Time Autonomous Systems

In real swarm deployments, however, execution and swarm: Lightweight models for real-time autonomous systems coordination is just one of the issues. The models that operate on each agent must also be small and robust in the presence of noise and fast enough to track the environment. This is a common limitation in swarm robotics, autonomous vehicles, and distributed sensing, where agents typically don’t have a lot of compute and may not be able to wait.

Using demographic data, Shalash et al.  [12] construct a multimodal deception detection system based on physiological signals, BPM, GSR and temperature. The accuracy of a Random Forest classifier optimized by Particle Swarm Optimization is 97% with an average inference time of 0.06 s, a significant improvement over those of the other classifiers, which demonstrates the ability of swarm-based optimization to produce fast and generalizable models even in sensitive detection tasks. Sallam et al. [13] on the other hand approach the Intelligent Dental Handpiece (IDH), a real-time dental skill assessment based on IMU. Both Logistic Regression and Random Forest achieved 100% accuracy at inference times under one millisecond, showing that lightweight ML is suitable for supporting precision-critical feedback for human-robot interaction.

In this work, Salah et al. [14] discuss another constraint, speaker verification in the context of an autonomous vehicle with stringent latency demands. Their Siamese network, which has both BiLSTM and Transformer encoders, achieves an EER of 0.0685 and an inference time of less than 0.2 s, a significant improvement over previous work. These three studies have something in common: in decentralized systems, it is not an option to have efficient model design, sensor fusion, or real-time execution. These are the minimum requirements.Shalash et al. [12] present an optimized polygraph-based deception detection system using multimodal physiological signals, including BPM, GSR, and temperature, together with demographic features. The data are processed with a Random Forest classifier that reaches 97% accuracy and 0.06 s inference latency. Particle Swarm Optimization (PSO) is used for hyperparameter tuning; it reduces the training-validation accuracy gap to under 1%. The authors also release a multimodal dataset collected from 49 participants. This work shows how swarm-inspired tuning can improve model generalization while keeping execution fast and resource-efficient, which is important in security and forensic applications where decisions must be quick and reliable.

Sallam et al. [13] introduce the Intelligent Dental Handpiece (IDH), an IMU-based motion analysis system for real-time skill assessment in dental training. The system classifies three motion states: Alert, Lever Range, and Stop Range. It uses features such as Deviation, Take Time, and Device type. Logistic Regression and Random Forest models both achieve 100% accuracy with sub-millisecond inference, allowing immediate visual and auditory feedback. The main point is direct: lightweight ML can support precision-critical human-machine interaction. This capability also transfers to swarm robotics, where individual agents need to carry out coordinated, sometimes dexterous actions with low latency.

Salah et al. [14] propose a lightweight speaker verification framework for autonomous vehicles. Their method combines Siamese networks with either BiLSTM or Transformer encoders, using triplet or quadruplet loss functions. On the LibriSpeech dataset, the best model, BiLSTM with triplet loss, achieves an equal error rate of 0.0685 and an inference time of 0.183 s, improving on prior work with an EER of 0.11. This highlights the need for efficient low-latency authentication in vehicular swarms, where secure identity verification must happen quickly to support personalized settings, command authentication, and driver monitoring without reducing safety or responsiveness.

Together, these studies show the same pattern. Scalable decentralized swarm intelligence needs efficient model design, sensor fusion, and real-time execution. Not as extra features, but as core requirements in dynamic environments.

2.3. Stigmergy and Virtual Pheromones

Stigmergy and Virtual Pheromones: Stigmergy is a type of coordination without direct communication between agents. Rather, they change a common environment, and other agents respond to the changes made  [5,15]. The idea is based on social insects, bees and ants, in which no central authority is in charge of the group and it arises from simple local actions [16]. The agents leave a trace and the swarm reads this trace to make adjustments.

The most popular computational implementation of this idea is Ant Colony Optimization (ACO) [16].Useful paths and distributed route information develop over time as ants leave pheromones on them that will affect subsequent ants [17]. In swarm robotics, the idea is taken to a different level and the physical pheromone fields are replaced by virtual pheromone fields, which are stored, modified, and sensed by the robots digitally.

The virtual pheromone can be used in the search and rescue (SAR) application, where it can facilitate the decentralized control without requiring the ongoing communication among agents [18]. Agents can tag explored areas, probable victim locations, or unsafe locations, and other agents within that area can update their behaviors accordingly. However, the problem in the present SAR systems is that the majority of the existing ones follow a fixed rule and some of the primitive pheromone design. It reduces their adaptability, particularly in a mission that demands exploring, locating victims and avoiding the dangers in fluctuating environments.

2.4. Reinforcement Learning and MARL for Coverage

Reinforcement Learning has been providing autonomous agents with a means to learn decision-making by interacting with the environment [19]. Multi-agent reinforcement learning (MARL) extends this to multi-agent systems with multiple agents needing to learn to cooperate with each other in a partially observed environment with decentralized execution. The combination of that makes MARL suitable for UAV search, exploration and coverage as agents the operate in dynamic environments that are uncertain and partially visible.

A few studies have focused on directly addressing multi-UAV coordination using MARL. Pham et al.  [7] presented a cooperative distributed RL approach for area-searching behavior in UAV field coverage, demonstrating that a set of multiple agents could learn coordinated area-searching behavior from local interaction without the need for centralized control. Later, MARVEL combined graph attention mechanisms into a MARL framework to enhance coordination for multi-robot environments with a large number of robots [8]. These works highlight the ability of MARL to learn complex cooperative behaviors without relying on handcrafted control policies.

The growing relevance of MARL for UAV coordination is reflected in recent surveys. Ekechi et al. identified MARL as a promising approach for UAV control in dynamic and uncertain environments [20], while Blais and Akhloufi reviewed advances in RL-based swarm robotics and emphasized the increasing role of learning-based coordination in multi-robot systems [21]. In disaster-response scenarios, Lee et al. investigated reinforcement learning for UAV-assisted emergency communication [22], and Tan and Zhao applied deep reinforcement learning to coordinate multiple UAVs for post-disaster search-and-rescue operations [1].

Despite these advances, most MARL-based approaches learn coordination solely from agent observations or explicit communication mechanisms. Relatively little attention has been given to incorporating structured environmental memory into the learning process. This is what leads to the need for agents to relearn useful coordination patterns through long training or rely on communication methods that may fail in disaster settings. This creates a clear need to combine stigmergic environmental cues with learned policies. Agents can then use shared information left in the environment while still keeping the flexibility of reinforcement learning.

2.5. Representation Learning in MARL

Representation Learning in MARL Methods for encoding observations directly influence coordination, their robustness and speed of convergence. Raw observations are sometimes not enough. Recent research in MARL has progressed towards the development of structured representations with spatial, temporal, or relational attributes, which enable agents to process partial observability, non-stationarity, and sparse exploration signals.

Hamdi et al. [23] introduce a multimodal attentional representation learning framework which integrates visual and temporal data. CNNs are used to extract features from the visual data, LSTMs to identify temporal patterns, and an attention mechanism to allow the model to concentrate on the most informative inputs. The multi-modality joint representation-based learning outperforms single-modality baselines, achieving up to 97% accuracy in classification and a 19%-57% boost in prediction performance. This finding is consistent with a larger hypothesis: when there are unclear or inconsistent observations, multimodal learning with attention is beneficial.

Li et al. [24] propose RACE that decouples the representation learning from policy learning in a MARL system. The agents are equipped with a shared observation encoder, while they have different policy representations, this means that transferring knowledge doesn’t result in the same behaviour being enforced. RACE deals with the challenges of partial observability by maximizing value-aware mutual information, thus learning features that contain globally relevant information. In several benchmark tasks, it achieves better coordination quality and convergence rate than baseline MARL approaches. The main point is clear: structured representations plus information-aware objectives can make multi-agent learning more effective.

Xu et al. [25] focus on multi-view representation learning, where data from several sources are fused into a more complete representation. Their Progressive Deep Multi-view Fusion (PDMF) method uses two stages. It consists of pre-training to capture relationships between views, then fine-tuning to map partial representations into complete ones. The model captures both consistency and complementarity across views which fixes some limits of standard fusion methods. Experiments on multiple datasets show steady gains over state-of-the-art approaches with classification accuracy improving by up to several percentage points. This supports the value of structured feature integration and progressive learning for better robustness and generalization.

Together, these studies show that stronger representation learning, through multimodal fusion, shared encoders, or multi-view integration, can improve learning efficiency, stability, and coordination in complex environments. For MARL, this supports the use of structured observation representations, where richer inputs can directly shape better policy performance.

2.6. Communication and Coordination in MARL

Communication and coordination are still hard problems in multi-agent reinforcement learning (MARL) especially when the system becomes larger, agents see only part of the environment and bandwidth is limited. Recent work has started to use learnable communication methods like dynamic coordination set selection, noise-aware communication and shared memory structures [26]. These methods do not treat communication as a fixed channel. Instead, they learn when agents should communicate, what information they should send, and how it should be shared.

Zhang and Lesser [27] propose a utility-based method helping agents find useful coordination sets during learning and reducing communication cost while keeping performance. Tung et al. [28] study learning-to-communicate over noisy physical channels where policies and channel coding are learned together rather than separately. Pesce and Montana [29] introduce a memory-based communication architecture that lets agents form emergent protocols for tasks that need close synchronization. Together, these works show a clear shift in MARL where agents coordinate better when communication and policy learning are designed jointly.

Zhang and Lesser [27] deal with scalability by allowing agents to find useful coordination sets while they learn. Their method uses an interaction measure to estimate how much performance may be lost if agents do not coordinate. Agents communicate only when coordination is important. This reduces overhead while keeping performance close to the full-communication setting. In sensor network target tracking the method reduces communication by more than 80% with only a small performance drop. This shows the trade-off between performance and bandwidth use.

Tung et al. [28] study communication under noisy physical channels. They formulate the problem as a MARL setting, where noisy channels are part of the environment dynamics rather than an external issue. Their MA-POMDP model moves beyond the common error-free communication assumption and learns cooperative policies together with robust channel coding. Experiments on binary symmetric, AWGN, and bursty noise channels show that joint learning performs better than separation-based methods. In joint channel coding, the learned schemes outperform Hamming codes by up to 3.70 dB in block error rate.

Pesce and Montana [29] propose MD-MADDPG which uses a shared memory device as a learnable communication channel under the centralized training and decentralized execution paradigm. Agents learn gated read and write operations so they can update and interpret memory while also learning their policies. Tested on six coordination-heavy navigation tasks, MD-MADDPG performs better than MADDPG and other baselines, mainly in synchronization tasks. The memory visualizations show task-related usage patterns, and the ablation results confirm that the main architectural components are needed.

2.7. UAVs in Disaster SAR

In SAR missions, UAVs are highly important since they enable fast exploration of hazardous, damaged, or difficult-to-reach areas without the danger to human lives of the searchers. According to recent literature, UAVs aid SAR operations with respect to rapid deployment, aerial surveillance, sensor-based victim identification, and operating in restricted areas [30].

The use of multiple UAVs enhances the performance of SAR tasks through the parallel exploration of various regions. Yet, several issues arise when employing multi-UAVs, including mission allocation, avoiding collisions, limited battery capacity, and poor communication in disaster zones. According to Ghauri et al. [31], the problem of dynamic mission allocation remains one of the critical issues for multiple UAVs conducting SAR activities.

2.8. Research Gap

Existing MARL approaches focus on learning coordination through policy optimization or explicit communication, while stigmergy-based systems rely on fixed rules without adaptability. However, little work explores structured, directional representations of environmental memory that can guide decentralized policies without communication overhead.

In particular, current approaches either depend on learned communication strategies, which introduce additional system complexity, or rely on static stigmergic rules that lack adaptability in dynamic environments. This creates a gap in developing methods that retain the efficiency of stigmergy while enabling adaptive, learned coordination.

These particular gaps prompt this research in three ways. First, Salman et al. [18] perform automated stigmergy behavior design yet lack the learning mechanism whereby their system cannot prioritize adaptively. Second, Tan and Zhao [1] implement a deep RL strategy for multiple UAVs in SAR tasks using a centralized approach that falls short when the base station gets compromised. Third, Pham et al. [7] and MARVEL [8] illustrate the use of reinforcement learning techniques in multi-robot task accomplishment while excluding the use of pheromones to communicate past actions. This means that the robots have to learn how to coordinate with each other based solely on observation. To the best of our knowledge, this is among the first SAR awareness under a CTDE-based MARL setting.

3. Methodology

The methodology is designed to isolate the impact of directional pheromone gradient observations on decentralized coordination performance, ensuring that all other components of the system remain consistent across compared approaches.

3.1. Problem Formulation and Environment

Disaster search and rescue (SAR) operations are inherently stochastic and partially observable. The locations of victims are not known, there may be obstacles in the way, and communication between agents may be faulty. Hence, the problem is formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where each UAV makes decisions based on its own local observations.

The disaster site is modeled by a grid G, with each square $c\in G$ possibly being free, obstructed, dangerous, searched, or occupied by a victim. At each time step t, UAV i has a local observation $o_i^t$, current location $p_i^t$, current battery life $b_i^t$, and an action $a_i^t$. Each agent can see within its own local window, with sensing range $r_s=2$.

The environment is generated procedurally using a fixed random seed per episode. Obstacles, hazard zones, and victim locations are sampled uniformly at random without replacement from all grid cells outside a $9\times 9$ central exclusion zone reserved for drone initialisation. Obstacles are impassable; hazard zones are traversable but penalise entry (reward $-\eta$) and deposit risk pheromone ${\tau }_r$. Both are static for the duration of each episode — neither obstacles nor hazard zones move. Victims are likewise stationary; their locations are unknown to agents until physically detected within sensing range ($r_s=2$). The drones are uniformly randomized in the central $7\times 7$ square. No two drones occupy the same grid point in the starting position. Each of the 30 iterations uses a distinct map so that the performance on different maps can be measured. The entire source code of the simulation is provided along with this paper.

3.2. System Architecture

The proposed system has five layers, shown in Figure 1. The sensing layer collects RGB, thermal, LiDAR, IMU, and communication-signal data. The mapping layer builds local occupancy grids and victim-probability grids, then merges them across drones when communication is available. The stigmergy layer maintains and spreads the four pheromone maps using Equation (1). The learning layer runs the MARL policy. An optional mission-control layer tracks global progress and assigns area priorities. When it is disconnected, the system still runs in a fully decentralized way.

The proposed observation formulation is independent of the specific environment structure and can be applied to any decentralized MARL setting where spatial information is represented through local maps or distributed environmental fields.

3.3. Virtual Pheromone Model

To support indirect coordination, the proposed system uses four virtual pheromone layers inspired by stigmergy. The coverage layer ${\tau }_c$ marks scanned cells, the victim-likelihood layer ${\tau }_v$ stores possible human-presence evidence, the risk layer ${\tau }_r$ marks hazardous regions, and the communication layer ${\tau }_m$ identifies useful relay areas.

Each pheromone layer is updated through evaporation, diffusion, and new deposits:

$${\tau }_k^{t+1}\left(c\right)=(1-{\rho }_k){\tau }_k^t\left(c\right)+D_k\sum _{c^{\prime }\in \mathcal{N}\left(c\right)}\left[{\tau }_k^t\left(c^{\prime }\right)-{\tau }_k^t\left(c\right)\right]+\Delta {\tau }_k^t\left(c\right),$$

(1)

where $k\in \{c,v,r,m\}$, ${\rho }_k$ is the evaporation rate, $D_k$ is the diffusion coefficient, $\mathcal{N}\left(c\right)$ is the four-cell neighborhood, and $\Delta {\tau }_k^t\left(c\right)$ is the new deposited pheromone. The victim-likelihood layer uses the slowest evaporation rate, ${\rho }_v=0.02$, so possible victim evidence remains available for longer.

3.4. Directional Pheromone Gradient Observation

The central design contribution is an enriched observation vector. Each drone i observes:

$$o_i=\left({\tau }_c,{\tau }_v,{\tau }_r,{\tau }_m,b_{\mathrm{norm}},{\nabla }_v,{\nabla }_c,{\rho }_{\mathrm{neigh}},w_N,w_S,w_W,w_E\right)$$

(2)

where ${\nabla }_v=({\overline{\tau }}_v^N,{\overline{\tau }}_v^S,{\overline{\tau }}_v^W,{\overline{\tau }}_v^E)$ is the directional victim-pheromone gradient. It is computed as the mean ${\tau }_v$ over a 3-cell cone in each cardinal direction. The term ${\nabla }_c$ is computed in the same way, but for the coverage pheromone. ${\rho }_{\mathrm{neigh}}$ gives the normalized number of drones within $r_c$, while $w_d$ are binary obstacle indicators for the four directions. Neighbor density is taken from physical drone positions, so it gives short-range proximity awareness without active message exchange. It remains available even when communication is lost.

This directional representation reduces decision ambiguity. With only scalar pheromone values at the current cell, a drone knows what is happening where it stands, but not where the useful signal is stronger. It leads to inefficient exploration or back and forth moves under partial observability conditions. Representation by directional trends ${\nabla }_v$ and ${\nabla }_c$ provides a straightforward hint to the policy about where more victims’ signs are observed or, alternatively, where there is less coverage. In such a way, drones can explore more purposefully without the need for a global map or additional communication.

The detection of a victim takes place based on a deterministic binary scheme. The presence of a victim at the grid cell $(r,c)$ is determined if the drone visits any cell $(r^{\prime },c^{\prime })$, for which $max\left(\right|r^{\prime }-r|,|c^{\prime }-c\left|\right)\le r_s$. This means that the victim is located within a drone’s $5\times 5$ sensing window, where $r_s=2$. Upon detection, the value of $\Delta {\tau }_v=1.5$ is added to the drone’s current position, and the victim is considered to be discovered. Positions of victims are unknown to all agents in the beginning of each episode. They are not included in the initial state or observation. The model abstracts a short-range thermal or visual sensor. A probabilistic detection model with distance-based false alarms is left for future work.

Overall, the added directional features make the observation more informative than scalar pheromone inputs alone. They let each agent infer local spatial gradients of coverage and victim likelihood, reduce ambiguity in action selection, and support more stable navigation in partially observable environments.

These features add more than scalar pheromone values alone. If a drone only observes ${\tau }_v$ at its current cell, it knows there is some victim-related signal nearby, but not where that signal is coming from. With ${\nabla }_v$, it can move toward the strongest victim-likelihood direction without needing a global map or explicit message passing.

Unlike conventional methods based only on scalar local observations the proposed representation encodes directional trends. This lets each agent infer spatial gradients of relevant environmental signals, reduces ambiguity in action selection and supports more purposeful navigation under partial observability.

Similarly, ${\nabla }_c$ encodes which direction has the least explored space, enabling learned dispersion without inter-agent negotiation. The neighbor density ${\rho }_{\mathrm{neigh}}$ allows the policy to avoid redundant clustering even when no explicit task-assignment protocol exists. An ablation study in Section V quantifies the individual contribution of each feature.

The proposed directional features introduce minimal computational overhead, as gradient values are computed from local pheromone maps using simple averaging operations over small spatial neighborhoods. A comparison between conventional scalar observations and the proposed directional representation is summarized in Table 1.

3.5. Hybrid Decision Process

Algorithm 1 formalizes the per-step procedure.

Algorithm 1  Hybrid Stigmergy-RL Step (drone i, step t)

Require: Observation $o_i^t$, pheromone maps $\left\{{\tau }_k\right\}$, Q-table $Q_i$, $\epsilon$ Ensure: Action $a_i^t$, updated pheromone deposits 1: Observe local environment within radius $r_s$ 2: Deposit $\Delta {\tau }_c+=0.8$ for scanned cells 3: if victim at $(r,c)$ within sensing window (deterministic, $r_s=2$) then 4:    $\Delta {\tau }_v(r,c)+=1.5$ 5: end if 6: if hazard detected at $(r,c)$ then 7:    $\Delta {\tau }_r(r,c)+=1.0$ 8: end if 9: Compute ${\nabla }_v$, ${\nabla }_c$ from 3-cell directional cones 10: Compute ${\rho }_{\mathrm{neigh}}$ from drones within $r_c$ 11: Construct $o_i^t$ from Equation (2) 12: Select $a_i^t$ via $\epsilon$-greedy on $Q_i(o_i^t,·)$ 13: Execute $a_i^t$; receive reward $R_t$ (Equation (3)) 14: Update: $Q_i(s,a)+={\alpha }_{\mathrm{RL}}[R_t+{\gamma }_{\mathrm{RL}}{max}_{a^{\prime }}{Q}_i(s^{\prime },a^{\prime })-Q_i(s,a)]$ 15: if drone j within comm. radius $r_c$ then 16:    Broadcast compressed $\Delta {\tau }_k$ to drone j 17: end if 18: Apply evaporation-diffusion (Equation (1))

3.6. Non-Learned Baselines

Three non-learned baselines are included for comparison. Random walk selects actions uniformly at random each step. Frontier search directs each drone to the nearest unvisited cell using Manhattan distance, with random tie-breaking. Stigmergy-only uses fixed hand-coded rules based solely on the shared pheromone field, with no learning: at each step, each drone scores the four cardinal moves using the linear combination $s=3{\tau }_v-2{\tau }_c-1.5{\tau }_r-2{\delta }_{\mathrm{crowd}}$, where ${\tau }_v$, ${\tau }_c$, ${\tau }_r$ are the pheromone values at the candidate cell and ${\delta }_{\mathrm{crowd}}$ penalises cells occupied by other drones. A small uniform noise term $U(-0.05,0.05)$ breaks ties. The drone moves to the highest-scoring valid cell, or hovers if all moves are blocked. This rule prioritises victim-likelihood signals over coverage and avoids hazards and crowding — but the weights are fixed and cannot be adapted at runtime.

3.7. Training Strategy

$$R_t=\alpha R_{\mathrm{victim}}+\beta R_{\mathrm{cov}}+\gamma R_{\mathrm{comm}}-\lambda R_{\mathrm{energy}}-\mu R_{\mathrm{coll}}-\eta R_{\mathrm{risk}}$$

(3)

Victim detection and cell creation are rewarded positively; energy consumption, collision, and entry into the danger zone are penalized. The victim coefficient $\alpha =5.0$ takes precedence over all others, guaranteeing that the drone algorithm favors victim detection above cell creation. The communication reward $\gamma$ ensures that the drones will stay in relay formation when no other high-priority task is available.

Tabular Q-learning is chosen deliberately for these experiments. It allows complete control of the observation space and makes it possible to separate the contribution of the suggested representation. It also helps avoid additional sources of noise caused by function approximation, making performance improvements associated solely with directional gradient representations easier to attribute.

The training regime adheres to the CTDE setup. During the training the simulator generates the global state used to calculate the reward, while during evaluation each drone selects actions based only on its individual observation. The Q-tables have a 6-bin quantization for each pheromone-relevant observation feature and are initialized in an optimistic way ($+2.0$) to promote exploration early in the training process. To avoid conflict with the victim reward weight $\alpha$ in Equation (3) the Q-learning step size is written as ${\eta }_{\mathrm{RL}}=0.18$ in Table 2. Exploration uses multiplicative decay, ${\epsilon }_{t+1}={\epsilon }_t\times 0.9996$, applied once per episode with a minimum value of 0.05. Across 5,000 training episodes, $\epsilon$ drops from 1.0 to about 0.135 so exploration does not vanish. Each learned method is trained with five independent random seeds and results are reported as mean ± std across seeds to separate training-seed variance from evaluation variance. Statistical significance is tested using Welch’s t-test on pooled evaluation runs ($n=150$) along with Cohen’s d effect size and bootstrap 95% confidence intervals. For a fair comparison all models use the same reward function, training process and environment conditions. However, only the observation representation changes.

The RL-only baseline uses the same learning rate, discount factor, exploration schedule, training length and evaluation protocol as the hybrid model. It differs only in the observation or policy state. Table 3 gives the exact features for each model. Both models share five base features: local coverage pheromone ${\tau }_c$ with 6 bins, local victim pheromone ${\tau }_v$ with 6 bins, a binary risk flag from ${\tau }_r$, normalized battery with 4 bins and four binary obstacle indicators. The hybrid model adds three more features: victim-gradient direction ${\nabla }_v$, defined as the dominant cardinal direction of ${\tau }_v$; coverage-gradient direction ${\nabla }_c$, defined as the direction of least-explored space; and neighbor density ${\rho }_{\mathrm{neigh}}$ with 3 bins. The reported improvements are therefore attributable solely to these three added observation features. The RL-only baseline is intentionally matched to the hybrid in architecture, reward function, and training procedure, differing only in observation features (Table 3), so that any performance difference is attributable to the three added gradient features rather than to architectural differences.

The state space increases from 9 to 18 dimensions; however, due to discretization, the resulting Q-table remains computationally tractable within the evaluated environment size.

4. Results and Discussion

4.1. Training Convergence

Figure 2 shows the training reward smoothed with a 100-episode uniform moving average over 5,000 episodes. The hybrid model shows faster and more stable reward improvement than the RL-only baseline, indicating that directional pheromone-gradient observations provide useful structure for learning coordinated search behavior. This supports the role of pheromone-guided observations in reducing sample complexity compared with learning from local state features alone.

Not only achieving better performance at the end but this hybrid approach also converges much faster. Looking at the learning curve we see that it reaches stable behavior sooner compared to the RL-only baseline which implies that the knowledge of directional gradients helps the agent learn more efficiently.

4.2. Coverage and Victim Detection

Figure 3 depicts area coverage over time for all five approaches in 30 trials with $\pm 1\sigma$ confidence intervals. Evaluation results are given in Table 4. Hybrid agent achieves 98.9% area coverage and 93.3% victim detection, outperforming the pure RL approach by 17.1 pp and 21.6 pp respectively. Welch’s t-test shows that these gains are significant for both coverage ($t=18.88$, $df=168$, $p<0.001$, Cohen’s $d=2.18$) and victim detection ($t=16.49$, $df=218$, $p<0.001$, Cohen’s $d=1.90$). In addition, the coverage variance reduces from $\pm 10.8\%$ with RL-only approach to $\pm 2.7\%$ with the hybrid one indicating increased stability in decentralized execution due to observations of pheromone gradients.

4.3. Comparative Analysis

Benchmark results are shown in Figure 4. Hybrid algorithm provides 98.9% coverage and 93.3% victims detection greatly outperforming RL-based only algorithm providing 81.8% coverage and 71.7% victim detection. This is equivalent to gains of 17.1 percentage points in coverage and 21.6 percentage points in victims detection.

Stigmergy-only approach is also quite good achieving 86.5% coverage and 77.7% victims detection. Its fixed exhaustive exploration rules help it achieve better coverage than both random walk and RL-only, although it still remains below the hybrid model overall. However, unlike the learned hybrid model, it cannot adapt its behavior at runtime and therefore cannot flexibly prioritize changing mission objectives, energy constraints, or hazard conditions. The hybrid method outperforms stigmergy-only by 12.4 pp in coverage and 15.6 pp in victim detection while maintaining a fully learned, adaptive policy.

4.4. Ablation Study

To quantify the contribution of each proposed feature, we train and evaluate three variants: the full hybrid model, an ablated model without directional gradients (${\nabla }_v$, ${\nabla }_c$ removed), and an ablated model without neighbor density (${\rho }_{\mathrm{neigh}}$ removed). Results are shown in Figure 5 and Table 5.

Removing directional gradients reduces coverage by 16.7 pp (98.9% → 82.3%) and victim detection by 21.6 pp (93.3% → 71.7%). This confirms that directional victim-likelihood and coverage gradients are the dominant contributors to the hybrid model’s performance. Excluding neighbor density reduces coverage by 0.2 pp (from 98.9% to 99.2%) and victim detection by 1.0 pp (from 93.3% to 94.3%), suggesting that neighbor density awareness plays a role in performance, though is less important than gradient information.

4.5. Robustness Under Communication Loss

To evaluate robustness under realistic degraded-communication conditions, each drone maintains its own local pheromone map during evaluation, merging (element-wise maximum) with neighbours whose link is currently up. Both models were evaluated across a sweep of communication-link failure probabilities: 0%, 25%, 50%, 75%, and 100%. At each level, each active pairwise link is severed independently with the given probability, preventing map merging between the affected drones and forcing each to rely solely on its own locally accumulated pheromone evidence. Note that ${\rho }_{\mathrm{neigh}}$ is computed from physical drone positions rather than message exchange, and therefore remains available at all dropout levels; only pheromone-map sharing is gated by the dropout probability. Results are shown in Figure 6 and Table 6.

The hybrid model demonstrates strong robustness across the full sweep: coverage remains at 99.1% even at 100% link failure, a degradation of only 0.2 pp from the fully-connected baseline (99.3%). The RL-only agent similarly maintains stable performance, with both models relying on locally available pheromone information and learned policies that do not require active inter-agent coordination to function effectively. The hybrid model keeps a consistent $\approx 14$–15 pp coverage lead over RL-only at every dropout level. This suggests that pheromone-based stigmergy gives a structural coordination advantage, and that advantage remains even when communication is fully lost.

4.6. Limitations

Four limitations are acknowledged. First, pheromone parameters (${\rho }_k$, $D_k$) require per-layer tuning; misspecification may cause premature forgetting of victim evidence or persistent misleading traces. Adaptive online tuning via meta-learning is a natural extension. Second, the sim-to-real gap, GPS denial, sensor noise, aerodynamic disturbances, irregular 3D terrain, must be addressed before physical deployment; AirSim or Webots validation is the recommended next step. Third, tabular Q-learning may underperform deep function approximators (QMIX, MAPPO) on larger or more complex environments, and the absolute performance ceiling of both the hybrid and RL-only models is therefore lower than what deep MARL would achieve on this task. The RL-only baseline is intentionally matched in architecture to the hybrid to isolate the contribution of the directional gradient observation features; comparison against deep MARL baselines (QMIX, MAPPO, independent DQN) is deferred to future work. Fourth, safety-critical outdoor UAV operations require geofencing, fail-safe landing protocols, and regulatory compliance beyond the scope of this work. Fifth, while the communication-dropout sweep implements per-drone local pheromone maps with independent link failures, it does not model message delays, partial map corruption, or time-varying link quality. These represent important aspects of realistic disaster communication channels and are acknowledged as a current limitation of the robustness evaluation.

Additionally, the proposed representation is evaluated within a discretized state space using tabular reinforcement learning. While this setup allows controlled evaluation of the representation itself, further validation with deep function approximators and continuous observation spaces is required to assess scalability in more complex environments.

5. Conclusions

This paper introduced a directional pheromone gradient observation for decentralized swarm-drone disaster SAR combining stigmergy and MARL. The key insight is that feeding directional gradients of victim likelihood and coverage, rather than scalar local values, into the RL policy enables purposeful navigation toward victim evidence and unexplored regions without global map access or explicit inter-agent negotiation. Four independently parameterized pheromone layers provide evaporating shared memory that serves as a structured coordination substrate under the CTDE paradigm.

Experiments over 5,000 training episodes and 30 independent evaluation runs on a $40\times 40$ grid demonstrate that the hybrid model achieved 98.9% coverage and 93.3% victim detection. Compared with the RL-only baseline the hybrid model improves coverage by 17.1 pp and victim detection by 21.6 pp. The differences are statistically significant for both coverage ($t=18.88$, $df=168$, $p<0.001$, Cohen’s $d=2.18$) and victim detection ($t=16.49$, $df=218$, $p<0.001$, Cohen’s $d=1.90$). A similar conclusion can be reached from the ablative experiment where removal of the directional gradient leads to a coverage decrease of 16.7 pp and detection decrease of 21.6 pp. Communication-dropout experiments in the range from 0 to 100% link failures demonstrate that the hybrid solution preserves at least 99% coverage on all stages with consistent 14–15 pp gain over RL-only.

The future works are aimed at shifting away from the tabular Q learning to advanced techniques such as QMIX and MAPPO, along with neural pheromone encoders for continuous state spaces and large maps. Adaptive meta-learning of per-layer evaporation coefficients ${\rho }_k$ will be implemented to reduce the number of hyperparameters. The scalability analysis will be performed up to $100\times 100$ grid maps with up to 50 drones. More complex communication experiments will be carried out using time-varying dropouts, delayed messages, and partial map destruction rather than simple link-failure scenarios considered in this study. Physical validation via AirSim or Webots simulations along with field experiments will be useful for reducing the sim-to-real gap before deployment. The directional gradient observation is architecture-agnostic and directly transferable to any MARL framework receiving environmental map features.