ER-PASS: Experience Replay with Performance-Aware Submodular Sampling for Domain-Incremental Learning in Remote Sensing

1. Introduction

Recent advances in remote sensing technologies have enabled the acquisition of large volumes of imagery from diverse platforms, including satellites, drones, and airborne systems. This has led to the extensive application of remote sensing data in various fields such as urban planning [1,2,3,4], environmental monitoring [5,6,7], and disaster management [8,9,10]. Furthermore, the introduction of deep learning techniques has significantly enhanced the efficiency and accuracy of remote sensing image analysis, achieving state-of-the-art performance across a wide range of applications [11,12,13,14,15]. For instance, tasks such as building segmentation and land use/land cover (LULC) classification, which previously relied on manual interpretation or basic analytical methods, have seen remarkable improvements in both accuracy and computational efficiency through the application of deep learning [16,17,18]. Consequently, the research paradigm in remote sensing has shifted from traditional rule-based approaches to data-driven, deep learning-based methods.

Most deep learning studies in remote sensing have focused on improving task performance using fixed datasets under static experimental settings, with the development of state-of-the-art models becoming a dominant research trend. However, in real-world scenarios, imagery is continuously collected from various platforms, leading to significant variations in visual characteristics—such as object size, shape, and texture—as illustrated in Figure 1. Due to these domain-specific discrepancies, models trained on a single domain (i.e., a set of images from a specific platform) often suffer from severe performance degradation when applied to new domains [19,20]. This necessitates retraining the model for each new domain, which is not only computationally expensive but also risks catastrophic forgetting [21,22], where previously learned knowledge is lost during adaptation. These limitations hinder the practical deployment of deep learning-based remote sensing methods and highlight the need for more flexible and adaptive learning strategies. In this context, continual learning has emerged as a promising approach capable of mitigating both domain shift problems and catastrophic forgetting.

Continual learning is a learning paradigm in which a model incrementally learns from a sequence of tasks, aiming to integrate new knowledge without forgetting what has already been learned [23,24]. In this context, a "task" refers to a specific learning objective that arises sequentially over time. Based on learning scenarios, continual learning is typically categorized into Task-Incremental Learning (TIL), Class-Incremental Learning (CIL), and Domain-Incremental Learning (DIL) [25,26]. In TIL, each task has a distinct label space, and a task identifier is available during both the training and inference to specify which task the model should address. In contrast, CIL does not provide a task identifier during inference, requiring the model to classify all previously encountered classes jointly. DIL differs from the previous two scenarios by maintaining a shared label space across tasks, while varying the input distribution or domain characteristics. Among the three, DIL is particularly relevant to real-world remote sensing applications, where domain shifts frequently occur. Nevertheless, research on DIL remains relatively underexplored compared to the TIL and CIL. Therefore, this study focuses on addressing challenges within the DIL scenario.

In addition to scenario-based categorization, continual learning can also be classified based on implementation strategies: regularization-based, architecture-based, and replay-based methods [27,28]. Regularization-based methods preserve prior knowledge by incorporating additional terms into the loss function that penalize changes to parameters important for previous tasks. Architecture-based methods maintain prior knowledge by isolating parameters for each task, either by fixing and masking them during the training of new tasks or by using dynamic architectures that freeze existing parameters while expanding the model for task-specific learning. Replay-based methods, on the other hand, preserve prior knowledge more intuitively by explicitly storing a subset of data from previous tasks in a memory buffer and reusing it when training on new tasks.

Most existing studies on DIL have primarily employed regularization-based or architecture-based approaches, often citing concerns regarding data storage costs and privacy issues associated with replay-based methods [29,30,31]. However, in the field of remote sensing, long-term data collection is standard practice, making data storage costs a less critical concern. Furthermore, remote sensing imagery typically does not contain personally identifiable information, reducing privacy risks compared to other fields—except in sensitive areas such as national defense. Instead, the limitations of regularization-based and architecture-based methods pose significant challenges in practical remote sensing applications. Regularization-based methods are inherently vulnerable to domain shifts, which can lead to catastrophic forgetting. Meanwhile, architecture-based methods tend to increase model complexity as tasks accumulate, requiring additional memory resources. Given these factors, replay-based methods present a more practical solution for DIL within remote sensing.

Therefore, this study aims to develop a replay-based learning algorithm tailored for remote sensing in DIL scenarios. Specifically, we propose an experience replay with performance-aware submodular sampling (ER-PASS). ER-PASS is designed to improve the adaptability of deep learning models across remote sensing domains, while mitigating catastrophic forgetting and ensuring computational efficiency. To this end, ER-PASS integrates the robustness of joint learning—by processing samples from multiple domains as a unified input—with the efficiency of replay-based learning in reducing training time. Additionally, we introduce a performance-aware submodular sample selection strategy to enhance the stability of the learning process. To evaluate the generalizability of our approach, we conduct extensive experiments on two representative remote sensing applications: building segmentation and LULC classification. The main contributions of this study are as follows:

(1)

We propose a replay-based learning algorithm that incorporates a performance-aware submodular sample selection strategy, namely ER-PASS, which is model-agnostic and can be applied across various deep learning models.

(2)

We demonstrate that ER-PASS effectively mitigates catastrophic forgetting compared to existing methods, while requiring relatively low resource demands.

(3)

Experimental results on building segmentation and LULC classification demonstrate that ER-PASS exhibits generalizability across diverse remote sensing applications.

2. Related Work

2.1. Domain-Incremental Learning in Remote Sensing

Recently, continual learning has received increasing attention in the field of remote sensing. However, most existing studies have primarily focused on TIL or CIL [32,33,34,35,36], while research on DIL remains limited. To address this gap, several recent works have begun investigating DIL in remote sensing contexts. Marsocci and Scardapane [37] were the first to explore semantic segmentation of remote sensing imagery in the context of DIL, proposing a loss function that combines Elastic Weight Consolidation (EWC) [38] and Barlow Twins [39] to simultaneously address label scarcity and catastrophic forgetting. Rui et al. [29] introduced the DILRS architecture, which employs a shared encoder and domain-specific decoders to capture both domain-invariant and domain-specific features. They addressed label space shifts across domains by applying a multi-level class-specific knowledge distillation loss. Wang et al. [30] developed a self-knowledge distillation model that incorporates dynamic equilibrium constraints and learnable loss weights, effectively closing the performance gap with joint training in cross-domain fire detection. Additionally, Huang et al. [31] proposed a DIL framework leveraging frozen feature layers and a multi-feature joint loss to alleviate catastrophic forgetting. Although these studies offer valuable contributions, most rely predominantly on regularization-based or architecture-based approaches. Given the characteristics of remote sensing imagery, the use of replay-based methods, though less explored, hold significant promise, as discussed in the following section.

2.2. Replay-Based Continual Learning Algorithms

The core challenges of continual learning stem from the need to balance knowledge transfer and interference between tasks. This balance is closely related to the angles between the gradients of the loss functions for different tasks during stochastic gradient descent [40]. Specifically, learning interference occurs when the angle between these gradients exceeds 90 degrees, hindering knowledge transfer. Motivated by this insight, replay-based continual learning methods seek to reduce interference by controlling or projecting gradients to keep their angles below this threshold. For instance, Lopez-Paz and Ranzato [41] proposed Gradient Episodic Memory (GEM), which compares the current task’s gradient with those of previous tasks stored in a memory buffer and projects the current gradient to ensure the angle with previous task gradients does not exceed 90 degrees, thereby preventing interference. To improve computational efficiency, Chaudhry et al. [42] introduced Averaged GEM (A-GEM), which replaces task-specific gradient comparisons with a single comparison against the average gradient computed from the memory buffer, avoiding per-task gradient computations. Alternatively, Experience Replay (ER) [43] adopts a different approach by directly reusing stored samples instead of manipulating gradients. ER jointly samples mini-batches from the current task and the memory buffer for optimization, avoiding explicit gradient projection while still effectively mitigating catastrophic forgetting. Building on this approach, several replay-based continual learning methods have been developed in the computer vision community, including MER, DER and iCaRL [44,45,46].

Despite their success in computer vision, replay-based methods have seen limited application in remote sensing. Among the few studies, Bhat et al. [47] proposed a curriculum-based memory replay strategy to improve scene classification in remote sensing imagery. Sun et al. [48] evaluated various continual learning methods and highlighted that combining replay of representative samples with knowledge distillation enhances synthetic aperture radar (SAR)-based automatic target recognition accuracy. These early efforts demonstrate the potential of replay-based methods in remote sensing. However, they mainly focus on CIL rather than DIL, which is a common in real-world scenarios. To fill this gap, we propose an effective replay-based learning algorithm tailored for DIL.

2.3. Sample Selection Strategies for Replay

Sample selection, also called coreset selection, plays a crucial role in replay-based continual learning by efficiently managing the memory buffer, directly impacting both model performance and memory utilization. Early studies often employed simple strategies such as reservoir sampling [49] and ring buffers [41], which generally preserve data diversity but do not explicitly consider the importance of individual samples. Subsequent methods, including k-Center Greedy [50] and Mean of Features (MoF) [46] aim to select representative samples in the feature space, thereby improving representativeness. However, these methods may neglect rare or boundary samples that are crucial for robust generalization.

Recent submodular-based sample selection methods have gained attention. For instance, Yoon et al. [51] proposed a method that considers intra-batch similarity, sample diversity, and affinity to past tasks, thereby enhancing model adaptation while reducing catastrophic forgetting. Similarly, Lee et al. [52] addressed the object detection by generating representative feature vectors for multiple objects of the same class within each image and applying submodular-based greedy selection, showing better performance that random sampling. While these approaches provide a solid framework for capturing sample diversity and coverage, they do not explicitly account for the each sample’s influence on model optimization.

On the other hand, gradient-based methods emphasize the impact of individual samples on model optimization. Notably, Aljundi et al. [53] formulated sample selection as a constraint reduction problem, selecting coreset examples whose aggregated gradients closely approximate the average gradient of the full dataset. This approach helps maintain stable optimization during continual learning. However, by focusing on average gradient preservation, it may dilute the influence of critical samples near decision boundaries that are crucial for model performance. To overcome this limitation, we propose a performance-aware submodular sample selection strategy that incorporates model predictions into the selection criteria. This enables the memory buffer to retain samples that are not only representative and diverse, but also particularly informative for refining decision boundaries, thereby enhancing the stability of the continual learning process.

3. Methodology

3.1. Overview

This study is conducted under the DIL scenario, where data from multiple platforms are incrementally collected, and the model is continuously updated to accommodate newly acquired data. In this setting, learning for each domain is treated as an individual task, and the number of tasks increases as new domain data becomes available. A strict DIL setup assumes that class labels remain consistent across tasks. To align with this assumption, we adopt building segmentation as the primary downstream application. However, real-world scenarios often involve the emergence of new classes over time. To reflect such practical considerations, we additionally evaluate our approach under a more relaxed setting using LULC classification, where new classes can emerge in later tasks.

The overall learning process of ER-PASS proposed in this study is illustrated in Figure 2. For each domain, the segmentation model is trained sequentially, followed by a sample selection step that stores representative samples in the memory buffer. This buffer helps retain knowledge from previous domains and supports learning in future domains. During evaluation, the model is assessed not only on the current domain but also on all previously encountered domains. We adopt UNet [54] and DeepLabV3+ [55] as baseline models for both the building segmentation and LULC classification. Notably, ER-PASS is model-agnostic and can be readily integrated into other segmentation networks without requiring any architectural modifications.

3.2. Proposed Algorithm

ER-PASS is a replay-based continual learning algorithm tailored for DIL in remote sensing applications. It is motivated by the observation that joint learning has been shown to improve generalization across heterogeneous domains and that dataset-level memory integration—rather than batch-level—offers greater computational efficiency.

Let the continual learning process consist of a sequence of tasks $\{{\mathcal{T}}_1,{\mathcal{T}}_2,\dots ,{\mathcal{T}}_k\}$, each corresponding to a distinct remote sensing domain. For the k-th task ${\mathcal{T}}_k$, a labeled dataset ${\mathcal{D}}_k={\left\{(x_i^k,y_i^k)\right\}}_{i=1}^{N_k}$ is provided, where $x_i^k$ and $y_i^k$ denote the input image and its corresponding label, and $N_k$ is the number of training samples in task ${\mathcal{T}}_k$.

Unlike conventional experience replay methods [43,44,45], which merge current task data and memory buffer samples within each mini-batch, ER-PASS performs dataset-level integration by combining the two into a unified training set. Specifically, for task ${\mathcal{T}}_k$, the new training set ${\tilde{\mathcal{D}}}_k$ is defined as:

$${\tilde{\mathcal{D}}}_k={\mathcal{D}}_k\cup \mathcal{M}$$

(1)

where $\mathcal{M}$ denotes the memory buffer containing representative samples from previous tasks. The model parameters ${\theta }_k$ for task ${\mathcal{T}}_k$ are then optimized by minimizing the expected loss over the combined dataset ${\tilde{\mathcal{D}}}_k$:

$${\theta }_k=arg\underset{\theta }{min}{\mathbb{E}}_{(x,y)\sim {\tilde{\mathcal{D}}}_k}\left[\mathcal{L}(f_{\theta }\left(x\right),y)\right]$$

(2)

where $\mathcal{L}(·)$ denotes the downstream segmentation loss (e.g., binary cross-entropy or cross-entropy), and $f_{\theta }$ represents the segmentation model.

Through this simple yet effective approach, ER-PASS computes a single gradient per iteration without requiring gradient aggregation, while maintaining a consistent batch size. This design reduces memory overhead compared to conventional replay-based methods and shortens training time compared to joint learning by relying only on a compact memory buffer rather than the full dataset of previous tasks.

After training on task ${\mathcal{T}}_k$, a sample selection algorithm is employed to extract a representative subset of samples from ${\tilde{\mathcal{D}}}_k$ for updating the memory buffer. The selection is based on feature similarity and task-specific performance, as described in Section 3.3:

$$\mathcal{M}\leftarrow SampleSelection({\tilde{\mathcal{D}}}_k,f_{{\theta }_k})$$

(3)

The updated memory $\mathcal{M}$ is then used for training on the next task ${\mathcal{T}}_{k+1}$. The overall learning process of ER-PASS is summarized in Algorithm 1.

Algorithm 1 Learning process of ER-PASS

Input:${\left\{{\mathcal{D}}_t\right\}}_{t=1}^k$▹ Dataset corresponding to each task Require:$f_{\theta }$▹ Neural network Initialize:$\mathcal{M}\leftarrow \left\{\right\}$▹ Memory buffer 1: Define ${\tilde{\mathcal{D}}}_k$ as the training set used for task k 2: for task ${\mathcal{T}}_t={\mathcal{T}}_1,\dots ,{\mathcal{T}}_k$ do 3:     if t = 1 then 4:         ${\tilde{\mathcal{D}}}_t\leftarrow {\mathcal{D}}_1$ 5:         ${\theta }_t\leftarrow RandInit\left({\theta }_t\right)$▹ Initialize model parameters 6:     else 7:         ${\tilde{\mathcal{D}}}_t\leftarrow {\mathcal{D}}_t\cup \mathcal{M}$ 8:         ${\theta }_t\leftarrow {\theta }_{t-1}$ 9:     end if 10:     Define $N_{B,t}$ as the total number of mini-batches in ${\tilde{\mathcal{D}}}_t$ 11:     for $i=1,\dots ,N_{B,t}$ do 12:         ${\theta }_t\leftarrow {\theta }_t-\nabla f_{{\theta }_t}\left({\mathcal{B}}_{t,i}\right)$▹ Update model 13:     end for 14:     $\mathcal{M}\leftarrow SampleSelection({\tilde{\mathcal{D}}}_t,f_{{\theta }_t})$▹ Update memory buffer 15: end for

3.3. Performance-Aware Submodular Sample Selection

In ER-PASS, the samples stored in the memory buffer from previous tasks are essential to preserving knowledge. To ensure effective knowledge retention, we propose a performance-aware submodular sample selection strategy that jointly considers both feature-level diversity and task-specific prediction performance. This strategy promotes the retention of diverse, representative, and informative samples, thereby improving stability and mitigating catastrophic forgetting during learning process.

For each candidate sample $x_i\in {\tilde{\mathcal{D}}}_k$, we extract the feature representation ${\mathbf{z}}_i\in {\mathbb{R}}^d$ by applying global average pooling to the encoder output of the trained model $f_{{\theta }_k}$. Here, d denotes the dimensionality of the features space, which corresponds to 512 for UNet and 2048 for DeepLabV3+. Specifically, we use the output of the final downsampling block in the UNet encoder and the output of layer4 in the ResNet50 backbone of DeepLabV3+ as the feature map. Each feature vector is then ${\ell }_2$-normalized to ensure consistent scaling, using the following equation:

$${\tilde{\mathbf{z}}}_i={\displaystyle \frac{{\mathbf{z}}_i}{\parallel {\mathbf{z}}_i{\parallel }_2}}$$

(4)

We also compute a task-specific evaluation score $\mathcal{S}\left(x_i\right)\in [0,1]$, such as the Intersection-over-Union (IoU) or mean Intersection-over-Union (mIoU) between the model’s prediction and the ground truth. This score serves as a performance-based weight during sample selection.

Let ${\mathcal{P}}_c=\{{\tilde{\mathbf{z}}}_1,{\tilde{\mathbf{z}}}_2,\dots ,{\tilde{\mathbf{z}}}_n\}\subset {\mathbb{R}}^d$ denote the set of normalized feature vectors extracted from all candidate samples. The intra-similarity of a candidate sample $x_i$ is defined as the total cosine similarity between its feature vector and those of all other vectors in ${\mathcal{P}}_c$:

$$intra\left(x_i\right)=\sum _{p\in {\mathcal{P}}_c}{\tilde{\mathbf{z}}}_i^{\top }p$$

(5)

This term reflects the alignment of $x_i$ with the candidate sample distribution in the feature space, promoting the selection of representative and diverse samples.

Let ${\mathcal{Q}}_c\subset {\mathcal{P}}_c$ be the set of feature vectors corresponding to the samples already selected into the memory buffer. The inter-similarity of $x_i$ with respect to ${\mathcal{Q}}_c$ is then defined as:

$$inter\left(x_i\right)=\sum _{q\in {\mathcal{Q}}_c}{\tilde{\mathbf{z}}}_i^{\top }q$$

(6)

This measures the redundancy of $x_i$ relative to the selected set. A lower inter-similarity indicates that the sample introduces novel information, contributing to buffer diversity.

We define the selection scores for each candidate $x_i$ as the submodular gain—the difference between intra- and inter-similarity—weighted by task-specific evaluation score:

$$Score\left(x_i\right)=\left(intra\left(x_i\right)-inter\left(x_i\right)\right)·\mathcal{S}\left(x_i\right)$$

(7)

At each iteration, the candidate $x_{i^{★}}$ with the highest score is greedily selected and added to the memory buffer:

$$i^{★}=arg\underset{i\in {\mathcal{P}}_c\setminus {\mathcal{Q}}_c}{max}Score\left(x_i\right)$$

(8)

This process continues until a predefined budget—such as a fixed number of samples or percentage of the dataset—is reached. By jointly optimizing representativeness (via intra-similarity), redundancy reduction (via inter-similarity), and task relevance (via evaluation score), the proposed strategy constructs a memory buffer that is both submodular-optimal and performance-sensitive, effectively supporting continual learning. The detailed procedure is described in Algorithm 2.

Algorithm 2 Performance-aware submodular sample selection

Input:${\tilde{\mathcal{D}}}_k$, $f_{{\theta }_k}$▹ Dataset and trained model corresponding to ${\mathcal{T}}_k$ Require:N▹ Memory budget (number of samples to select) Output:$\mathcal{M}$▹ Updated memory buffer 1: Initialize $\mathcal{M}\leftarrow \left\{\right\}$, $\mathcal{I}\leftarrow \left\{\right\}$, ${\mathcal{Q}}_c\leftarrow \left\{\right\}$ 2: Extract features ${\mathbf{z}}_i$ from $f_{{\theta }_k}$ for each sample $x_i\in {\tilde{\mathcal{D}}}_k$ 3: Compute normalized features: ${\tilde{\mathbf{z}}}_i\leftarrow {\mathbf{z}}_i/{\parallel {\mathbf{z}}_i\parallel }_2$ 4: Compute evaluation score ${\mathcal{S}}_i$ between $f_{{\theta }_k}\left(x_i\right)$ and ground truth $y_i$ for each sample 5: Let ${\mathcal{P}}_c\leftarrow \{{\tilde{\mathbf{z}}}_1,\dots ,{\tilde{\mathbf{z}}}_n\}$, where n is the total number of samples in ${\tilde{\mathcal{D}}}_k$ 6: Compute intra-similarity: ${\mathbf{intra}}_i\leftarrow {\sum }_{p\in {\mathcal{P}}_c}{\tilde{\mathbf{z}}}_i^{\top }p$ 7: for$t=1$ to N do 8:     for $i=1$ to n do 9:         if ${\mathcal{Q}}_c=\varnothing$ then 10:            ${Score}_i\leftarrow {\mathbf{intra}}_i$▹ Only intra-similarity 11:         else 12:            Compute inter-similarity: ${\mathbf{inter}}_i\leftarrow {\sum }_{q\in {\mathcal{Q}}_c}{\tilde{\mathbf{z}}}_i^{\top }q$ 13:            ${Score}_i\leftarrow ({\mathbf{intra}}_i-{\mathbf{inter}}_i)·{\mathcal{S}}_i$ 14:         end if 15:         ${Score}_i\leftarrow -\infty$ if $i\in \mathcal{I}$▹ Exclude already selected samples 16:     end for 17:     $i^{★}\leftarrow arg{max}_i{Score}_i$ 18:     $\mathcal{M}\leftarrow \mathcal{M}\cup \left\{x_{i^{★}}\right\}$, $\mathcal{I}\leftarrow \mathcal{I}\cup \left\{i^{★}\right\}$, ${\mathcal{Q}}_c\leftarrow {\mathcal{Q}}_c\cup \left\{{\tilde{\mathbf{z}}}_{i^{★}}\right\}$ 19: end for

4. Experimental Settings

4.1. Datasets

In this study, we designed a domain sequence to reflect practical scenarios that may arise when utilizing remote sensing imagery. The construction was guided by the following three criteria: (1) each dataset should be acquired from a distinct sensing platform; (2) data within each dataset should originate from a single, consistent platform; and (3) the datasets should vary in spatial resolution. Based on these criteria, we selected four representative LULC datasets—Potsdam [56], LoveDA [57], DeepGlobe [58], and GID [59]—each representing a distinct domain (${\mathcal{D}}_1$ to ${\mathcal{D}}_4$).

Details of the constructed domain sequence are summarized in Table 1. Depending on the dataset, data were either provided as training-only or with a predefined training/test split. For training-only datasets, we divided the data into training, validation, and test sets using an 4:1:1 ratio. For datasets with a predefined split, the training portion was further divided into training and validation subsets in an 8:2 ratio, while the provided test set was retained for evaluation. To ensure a comparable number of training samples across domains, all images were divided into $512\times 512$ patches using dataset-specific stride values. Additionally, since the class definitions and label structures differ across datasets, we grouped semantically similar classes into unified categories. We first conducted binary segmentation experiments focusing exclusively on the building class, where building pixels were labeled as 1 and all other as 0. Subsequently, we extended the experiments to multi-class LULC classification using the redefined label categories for each dataset.

4.2. Implementation Details

We selected building segmentation and LULC classification as the downstream applications and conducted incremental learning experiments for each. To demonstrate the model-agnostic nature of ER-PASS, we utilized UNet [54] and DeepLabV3+ [55] as base models for both applications. As the primary focus of this study is on learning strategy rather than architectural design, no modifications were made to the base network architectures. For building segmentation, we employed binary cross-entropy loss, while multi-class cross-entropy loss, based on the redefined class labels for each dataset, was used for LULC classification.

For benchmarking, ER-PASS was compared against representative continual learning methods, including EWC [38], Learning without Forgetting (LwF) [60], and ER [43]. The EWC and LwF methods incorporated regularization terms, with $\lambda$ set to 10 and 2, respectively. For ER and the proposed ER-PASS, a memory sampling ratio of 0.5 was applied.

In the incremental learning setup, the model was randomly initialized without pretrained weights for the first task. For for subsequent tasks, the model was initialized using the weights learned from previous ones to promote knowledge transfer. Training was conducted using the NAdam optimizer with a learning rate of 0.0002 and a batch size of 8. Each experiment was run for a maximum of 100 epochs, with early stopping applied if validation performance did not improve for 10 consecutive epochs. The model achieving the best validation performance was selected for final evaluation. All experiments were implemented using PyTorch 1.12.0 and performed on a workstation equipped with 64 GB RAM and on NVIDIA RTX A6000 GPU.

4.3. Evaluation Metrics

Continual learning performance should be evaluated in terms of two key aspects: the effectiveness in learning new tasks and the capacity to mitigate forgetting of prior knowledge. To quantify these aspects, we adopt Average Incremental Accuracy (AIA) [46] and Backward Transfer (BWT) [41], defined as follows:

$${\mathrm{AIA}}_k={\displaystyle \frac{1}{k}}\sum _{i=1}^k\left({\displaystyle \frac{1}{i}}\sum _{j=1}^i{a}_{ij}\right)$$

(9)

$${\mathrm{BWT}}_k={\displaystyle \frac{1}{k}}\sum _{j=1}^{k-1}\left(a_{kj}-a_{jj}\right)$$

(10)

here, $a_{kj}$ denotes the performance on the test set of the j-th task after training incrementally up to the k-th task, where $j\le k$.

AIA measures the overall performance of a model throughout the entire learning process. By averaging performance on each previously encountered task at every step, it captures the model’s ability to learn new tasks while retaining knowledge from earlier ones. Meanwhile, BWT quantifies the degree of knowledge retention by comparing the performance on earlier tasks before and after subsequent learning, thus serving as a direct indicator of catastrophic forgetting. As continual learning aims to balance learning new tasks and retaining prior knowledge, it is essential to consider both metrics for a comprehensive evaluation.

5. Experimental Results and Discussion

5.1. Building Segmentation (Strict DIL Setting)

As previously described, our incremental learning setup employs a step-wise training procedure across four domains in the following order: Potsdam, LoveDA, DeepGlobe and GID. The model is initially trained on Potsdam in Step 1 with randomly initialized weights. In Steps 2 to 4, the model is incrementally updated by leveraging the weights learned from the previous step, thereby facilitating knowledge transfer across domains. Under this setup, we conducted downstream evaluation on building segmentation. Experiments were primarily conducted using UNet, with additional comparisons performed using DeepLabV3+.

Table 2 presents the IoU performance at each incremental step using UNet for building segmentation, along with evaluation metrics at Step 4. At each step, the reported IoU values reflect the model’s performance on all previously encountered domains, evaluated using the model trained on the current task. Models trained via single-task learning lack adaptability to unseen domains and demonstrate satisfactory performance only within the domain on which they were trained. This limitation becomes particularly evident at Step 4, where performance on other domains significantly degrades—primarily caused by substantial domain shifts. While fine-tuning facilitates some knowledge transfer, its performance gains on previous tasks over single-task learning remain marginal, suggesting the occurrence of catastrophic forgetting. Notably, even continual learning methods such as EWC and LwF fail to provide meaningful improvements, yielding results comparable to naive fine-tuning and demonstrating limited ability to mitigate forgetting. In contrast, ER demonstrates better performance in terms of BWT, suggesting a stronger capability to retain prior knowledge. It also achieves higher AIA values compared to other baseline methods. However, comparison with single-task learning reveals that ER exhibits lower performance on the current task at each step, indicating that the high AIA is primarily due to the preservation of previous knowledge rather than effective learning of new tasks.

In theory, joint learning serves as the performance upper bound for incremental learning approaches. Our experimental results demonstrate that the proposed method achieves performance comparable to joint learning, and in some cases, even surpasses it. Furthermore, our method consistently outperforms baseline methods. These results indicate that our method not only mitigates catastrophic forgetting effectively but also maintains reasonable adaptability to new tasks. A similar trend is observed in Table 3, where DeepLabV3+ is used as the segmentation model instead of UNet. This consistent improvement across different architectures highlights that the effectiveness of the proposed algorithm is not limited to a specific model, demonstrating its model-agnostic nature.

To complement the quantitative results, Figure 3 and Figure 4 provide qualitative visualizations that further demonstrate the effectiveness of the proposed method. Figure 3 shows segmentation results from each domain using the UNet model trained up to Step 4, illustrating representative samples across domains. As expected, the segmentation performance on GID is relatively accurate since the model was directly trained on that domain. However, all benchmark methods exhibit noticeable degradation on previously learned domains. In particular, EWC and LwF fail to produce meaningful predictions, indicating that severe catastrophic forgetting has occurred. In contrast, ER performs better than these methods and partially mitigates forgetting. The proposed method maintains segmentation performance on previous domains at a level comparable to joint learning, clearly demonstrating its effectiveness in mitigating catastrophic forgetting.

Beyond the cross-domain comparison at Step 4, Figure 4 presents the step-by-step segmentation results for the first domain, analyzing how each method retains prior knowledge throughout the incremental learning process. In the case of EWC and LwF, prior knowledge is relatively well preserved up to Step 2. However, noticeable forgetting begins at Step 3, and severe catastrophic forgetting becomes evident by Step 4. ER maintains acceptable performance through Step 3, with slight degradation observed at Step 4. Consistent with earlier findings, the proposed method maintains stable performance across all steps, further demonstrating its robustness against catastrophic forgetting.

5.2. LULC Classification (Relaxed DIL Setting)

To further evaluate the generalizability of the proposed method in practical applications, we conduct experiments on LULC classification. Unlike building segmentation, which follows a strict DIL setting with consistent class labels, LULC classification represents a more relaxed and realistic scenario where new classes may emerge over time.

Table 4 and Table 5 present LULC classification results for UNet and DeepLabV3+, respectively. Similar to the building segmentation experiments, these tables provide step-wise mIoU scores and final-step metrics, with overall trends following a comparable pattern. Nonetheless, several noteworthy differences can be observed. In earlier experiments, EWC and LwF failed to retain prior knowledge, yielding near-zero scores at Step 4. In contrast, these methods exhibit relatively better performance in the LULC classification. This may be attributed to greater semantic consistency or overlap among LULC classes across domains, potentially facilitating improved knowledge retention.

Another interesting observation concerns the behavior of ER. While ER exhibited substantial forgetting at Step 4 in building segmentation, its performance degradation in LULC occurs earlier, at Step 2, and remains relatively stable in subsequent steps. This indicates that the dynamics of forgetting and retention can vary depending on the complexity and structure of the target application. Meanwhile, the proposed method consistently mitigates forgetting, as also observed in the building segmentation experiments.

Figure 5 illustrates the LULC classification results obtained with the UNet model trained up to Step 4. The results for EWC and LwF on previous domains show that predictions are primarily focused on classes such as forest, grassland, water, and agriculture, which correspond to the classes present in the GID. However, both methods largely fail to predict classes absent from GID, such as cars, roads, and barren areas, suggesting substantial forgetting of these classes. Notably, despite the building class being commonly present across all domains, both EWC and LwF still perform poorly on this class. This is likely due to substantial domain discrepancies, which hinder consistent recognition. In contrast, ER seems to retain partial retention of classes absent from GID, demonstrating better knowledge preservation across diverse classes and domains.

Figure 6 provides a detailed visualization of the forgetting dynamics across incremental steps. As confirmed by the quantitative results, substantial forgetting occurs at Step 4 in the building segmentation, whereas in the LULC classification, it emerges earlier, starting from Step 2. Specifically, in the case of ER, prediction performance on the grassland class noticeably degrades at Step 2, which corresponds to the second domain, LoveDA, where this class is absent. This implies that classes not present in the current task are more susceptible to forgetting. Similarly, at Step 3, performance on the road class deteriorates, consistent with its absence in the DeepGlobe, and this trend continues through Step 4. Notably, at Step 4, a performance degradation is also observed for the building class, which is present in the GID, aligning with earlier observations in the building segmentation experiments. In contrast, the proposed method effectively preserves a broader range of class representations across incremental steps, mitigating forgetting not only for classes currently being learned but also for those learned in previous steps. These findings suggest that the proposed method generalizes well beyond binary segmentation to more general semantic segmentation problems such as LULC classification, underscoring its robustness and scalability under both strict and relaxed DIL settings.

5.3. Ablation and Efficiency Analysis

5.3.1. Ablation Study on Proposed Sample Selection Strategy

As described earlier, the proposed sample selection strategy prioritizes samples based on a score function. This score is defined as the product of two components: the submodular gain, which promotes sample diversity and representativeness, and a task-specific evaluation score, which reflects the importance of each sample for the current task. To assess the individual contribution of each component, we conduct an ablation study using our primary experimental setup of incremental building segmentation with UNet.

Table 6 presents the results of the ablation study. Using only the submodular gain results in performance lower than the random baseline, suggesting that diversity alone may not be sufficient for effective sample selection. In contrast, using only the evaluation score yields improved performance over random selection, confirming the usefulness of task-specific information. Nevertheless, the improvement remains limited. The best performance is achieved when both components are combined, indicating that they complement each other. This demonstrates that by jointly considering both sample diversity and informativeness, the proposed sample selection method enhances robustness and overall performance throughout the learning process.

5.3.2. Effect of Sampling Ratio

In replay-based continual learning methods, the size of memory buffer—i.e., the number of stored samples—is a key factor in balancing performance and resource efficiency. Too few samples can lead to severe forgetting, while too many increase resource consumption and reduce practicality. Therefore, selecting an appropriate sampling ratio is essential for scalability in real-world applications. To better understand the impact of the sampling ratio on knowledge retention, we conducted experiments with varying ratios (0.1 to 0.9) using UNet for incremental building segmentation. For each sampling ratio, the BWT values were measured across three incremental steps (BWT₂, BWT₃, and BWT₄), and the results are visualized in Figure 7.

As can be seen in Figure 7, increasing the sampling ratio generally led to BWT values approaching zero across all steps. In the low sampling range (0.1 to 0.4), BWT₂ maintained relatively higher values, suggesting better knowledge retention in earlier stages. In contrast, BWT₃ and BWT₄ exhibited significantly lower values in the same range, reflecting more severe forgetting as the learning progressed. Interestingly, both BWT₃ and BWT₄ showed a sharp recovery at a sampling ratio of 0.5, with BWT values rapidly approached zero. Beyond this point, the BWT values remained relatively stable, with only marginal improvements, suggesting that the benefit of additional sample storage diminishes after a certain threshold. These findings indicate that, while increasing the sampling ratio is effective for reducing forgetting, storing excessive samples may not yield further performance gains and could instead introduce inefficiencies in resource usage. Therefore, we recommend using at least a 0.5 sampling ratio to ensure a desirable trade-off between performance and computational efficiency.

5.3.3. Computational Efficiency Analysis

In practical scenarios, training time and memory usage are also critical considerations for the deployment of continual learning systems. To assess these aspects, training time and memory consumption were monitored throughout our primary experimental setting. Training time was measured as the average duration per epoch during Step 4, and memory usage was recorded as the peak memory consumption observed throughout the training up to Step 4.

As shown in Table 7, joint learning took 902.55 seconds per epoch, while the proposed method only took 441.28 seconds, yielding a total reduction of about 13 hours over 100 epochs. As the number of tasks increases in future scenarios, this time gap is expected to grow even larger. This demonstrates that the proposed method offers a time-efficient solution for DIL, which is particularly beneficial in real-world applications with limited computational resources. Furthermore, the proposed algorithm demonstrated the lowest memory consumption among the continual learning benchmarks, reducing memory usage by almost half compared to ER. Although its training time is slightly longer than that of other continual learning methods, it provides a well-balanced trade-off between performance, training time, and memory efficiency. Therefore, the proposed method can be regarded as a practical and scalable solution for deploying deep learning models in remote sensing applications within DIL scenarios.

6. Conclusions

In this paper, we propose ER-PASS, an experience replay-based continual learning algorithm that leverages the strengths of joint learning and experience replay for DIL in remote sensing. ER-PASS integrates a performance-aware submodular sample selection strategy to enhance the stability of the learning process across evolving domains. We conducted experiments on two distinct remote sensing applications: building segmentation and LULC classification. The results demonstrate that ER-PASS effectively mitigates catastrophic forgetting in both cases, confirming its generalizability across diverse applications. Furthermore, experiments with multiple model architectures, including UNet and DeepLabV3+, show that ER-PASS is model-agnostic and can be flexibly integrated into various network structures. Additional analyses confirm that the proposed sample selection strategy positively impacts adaptability and forgetting mitigation, with stable performance maintained when the sampling ratio is 0.5 or higher. Finally, ER-PASS exhibits reasonable efficiency in terms of training time and memory consumption. These results suggest that ER-PASS can be practically applied in real-world remote sensing scenarios.

Nevertheless, ER-PASS has several limitations. First, this study focuses solely on optical imagery, and its generalizability to more heterogeneous remote sensing domains such as hyperspectral imagery or SAR remains to be validated in future work. Second, ER-PASS was evaluated on only two representative applications, and its applicability to a wider range of remote sensing applications, such as change detection or super-resolution, warrants further exploration. Lastly, ER-PASS, as a learning algorithm that is not dependent on specific model architectures, was validated using commonly adopted models such as UNet and DeepLabV3+. Given that most recent studies have proposed architecture-based methods, it is believed that further research is needed to compare and potentially integrate ER-PASS with these approaches.