Multi-Weather DomainShifter: A Comprehensive Multi-Weather Transfer LLM Agent for Handling Domain Shift in Aerial Image Processing

1. Introduction

Aerial Image Segmentation (AIS) is an essential task for various city monitoring purposes, such as environmental surveillance, target localization, and disaster response [1,2,3]. With semantic segmentation models trained on large-scale annotated data, humans can easily extract abundant geo-spatial information from aerial images captured by drones or satellites [4,5,6].

1.1. The Challenge of Weather Change caused Domain Shift in Aerial Imagery

While the performance of semantic segmentation algorithms has surged on common benchmarks, progress in handling the domain shift of unseen environmental conditions is still stagnant [7,8,9]. We demonstrate that the aerial segmentation performance of algorithms is prone to significant degradation due to Domain Shift, i.e., the transfer from one domain to another. In Figure 1, we illustrate this phenomenon by comparing the original data in the ISPRS datasets [10,11] with our generated domain-shifted versions, including illumination variations (clear to overcast), changing atmospheric conditions (clear to foggy, dusty), and physical scene changes (clear to snowy). Notably, the scene content and target information remain consistent across all weather variations, while atmospheric conditions and illumination characteristics change significantly in the first three weather domain. The last snowy condition presents additional challenges with physical scene changes including leaf drop, snow-covered roofs and ground, while preserving the information of target of interests. This figure demonstrates the challenge of cross-domain generalization in aerial image analysis.

The radar chart in Figure 2 , in which our evaluations of nine state-of- the-art segmentation models on the ISPRS dataset [10,11] and its domain-shifted version are shown, further illustrates that even within the same scene and objects, slightly altering the atmospheric conditions and varying lighting levels pose challenges for aerial image segmentation algorithms. As detailed in the caption of Fig. 2, the results show that after transferring the data from its original, intact domain to shifted domains, there is an average mIoU deterioration of {-3.35%, -3.92%, -10.55%, -28.59%} and mFscore deterioration of {-2.61%, -3.24%, -8.66%, -25.76%} under overcast, foggy, dusty, and snowy conditions on the Vaihingen dataset ($512\times 512$ resolution). Notably, we generated snowy image sets with five different random seeds, and the numeric results represent the average across these five sets. Compared to the original intact data, the illumination in the shifted overcast images is reduced, foggy and dusty weather additionally changes the atmospheric information, and the snowy scene add physical changes on the target of interest in the original scene, representing typical domain shift. However, the image content, layout, and geo-spatial information between the original and shifted data remain unchanged.

Closing the gap between model performance in the original domain and the shifted domain is a valuable problem to address. An intuitive solution is to incorporate multi-domain data into the model training process. The performance of aerial image segmentation models significantly relies on the availability of training data. Although data from adverse domains is essential to improve the robustness of aerial image segmentation models, such data—including aerial images captured under low illumination and harsh weather conditions is lacking in the current aerial image benchmarks [10,11,12]. On the other hand, even if sufficient data is obtained, annotation remains a time-consuming and labor-intensive task. This paper breaks the limitation of low-domain diversity while eliminating the need for additional annotations on shifted domain data.

1.2. Recent Developments in Generative Model and Image Synthesis

Recently, significant success has been achieved by generative models, which aim to mimic humanity’s ability to yield various modalities. The capacity of GPT-series [13,14], Llama series [15,16], Qwen series [17,18] and DeepSeek series [19,20] in Natural Language Processing field has significantly impacted human’s daily life. In the meantime, stable-diffusion [21,22], DALL-E [23,24] in Computer Vision, have been proposed for generating high-quality realistic images.

While earlier methodologies like Generative Adversarial Network (GAN)-based methods [25,26,27,28,29,30] and Variational Autoencoder (VAE)-based methods [31,32] demonstrate remarkable performance in yielding realistic samples, training instability remains a well-known issue. For instance, GANs require a delicate balance between the generator and discriminator, which can lead to problems like mode collapse—where the generator produces limited diversity in outputs. This motivated the development of more stable approaches like diffusion models.

Instead of traditional diffusion models (DMs) that denoise the input x in the image-scale (pixel level) [33,34], current general text-to-image (T2I) Latent Diffusion Models (LDMs) [21,22,23,35,36] adopt a VAE-like Encoder $\mathcal{E}$ and Decoder $\mathcal{D}$ structure. LDMs first compress the input into a latent representation $z=\mathcal{E}\left(x\right)$, and afterwards deploy the diffusion process within the latent space, such that the decoder output $\tilde{x}$ is the reconstructed input x. With the hallmark of achieving a favorable trade-off between reducing computational and memory costs while maintaining high-resolution and quality synthesis, operating on smaller spatial latent representations of the input has become a popular framework for recent generative models [37,38,39]. Based on the development of DM based image generation, some studies also focus on aerial image synthesis [6,38,40,41] but they all concentrate on the common weather rather than multiple domain data.

Beyond T2I image generation, Image-to-Image (I2I) Style Transfer [26,42,43,44,45,46,47,48,49] is a practical generative task that aims to extract style and texture information from one reference image and transfer it to another image while preserving semantic content. Prior methods can synthesize vivid and diverse results, such as converting a landscape photo into a painterly oil artwork. However, for de facto domain shifts in aerial imagery, the performance of these methods is limited for the following reasons: (1) Lack of diverse style references: These methods lack style reference imagery for various domains and a unified environment that provides diverse weather and illumination conditions; (2) Inadequate handling of complex domain shifts: While traditional neural network-based style transfer models [26,42,43,44] can handle atmospheric and illumination changes, they fail to tackle complex domain transfers such as snowy conditions, where physical snow/winter-related elements should be added to the scene, e.g., snow accumulation on rooftops and leafless trees; (3) Content alteration issues: Diffusion model (DM)-based methods [45,46,47,48,49] are prone to altering the original semantic content of images, such as shifts in object positions or deformations of large structures. While such alterations are acceptable in human face style transfer or art editing, preserving geo-spatial information is vital for remote sensing analysis. Moreover, such content alterations render the original semantic segmentation masks unusable, resulting in an additional expensive annotation burden.

ControlNet [35] has recently become a promising approach with the capability to control stable diffusion through various conditioning inputs such as Canny edges, skeletons, and segmentation masks. However, it requires detailed text prompts to achieve consistent target generation in the remote sensing domain. Therefore, in this work, in addition to leveraging segmentation maps as layout conditions, we also utilize them as input for LLM-assisted text descriptor generation. Specifically, for each aerial image’s corresponding segmentation mask, we calculate the pixel ratio for each class and assign each class to one of three levels: high, medium, or low, then construct a scene elements array as input to the LLM. With this approach, detailed and closely scene-corresponding text prompts are generated.

Though a variety of image generation or style transfer models are developed recently, they are still inadequate and encounter some specific problems on the way to handling domain shift in aerial image processing. However, generative models tend to specialize in a particular task and is equipped with its own merits, e.g., style transfer models can easily change the image scene illumination and atmospheric conditions with a fast inference speed; DM-based methods can greatly edit the images’ content while costing a sequence of sampling steps. Therefore, there is a critical need to adopt multiple models and leverage their advantages to generate diverse weather conditions for aerial imagery while preserving semantic content and geo-spatial information. Recently, Large Language Models (LLMs) [13,14,15,16,17,18,19,20] have emerged as powerful agents capable of orchestrating complex and multi-step tasks. Several pioneering works [50,51,52,53,54] have demonstrated that LLMs can effectively learn to coordinate and utilize diverse tools across multiple modalities and domains, achieving remarkable performance in language processing, computer vision, and other challenging applications. Leveraging LLMs as intelligent agents to automatically select and coordinate appropriate models for addressing diverse domain shift scenarios represents a promising and scalable solution.

1.3. Essence and Contributions of this paper

To address the underestimated domain shift challenge in current remote sensing analysis, particularly in aerial image segmentation, we propose DomainShifter (illustrated in Figure 3) to overcome the limitation of limited domain variety while eliminating the need for additional annotations on domain-shifted data. Specifically, for multi-weather scene transfer in aerial imagery, given a user’s text input, a LLM agent (e.g., Claude 3.7 Sonnet, GPT-4, DeepSeek R-1, etc.) decomposes the task into simpler steps and systematically plans the procedure for resource identification, appropriate generative tool selection, self-correction, and verification. This oaper comprises the following key components:

Aerial Weather Synthetic Dataset (AWSD): To complement existing datasets and address their limitations, we developed Aerial Weather Synthetic Dataset (AWSD), which introduces controlled variations in weather and lighting based on Unreal Engine [55]. This dataset provides an ideal benchmark for evaluating the robustness of segmentation models in diverse environmental conditions. Leveraging this dataset, we generated realistic domain-shifted data, which supplements existing aerial image segmentation datasets like ISPRS datasets [10,11]. We specifically focused on overcast, foggy, and dusty weather conditions, which are typical domain shift scenarios that change illumination and introduce atmospheric obscuration elements. This allowed us to demonstrate the effects of domain shift and present domain adaptation results.

Latent Aerial Style Transfer model (LAST): Based on the AWSD, we present a latent style transfer model for aerial images. This model transfers domain information from synthetic data. In particular, we first utilize a VAE encoder to simultaneously compress both the style reference image and the semantic content image into latent space. The interaction between the style and content is then processed through transformer blocks in this latent space. Finally, the transformed output is decoded back into the image space using the VAE decoder. We transfer clear weather aerial images from the original ISPRS dataset into overcast, foggy, and dusty weather conditions.

Multi-Modal Snowy Scene Diffusion Model (MSDM): In addition to changing illumination and atmospheric information, diffusion models are more appropriate for generating physical element (object) based scenes such as snowy scenes, e.g., snow-covered roofs and ground. To achieve consistency in image content (including targets of interest and layout), we propose a Multi-Modal Snowy Scene Diffusion Model by leveraging both image conditions and text conditions. Specifically, real aerial images’ segmentation masks are simultaneously served as image conditions controlled by ControlNet [35] and as initial scene descriptions that provide object information in the images. Then the object information is extended into detailed text prompts by a local-implemented Qwen3-14B [18] model.

Based on the above three contributions, we handle the scarcity of domain-specific data in aerial image segmentation. Moreover, we benchmark nine different state-of-the-art segmentation models on multi-domain datasets generated by Multi-Weather DomainShifter. Extensive experiments reveal the performance degradation caused by domain shifts, and we successfully adapted model performance in the shifted domain while maintaining its effectiveness in the source domain.

2. Related Work

2.1. Semantic Segmentation

Following the pioneer approach, i.e., Fully Convolutional Network (FCN) [56], encoder-decoder structure has been a prevalent paradigm for semantic segmentation task. In the early stage, these methods [57,58,59,60] combined the low level feature and its up-sampling high level to obtain the precise objects boundaries meanwhile capture the global information. Consequently, deeplab-series methods [61,62,63] developed the dilated convolutions to enlarge the receptive field of convolutional layers and further employed spatial pyramid pooling modules to obtain multi-level aggregated feature.

In addition to CNN-based semantic segmentation methods, vision transformer-based approaches [64,65,66,67] have also become popular due to their exceptional ability to capture long-range contextual information among tokens or embeddings. SETR [68] employs ViT as its backbone and utilizes a CNN decoder to frame semantic segmentation as a sequence-to-sequence task. Moreover, Segmentor [69] introduces a point-wise linear layer following the ViT backbone to generate patch-level class logits. Additionally, SegFormer [70] proposed a novel hierarchically structured Transformer encoder which outputs multiscale features and a MLP decoder to combine both local and global information. Notably, many recent Feature Pyramid Network (FPN) [71]-based affinity learning methods [4,5,72,73] are proposed to achieve better feature representation and successfully handle the scale-variation problem [12,74] in aerial image segmentation.

2.2. Image Style Transfer

Image style transfer [26,42,43,75] is a practical research field that applies the style of one reference image to the content of another. Image style transfer aims to generate a transferred image that contains the content, such as shapes, structures, and objects, of the original content image but adopts the style, such as colors, textures, and patterns, of the reference style image. The pioneer method [42] demonstrates that the hierarchical layers of CNNs can extract content and style information, proposing an optimization-based approach for iterative stylization. However, such optimization-based networks are often limited to a fixed set of styles and cannot adapt to arbitrary new ones. To address this limitation, AdaIN [76] presents a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Later work by Chen et al. [77] employs an internal-external learning scheme with two types of contrastive loss, enabling the generated image to be more visually plausible and harmonious. StyTr² [44] aims to keep the content consistency on art style transfer with a content-aware positional encoding (CAPE) transformer, which increases the computation cost and reduces the inference speed, making it less suitable for high-resolution remote-sensing applications.

Recently, with the great generative capability of latent diffusion models (LDM) [21,22,23,35,36], style transfer methods based on LDM have achieved tremendous progress [45,46,47,48,49]. However, except for DM’s inherent deficiency, i.e., low generation efficiency caused by the multi-step diffusion process, these methods cannot keep the precise layout of the original content image. Recently, LoRA-based [78] techniques [79,80,81,82,83] have shown remarkable efficacy in capturing style from a single image. In particular, B-LoRA [82] and ConsistLoRA [83] fine-tune two attention layers of up-sampling blocks in SDXL [22] to separately control content and style. However, for each reference image and content image pair, they [82,83] need extra LoRA training, which is inefficient for large-scale aerial image processing.

2.3. Domain Shift

Domain shift [84] is a well-known challenge that results in unforeseen performance degradation when a model encounters conditions different from those in its training phase. To address this issue, domain generalization (DG) algorithms [85,86,87,88] have been developed to generalize a model across weather conditions and environments unseen during training, where target domain data is unavailable. Additionally, as a sub-field of transfer learning, domain adaptation (DA) methods [89,90,91] have been proposed to adapt a model trained on a source domain to perform effectively on a target domain. Generally, DA algorithms aim to learn a model from labeled source data that generalizes to a target domain by minimizing the distribution gap between the two domains.

The practical application of domain shift solutions is often hampered by the availability of target domain data, which can be rare and difficult to acquire, especially for diverse weather conditions. Moreover, annotating data for new domains is a laborious and time-consuming task. Therefore, unlike methods that rely on real-world target data, our approach utilizes Unreal Engine [55] to build a synthetic dataset encompassing a wide variety of weather conditions (details in Section 4.3). Furthermore, we apply style transfer to augment the existing, finely-annotated ISPRS [10,11]. As a result, by performing joint training on both the source and the synthetically shifted domains, our method can effectively mitigate the domain shift problem and its associated performance degradation.

3. Methodology

3.1. Multi-weather DomainShifter

Multi-weather DomainShifter is our proposed comprehensive multi-weather domain transfer system that orchestrates multiple generative models to handle diverse weather change caused domain shift scenarios in aerial imagery. As shown in Figure 3, the system integrates both data resources and specialized tools, coordinated by a Reasoning and Acting (ReAct) framework [92] based LLM agent [13,14,15,16,17,18,19,20] that can interpret and execute complex, multi-step user commands delivered in natural language.

3.1.1. System Architecture

The architecture of Multi-Weather DomainShifter consists of the following three core components:

• Image Resources: This component serves as the data foundation for all operations. It is subdivided into three libraries: (1) a Style Image Library containing the target domain style references from our synthetic AWSD dataset (e.g., overcast, foggy, dusty), detailed in Section 4.3; (2) a Content Image Library storing the source domain images from real-world datasets like ISPRS [10,11]; and (3) a Content Mask Library with the corresponding semantic segmentation masks for the content images. The samples of style references, original content images and corresponding segmentation masks are demonstrated in the top part of Figure 3.
• Tool Resources: As shown in the bottom part of Figure 3. This is a curated library of specialized generative models and general-purpose utilities. All functions in this tool resources are abstracted as tools with descriptions, enabling the LLM agent to understand how they should be utilized. The primary generative tools are our proposed (1) LAST model, designed for efficient style transfer of illumination and atmospheric changes (overcast, foggy, dusty), details in Section 3.2; and (2) the MSDM, a multi-modal diffusion model for handling complex physical scene alterations like snowy conditions, details in Section 3.3. The library is augmented with general tools for tasks such as resource listing and data transferring.
• LLM Agent (ReAct Framework): The system’s intelligence is orchestrated by an LLM agent operating on the ReAct paradigm [92]. This agent synergistically combines reasoning and acting to process user needs, which is illustrated in Figure 3. For each step, it generates a thought process (reasoning), devises an action to execute, and then observes the outcome of that action. This iterative cycle of Thought → Action → Observation allows the agent to dynamically plan, execute, and self-correct until the user’s goal is fully accomplished.

3.1.2. Agent Workflow

The ReAct-based LLM agent follows a conceptual framework that enables autonomous task decomposition and execution, as illustrated in Figure 3. The agent’s workflow operates through several key phases: Task Understanding, where natural language instructions are parsed and resource requirements are identified; Strategic Planning, where the agent devises optimal execution strategies considering computational efficiency and resource availability; Tool Selection and Execution, where appropriate generative models are selected and invoked based on the specific domain transfer requirements; and Quality Assurance, where the agent validates outputs and ensures task completion.

This iterative reasoning-acting cycle enables the agent to handle complex, multi-modal domain transfer scenarios that would traditionally require manual intervention. The agent’s ability to dynamically select between LAST for atmospheric changes and MSDM for physical alterations, while coordinating batch processing and resource management, demonstrates the system’s capacity for intelligent orchestration of heterogeneous generative models. This architecture ensures that Multi-Weather DomainShifter can adapt to diverse user requirements and scale efficiently across different domain shift scenarios.

3.2. LAST

To achieve style transfer for aerial images, accounting for variations in weather conditions and illumination while reducing the computational cost of processing, we propose the Latent-space Aerial Style Transfer (LAST) model. This model operates in two spaces: image space and latent space, as depicted in Figure 4. Specifically, inspired by the Latent Diffusion Models (LDMs) [21], we first compress the input aerial images into the latent space using a pre-trained VAE. The style transformation is then performed in this latent space.

Overall, this model consists of the following parts: (1) A VAE encoder first compresses both images into a latent space. (2) The resulting latent representations are then flattened into sequences. (3) The core style transfer operation is performed by a latent style transformer that processes these sequences. (4) Finally, a VAE decoder reconstructs the modified latent representation back into the image space, producing the final stylized image. Additionally, the perceptual loss [75], computed via a pre-trained VGG-19 [93], is applied to optimize the model.

3.2.1. VAE for Image Compression

We first deploy the same setup as Latent Diffusion Models (LDMs) [21] to compress images into the latent space via a variational autoencoder (VAE [31,32]) pre-trained with a Kullback-Leibler (KL) divergence penalty.

Given an image $x\in {\mathbb{R}}^{H\times W\times 3}$ in the image space, the encoder $\mathcal{E}$ maps x to a latent representation $z\in {\mathbb{R}}^{h\times w\times C}$, where $h=H/f$, $w=W/f$ with a down-sampling factor $f=4$. Subsequently, the decoder $\mathcal{D}$ reconstructs the image $\tilde{x}=\mathcal{D}\left(z\right)$ from the latent vector z. Specifically, the process within LAST involves three steps:

1) As illustrated in the top-left part of Figure 3, an encoder $\mathcal{E}$ maps the input content image $x_c$ and style image $x_s$ to two separate Gaussian distributions:

$$\begin{array}{cc}\hfill \mathcal{N}({\mu }_c,{\sigma }_c^2)& =\mathcal{E}\left(x_c\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathcal{N}({\mu }_s,{\sigma }_s^2)& =\mathcal{E}\left(x_s\right).\hfill \end{array}$$

(2)

The reparameterization trick [31,94] is applied to sample the latent vectors $z_c$ and $z_s$ from their respective distributions:

$$\begin{array}{cc}\hfill z_c& ={\mu }_c+{\sigma }_c\odot ϵ,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill z_s& ={\mu }_s+{\sigma }_s\odot ϵ,\hfill \end{array}$$

(4)

where ⊙ denotes element-wise multiplication, $ϵ\sim \mathcal{N}(0,1)$ is a noise tensor, and both $z_c,z_s\in {\mathbb{R}}^{h\times w\times C}$.

2) Within the latent space, the vectors $z_c$ and $z_s$ are processed by the Latent Style Transformer (LSTrans), as shown in the bottom part of Figure 3, which outputs a new latent vector $z_t$:

$$z_t=\mathit{LSTrans}(z_c,z_s),$$

(5)

where $z_t\in {\mathbb{R}}^{h\times w\times C}$.

3) Finally, the VAE decoder $\mathcal{D}$ reconstructs the stylized image $x_t=\mathcal{D}\left(z_t\right)$, where $x_t\in {\mathbb{R}}^{H\times W\times 3}$, as indicated in top-right part of Figure 3.

3.2.2. Latent Style Transformer

The latent representations, denoted as $z\in {\mathbb{R}}^{h\times w\times C}$, are first flattened and embedded into latent sequences, represented as $s\in {\mathbb{R}}^{hw\times C}$. To transfer domain-specific information from the input style image to the content image while preserving original semantic details—such as objects, boundaries, and spatial relationships—we stack three sequential transformer blocks in the latent space to process the compressed latent representations. Each block consists of the following components: A Multi-head Self-Attention (MSA) module to grasp contextual information within the content features; A Multi-head Cross-Attention (MCA) module to facilitate interaction between the content and style sequences; A Feed-Forward Network (FFN) to enhance the model’s capacity for non-linear transformation and feature combination.

As a result, LSTrans outputs the transferred latent sequence. After being rearranged and projected back to the spatial domain, we obtain the transferred latent representation $z_t\in {\mathbb{R}}^{h\times w\times C}$, which is then decoded into the final image $x_t\in {\mathbb{R}}^{H\times W\times 3}$ in the image space.

3.2.3. Perceptual Loss for Model Optimization

To guide the model to generate a transferred image $x_t$ that preserves the content of $x_c$ while incorporating the style of $x_s$, we follow established style transfer approaches [44,75,76,77] and employ a perceptual loss (also known as VGG loss). The total loss ${\mathcal{L}}_{\mathrm{total}}$ is a weighted sum of a content loss and a style loss:

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_c{\mathcal{L}}_c+{\lambda }_s{\mathcal{L}}_s$$

(6)

where ${\mathcal{L}}_c$ computes the content discrepancy between $x_t$ and $x_c$, and ${\mathcal{L}}_s$ computes the style discrepancy between $x_t$ and $x_s$. The weights ${\lambda }_c$ and ${\lambda }_s$ balance these two components.

Given the pre-trained VGG-19 network and input image $x\in {\mathbb{R}}^{H\times W\times 3}$, we extract features at different depths to capture distinct visual characteristics. The first four convolutional layers output low-level features $f_l\left(x\right)$ that encode style and domain information, while the last two convolutional layers output high-level features $f_h\left(x\right)$ that encode semantic content. The content loss and style loss are computed as follows:

$${\mathcal{L}}_c={\parallel f_h\left(x_t\right)-f_h\left(x_c\right)\parallel }_2^2$$

(7)

$${\mathcal{L}}_s={\parallel f_l\left(x_t\right)-f_l\left(x_s\right)\parallel }_2^2$$

(8)

where $f_h\left(x_t\right)$ and $f_h\left(x_c\right)$ represent the high-level features of the transferred image $x_t$ and content image $x_c$ respectively, while $f_l\left(x_t\right)$ and $f_l\left(x_s\right)$ represent the low-level features of the transferred image $x_t$ and style image $x_s$ respectively.

3.3. MSDM

To address the challenge of generating realistic snowy aerial scenes while maintaining semantic consistency, we propose the Multi-modal Snowy Scene Diffusion Model (MSDM). This model integrates ControlNet for structural conditioning with an LLM-assisted scene descriptor to generate contextually rich textual prompts. As illustrated in Figure 5, MSDM ensures that generated snowy scenes preserve the spatial layout and semantic content of the original imagery while incorporating realistic weather-specific visual effects.

3.3.1. ControlNet for Segmentation Mask Conditioning Diffusion Model

As shown in the Image Prompt of Figure 5, we employ ControlNet [35] for snowy scene generation to maintain structural consistency between original domain and generated snowy images. ControlNet extends pre-trained diffusion models by introducing additional conditional inputs without requiring complete retraining of the base model [21]. We create a trainable duplicate of the U-Net [57] encoder blocks that processes spatial conditioning information while keeping the original model parameters frozen. Given a segmentation mask $\mathbf{c}\in {\mathbb{R}}^{H\times W\times 3}$ and noisy latent ${\mathbf{z}}_t$ at timestep t obtained by applying the forward diffusion process [21,34,95]on the clean latent representation ${\mathbf{z}}_0$, the ControlNet generates additional spatial features:

$${\mathbf{z}}_t=\sqrt{{\overline{\alpha }}_t}{\mathbf{z}}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ$$

(9)

$${\mathbf{F}}_{down},{\mathbf{F}}_{mid}=\mathrm{ControlNet}({\mathbf{z}}_t,t,{\mathbf{e}}_{text},\mathbf{c})$$

(10)

where $ϵ\sim \mathcal{N}(0,\mathbf{I})$ is the ground truth noise and ${\overline{\alpha }}_t$ the cumulative noise schedule parameter. ${\mathbf{e}}_{text}$ represents the CLIP [96] text embedding of the input prompt (shared with the original U-Net), ${\mathbf{F}}_{down}$ are the down-sampling block residuals, and ${\mathbf{F}}_{mid}$ is the middle block residual feature. The ControlNet features are integrated into the U-Net prediction through additive residual connections via zero-initialized convolution layers:

$${ϵ}_{\theta }=\mathrm{UNet}({\mathbf{z}}_t,t,{\mathbf{e}}_{text},{\mathbf{F}}_{down},{\mathbf{F}}_{mid})$$

(11)

Our training objective follows the standard diffusion loss with ControlNet conditioning:

$$\mathcal{L}={\mathbb{E}}_{{\mathbf{z}}_0,\mathbf{c},ϵ,t}\left[\parallel ϵ-{ϵ}_{\theta }({\mathbf{z}}_t,t,{\mathbf{e}}_{text},\mathbf{c}){\parallel }_2^2\right]$$

(12)

For our application, we utilize segmentation masks from the merged ISPRS Vaihingen dataset as control signals. Each mask contains semantic classes including buildings, roads, trees, low vegetation, vehicles, and clutters, converted to RGB format using pre-defined color mapping. Please refer to original paper of DDPM[34], DDIM [95], StableDiffusion[21] and ControlNet [35] for detailed architecture and mechanisms.

3.3.2. LLM-assisted Scene Descriptor

To enhance the quality and realism of generated snowy scenes, we incorporate textual descriptions generated by Qwen3-14B [18]. Unlike fixed templates or manual annotations, our method performs intelligent scene analysis to generate contextually rich and semantically accurate prompts. As indicated in the bottom part of Figure 5, the LLM analyzes the segmentation masks and generates detailed scene descriptions that capture the semantic content, which are then used as additional conditioning information for the diffusion model.

Segmentation Mask Analysis The LLM-assisted descriptor begins with quantitative analysis of the segmentation mask. For each semantic class k, we compute the pixel ratio:

$$r_k={\displaystyle \frac{{\sum }_{i,j}\mathbf{1}[\mathbf{m}(i,j)=k]}{H\times W}}$$

(13)

We assign semantic importance levels based on class-adaptive thresholds:

$${\mathrm{level}}_k=\left\{\begin{array}{cc}\mathrm{high}\hfill & \mathrm{if}{r}_k\ge {\tau }_k^{high}\hfill \\ \mathrm{medium}\hfill & \mathrm{if}{\tau }_k^{medium}\le r_k<{\tau }_k^{high}\hfill \\ \mathrm{low}\hfill & \mathrm{if}{\tau }_k^{low}\le r_k<{\tau }_k^{medium}\hfill \\ \mathrm{None}\hfill & \mathrm{if}{r}_k<{\tau }_k^{low}\hfill \end{array}\right.$$

(14)

The thresholds ${\tau }_k$ are empirically determined based on typical class distributions in aerial imagery. For instance, vehicles require lower thresholds (${\tau }_{vehicle}^{low}=0.01$) due to their smaller spatial footprint, while buildings and roads use higher thresholds (${\tau }_{building}^{low}=0.10$).

Structured Prompt Generation We select the top-3 most prominent scene elements based on pixel ratios and construct structured LLM input:

$$\mathbf{S}={\{{\mathrm{element}}_i,{\mathrm{level}}_i,r_i\}}_{i=1}^3$$

(15)

where elements are sorted by descending ratio: $r_1\ge r_2\ge r_3$. The LLM input combines scene context with quantitative analysis:

$$\mathrm{Input}=\{\text{city\_type}:c,\mathrm{weather}:w,\mathrm{time}:t,\text{scene\_elements}:\mathbf{S}\}$$

(16)

where c represents the urban/suburban classification, w specifies the target weather condition (snowy), and t indicates the temporal context (day/night). The LLM generates structured textual descriptions that serve as conditioning prompts for ControlNet training:

$$\mathrm{prompt}=\mathrm{LLM}\left(\mathrm{Input}\right)=\mathrm{Qwen}3\text{-}14\mathrm{B}(c,w,t,\mathbf{S})$$

(17)

This generated prompt is subsequently encoded by the CLIP text encoder to produce the text embeddings ${\mathbf{e}}_{text}$ used in Equations (10), (11) and (12)

4. Experiments

Existing aerial image segmentation datasets, such as ISPRS Vaihingen and Potsdam [10,11], serve as widely-used benchmarks, offering high-resolution, annotated images of urban environments. While these datasets are invaluable for training and evaluating segmentation models, they have significant limitations in real-world applications. A key issue is the lack of diversity in environmental conditions. As a result, it does not accurately reflect the variability present in real-world aerial imagery [5]. Consequently, models trained on these datasets often struggle with domain shifts—environmental changes like weather or lighting variations that can drastically reduce segmentation accuracy.

In real-world scenarios, such as disaster response or urban planning, aerial images are frequently taken under challenging conditions, including overcast, fog, snow. The absence of such environmental diversity in standard datasets limits the robustness and adaptability of segmentation models when deployed in dynamic environments. To address this shortcoming, there is a need for a new dataset that not only mirrors the spatial characteristics of datasets like ISPRS but also includes diverse weather conditions to simulate domain shifts.

The experimental evaluation in this section is organized as follows: We first introduce the ISPRS Vaihingen and Potsdam datasets [10,11] in Section 4.1. In Section 4.2, we demonstrate the weather change caused domain shift effects on model performances using the Vaihingen dataset. Section 4.3 and Section 4.4 introduce our proposed AWSD dataset and the implementation details of our LAST/MSDM models. In Section 4.5, we conduct ablation study to verify our generated data effectiveness and generalization capability, including intra-distribution experiments and cross-distribution experiments. Finally, in Section 4.6, we present a comprehensive study to demonstrate the domain adaptation effects.

4.1. ISPRS Dataset

The International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen dataset and Potsdam datasets [10,11] are widely used benchmarks from the ISPRS 2D Semantic Labeling Contest.

The Vaihingen dataset consists of high-resolution true orthophotos of Vaihingen, Germany, with a ground sampling distance (GSD) of 9 cm. It includes 33 image tiles, 16 of which are annotated with six semantic categories: impervious surfaces, buildings, low vegetation, trees, cars, and clutter (background). The Potsdam dataset offers a finer GSD of 5 cm, containing 38 tiles of diverse urban scenes with the same six-class annotation scheme. Specifically, the original high-resolution images were processed into non-overlapping 512×512 patches. The resulting Vaihingen dataset contains 344 patches for training and 398 for validation and Potsdam dataset contains 3,456 patches for training and 2,016 for validation.

For our experiments, we mainly use Vaihingen dataset for numeric comparison, including domain shift effect demonstration (details in Section 4.2) and final comparison study (details in Section 4.6). Meanwhile we cooperate both Vaihingen and Potsdam dataset for the capacity verification (details in Section 4.5) of synthetic data generated by LAST and MSDM.

4.2. Effect of Weather Change caused Domain Shift

To demonstrate the effect of weather change caused domain shift, using the original ISPRS training set, we trained the following nine semantic segmentation models: UperNet with three different backbones (Swin Transformer [65], ResNet-50 [97], and ViT-Base [64]), DeepLabV3Plus-ResNet-50, DANet-ResNet-50, PointRend-ResNet-50, FCN-ResNet-50, Segmentor-ViT-Base, and PSPNet-ResNet-50 [56,59,63,66,69,72,98]. We then evaluated their performance on five domains: the original ISPRS validation set and its style-transferred counterparts {overcast, foggy, dusty, snowy} generated by LAST and MSDM. The overview results are illustrated in the radar chart of Figure 2 and detailed numeric results in mIOU and mFscore are shown in Table 1 and Table 2, respectively.

Table 1 and Table 2 reveal a clear pattern of performance degradation (Compared to the original performance under clear weather) as domain shift severity increases. Under overcast conditions, where image content remains unchanged but illumination is slightly reduced, all models experience performance drops with average mIoU and mFscore deterioration of 3.35% and 2.62% respectively. When atmospheric conditions are further compromised in foggy and dusty scenarios—where both illumination changes and reduced atmospheric visibility occur—more severe domain shift leads to progressively worse performance, with drops of 3.92%/3.24% and 10.55%/8.66% for mIoU/mFscore respectively. The most dramatic degradation occurs under snowy conditions, where scene targets are partially occluded or color-altered (e.g., snow covering rooftops), resulting in substantial performance drops of 28.59% and 25.76% for mIoU and mFscore respectively. These results underscore the critical impact of domain shift on semantic segmentation performance, even when the underlying scene structure remains unchanged.

Notably, our analysis reveals that ViT-based backbones demonstrate superior domain robustness compared to CNN-based alternatives. UperNet-ViT-B exhibits the best resilience under mild weather variations with minimal drops (1.00% mIoU under overcast/foggy conditions), while Segmenter-ViT-B shows the most robust performance under severe conditions (dusty: 4.32% mIoU drop, snowy: 23.95% mIoU drop), significantly outperforming ResNet-50 based models which suffer up to 30.81% mIoU degradation under snowy conditions.

4.3. Synthetic Dataset

To rigorously evaluate model performance under domain shift, we introduce the Aerial Weather Synthetic Dataset (AWSD), a synthetic dataset created using Unreal Engine 5 [55]. AWSD is designed to replicate realistic urban environments modeled based on the Potsdam and Vaihingen datasets. Images are captured from a 200-meter aerial perspective, maintaining consistency with the original benchmarks in terms of viewpoint and object layout. Visual examples of our synthetic data are presented in Figure 6.

In contrast to the static, clear-sky conditions of the ISPRS datasets [10,11], AWSD incorporates a diverse range of environmental variations, including challenging weather conditions as well as different illumination settings. As shown the samples from Figure 6, from top to bottom, we modulate the weather from overcast, foggy to the dusty based on Unreal Engine 5 environment [55]. These scenarios were purposefully introduced to assess the adaptability of segmentation models to significant domain shifts. Crucially, AWSD retains the same pixel-level semantic annotations across the six urban categories as ISPRS, ensuring a fair and precise evaluation for both small and large objects in complex environments.

Therefore, by systematically introducing these varied scenarios, AWSD directly addresses the challenge of domain generalization. Its synthetic nature enables the consistent and controllable simulation of environmental variations that are difficult and costly to capture in real-world data acquisition solutions. This makes AWSD a valuable resource for developing and benchmarking aerial segmentation algorithms with enhanced robustness for real-world applications.

4.4. Model Implementation Details

Detailed Set up of LAST The LAST model (introduced in Section 3.2) uses both source-domain content images and target-domain style references. We use ISPRS Vaihingen as the source-domain content images, while the target-domain style is derived from the 1386 synthetic images from our AWSD dataset (462 images for each weather condition: overcast, foggy, dusty), which were generated using Unreal Engine 5 [55].

The latent style transformer (Section 3.2.2) is trained for 160,000 iterations on two NVIDIA RTX 4090 24GB GPUs using the Adam optimizer [99] with learning rate 5e-4 and learning rate decay 1e-5. The batch size is set to 8. To preserve their pre-trained representations, the parameters of both the VAE (Section 3.2.2) and the perceptual VGG-19 feature extractor (Section 3.2.1) remain frozen throughout the training process.

Moreover, during DomainShifter’s Tool Selection and Execution process in Section 3.1.2, we use a strength parameter $\alpha \in [0,1]$ in post-processing that controls style intensity through linear interpolation between the transferred/stylized output and the original content image. This process can be formally described as:

$$x_{final}=\alpha ·x_t+(1-\alpha )·x_c$$

(18)

where $x_{final}$ represents the final output image, $x_t$ is the transferred image from LAST and $x_c$ is the original content image (see Section 3.2).

Detailed Set up of MSDM MSDM approach (details in Section 3.3) leverages both visual and textual information as generation condition. We utilize ISPRS Vaihingen as the source domain image set along with its corresponding segmentation masks as image input. Subsequently, based on these segmentation masks, LLM-assisted scene descriptors generate the corresponding text input. We implement the training pipeline with Accelerate for distributed training support. The ControlNet model is initialized from the pre-trained segmentation ControlNet checkpoint from Huggingface and then fine-tuned on our weather-augmented dataset.

The base model uses Stable Diffusion v1.5 [21]. To preserve pre-trained knowledge, the VAE encoder/decoder, U-Net, and CLIP text encoder parameters are frozen, while only ControlNet parameters are optimized, significantly reducing memory requirements. We employ standard MSE loss between predicted and ground truth noise in the latent space. For optimization, we employ 4 × RTX5090 32GB GPUs with AdamW optimizer [100] (learning rate: 1e-5, weight decay: 0.01, constant warm-up scheduler). The LLM (used in Section 3.3.2 operates with temperature $T=0.7$ and 200-token limit, producing 70-100 word prompts optimized for diffusion model performance. During inference, the model is implemented on two RTX 4090 24GB GPUs with device mapping set as auto. Generation uses DDIM [95] sampling with 30 steps and guidance scale 7.5. Finally, we generate 5 different sets of snowy scene data with random seeds {46, 50, 51, 53, 54}.

4.5. Ablation Study of Synthetic Data Verification

To evaluate the effectiveness and transferable domain adaptation ability of synthetic data generated by Multi-Weather DomainShifter, we augment both the original ISPRS Vaihingen training set and validation set with generated four different domain images. Meanwhile, we generate the various domain data for ONLY Potsdam validation set, because Potsdam contains only the original data but not any weather shifted data.

For simplicity, we take the prevalent Deeplabv3+ [63] model with ResNet-50 as backbone [97] for the ablation study and conduct the following comprehensive seven experiments explained below as Exp. 1 to Exp. 7, where the numerical results of Exp. 1 to Exp. 7 are shown in Table 3 and Table 4. Meanwhile the following abbreviations respectively denote VN: Vaihingen, Ori: Original, VN Weather (w/o. snow): atmospheric changed data, i.e., overcast, foggy and dusty, VN All Weather (w. snow): All synthetic data in Vaihingen Training set. Notably, for snowy scene generation by Diffusion Model based MSDM, we average the 5 sets generated results.

Exp.1 

Train model on only original Vaihingen training set and test on all domains validation set of Vaihingen;

Exp.2 

Train model on both original Vaihingen training set and LAST generated atmospheric changed data, i.e., overcast, foggy and dusty, abbreviated in VN weather (w/o. snow);

Exp.3 

Train model on all the Vaihingen Domain data, including the generation from LAST and 5 different set of snowy scene from MSDM, abbreviated in VN ALL Weather (w. snow);

Exp.4 

Train model on only original Potsdam training set and test on all domains validation set of Potsdam;

Exp.5 

Train model on both original Potsdam training set and Vaihingen training set;

Exp.6 

Train model on both original Potsdam training set and the same various domain data in Exp.2, i.e., VN weather (w/o. snow);

Exp.7 

Training model on original Potsdam training set and all domains training sets in Exp.3, i.e., VN ALL Weather (w. snow).

In general, the ablation study are divided into two main stages. We first conduct intra-distribution validation within the same geographic distribution (Exp.1-3 for Vaihingen Dataset) and cross-distribution validation by transferring the weather knowledge from generated data in Vaihingen training set into a new, unseen geographical distribution, i.e., ISPRS Potsdam dataset (Exp.4-7).

Stage 1: Intra-Distribution Domain Adaptation

The results from Experiments 1-3 demonstrate the effectiveness of synthetic weather data augmentation for enhancing domain adaptation capabilities within the same geographical distribution.

The introduction of atmospheric weather data without snow (Exp.2) yields consistent performance gains: +3.43% mIoU for overcast, +4.40% mIoU for foggy, and a remarkable +14.31% mIoU improvement for the harsher dusty conditions. Similarly, mFscore improvements of +3.04%, +4.03%, and +12.01% are observed for overcast, foggy, and dusty conditions, respectively. The comprehensive weather augmentation including snow data (Exp.3) further enhances model robustness, achieving a substantial +19.39% mIoU and +17.44% mFscore improvement in snowy conditions while maintaining competitive performance across other weather scenarios. Comparing the baseline model trained solely on original Vaihingen data (Exp.1) with the weather-augmented configurations reveals substantial improvements across all atmospheric conditions.

Stage 2: Cross-Distribution Knowledge Transfer

The cross-distribution validation experiments (Exp.4-7) provide crucial evidence that synthetic data introduces genuine weather-related knowledge rather than causing data leakage artifacts that refers to the potential issue where performance improvements might result from the model simply memorizing training data patterns or exploiting unintended correlations between training and validation sets, rather than learning generalizable weather-related visual features. This stage validates the generalization capability of weather-specific features learned from synthetic data.

A critical observation emerges from Exp.5, in which adding only original Vaihingen real data to Potsdam training set results in performance degradation for overcast (-2.95% mIoU, -2.51% mFscore) and foggy (-3.99% mIoU, -3.16% mFscore) conditions compared to the Potsdam baseline (Exp.4). Meanwhile, dusty domain performance shows improvement (+10.19% mIoU, +10.20% mFscore) and snowy performance remains almost unchanged, suggesting that simple dataset combination without weather-specific augmentation provides limited domain adaptation benefits.

The additional atmospheric weather data without snow (Exp.6) demonstrates the effectiveness of weather-specific knowledge transfer. Performance is restored for overcast and foggy conditions while achieving dramatic improvements in dusty domain (+27.98% mIoU, +26.01% mFscore). Notably, snowy scene performance remains comparable to baseline (+1.23% mIoU, +1.39% mFscore), confirming that without snow-specific training data, the model cannot effectively adapt to snowy conditions through other weather augmentations alone.

The final evaluation (Exp.7) utilizing all synthetic weather data validates the full potential of the proposed approach. Compared to the Potsdam baseline (Exp.4), substantial improvements are achieved across all weather conditions: +2.12% mIoU and +1.86% mFscore for overcast, +1.72% mIoU and +1.50% mFscore for foggy, +29.92% mIoU and +27.47% mFscore for dusty, and +5.87% mIoU and +6.34% mFscore for snowy conditions.

The systematic comparison between Experiments 6-7 against both the baseline (Exp.4) and control group (Exp.5) provides compelling evidence for the effectiveness and generalization capability of synthetic weather data. The results conclusively verify this paper’s essence that incorporating additional domain-specific synthetic data significantly enhances model domain adaptation ability and robustness against domain shift.

Moreover, the successful cross-distribution transfer from Vaihingen to Potsdam particularly demonstrates that synthetic weather knowledge generalizes beyond the original geographical context, indicating that the generated data captures fundamental weather-related visual features rather than dataset-specific artifacts. This generalization capability is essential for practical deployment scenarios where models encounter diverse geographical and environmental conditions not represented in the original training distribution.

4.6. Comprehensive Study of Domain Adaptation

Finally, we employ all the generated domain data to re-train and benchmark all the nine semantic segmentation models on ISPRS Vaihingen dataset [10,11]. The comprehensive results are detailed in Table 5 and Table 6. In addition, for better visualization, the comparison between degradation results caused by domain shift in Table 5 and Table 6 and Table 1, Table 2 is shown in the radar chart of Figure 8. Moreover, some samples of prediction are also visualized in Figure 7, where we still take Deeplabv3+ [63] here for simplicity. In particular, we sampled two images from validation set of ISPRS Vaihingen dataset [10,11] and compared their prediction results under diverse weather domain with only original training data (denotes as Prediction w/o. Synthetic data, i.e., domain shift results) and with additonal all the synthetic data from {overcast, foggy, dusty and snowy} (denotes as Prediction w. Synthetic data, i.e., domain adaptation results).

Figure 7. Visualization of predictions results of two set of samples from Vaihingen dataset. Notably, the Snowy1 and Snowy2 are generated from random seed 46 and 51 respectively. (High-resolution figure, zoom in for a better view).

Figure 7. Visualization of predictions results of two set of samples from Vaihingen dataset. Notably, the Snowy1 and Snowy2 are generated from random seed 46 and 51 respectively. (High-resolution figure, zoom in for a better view).

Figure 8. Visualization of domain adaptation recovery (Green Line) with generated various domain-specific data compared to domain shift (Red Line) on the ISPRS Vaihingen datasets. This comparison study, reports both mIoU (left) and mFscore (right) metrics. We average the results of 9 prevalent segmentation methods. (High-resolution figure, zoom in for a better view).

Figure 8. Visualization of domain adaptation recovery (Green Line) with generated various domain-specific data compared to domain shift (Red Line) on the ISPRS Vaihingen datasets. This comparison study, reports both mIoU (left) and mFscore (right) metrics. We average the results of 9 prevalent segmentation methods. (High-resolution figure, zoom in for a better view).

Our primary evaluation focuses on the robustness and domain adaptation capacity of these models against domain shifts. By retraining on the augmented data, the models exhibit significant performance improvements on the shifted validation sets. As demonstrated in Figure 7, the predictions (i.e., segmentation masks) of Deeplabv3+ without any synthetic data (shown in the column Prediction w/o. Synthetic data) gradually deteriorate as the weather conditions progress from easy to difficult. When the weather becomes dusty, where illumination and atmospheric conditions are altered, the shadow regions in Image1 and Image2 cannot be correctly handled by the model. Additionally, when the weather shifts to snowy scenes where objects (e.g., the tree in Image1 and buildings in Image2) become white or are covered by snow, the model fails to classify them correctly. In contrast, under the same conditions, after retraining the model with synthetic data from all shifted domains, the model demonstrates clear robustness across all weather conditions (shown in the column Prediction w. Synthetic data).

Numerically, compared to the deterioration results in Table 1 and Table 2 (Red Line in the radar chart of Figure 8), substantial improvements are achieved across all weather conditions: +2.75% mIoU and +2.47% mFscore for overcast, +3.48% mIoU and +3.20% mFscore for foggy, +10.53% mIoU and +8.89% mFscore for dusty, and +17.55% mIoU and +15.74% mFscore for snowy conditions. The results demonstrate significant performance improvement with the synthetic weather data, meanwhile, the model performances on the original real set keeps steady. The comprehensive results demonstrate the effectiveness of our proposals for improving robustness against various domain shifts and enhancing domain adaptation capability.

5. Conclusions

In this paper, we have addressed the critical challenge of domain shift in aerial image segmentation by proposing Multi-Weather DomainShifter, a comprehensive multi-weather domain transfer system. Equipped with AWSD, LAST and MSDM, our approach leverages synthetic style transfer coordinated by an LLM agent to enhance model robustness without requiring manual annotation of target domain data. The LLM agent coordination proves valuable in automatically selecting appropriate tools based on specific domain transfer requirements, making the system more practical for real-world deployment where diverse weather conditions need to be handled systematically.

Extensive experiments on nine state-of-the-art segmentation models demonstrate the effect of weather change caused domain shift, then ablation study validate our approach’s effectiveness, and the final comprehensive study demonstrate significant improvements in domain-shifted scenarios while preserving performance in the original domain.

As a result, this paper provides a practical and cost-effective solution for improving the real-world deployability of aerial segmentation models, addressing the critical gap between laboratory benchmarks and practical applications in diverse environmental conditions.