Federated Learning for XSS Detection: Analysing OOD, Non-IID Challenges, and Embedding Sensitivity

1. Introduction

Cross-site scripting (XSS) attacks remain a persistent security threat due to their widespread occurrence and ease of exploitation [8]. Machine learning-based detection, including reinforcement learning [7,17] and ensemble learning [6,38], has advanced significantly, with earlier studies [4,6,12] and more recent works [1,3,5,10,38] focusing on improving model architectures and feature extraction.

However, many methods still face generalisation issues due to the highly distributed data structure and privacy concerns. Federated Learning (FL) has emerged as a privacy-preserving alternative, allowing collaborative training without exposing raw data. This study explores the use of FL for XSS detection, addressing key challenges such as non-independent and identically distributed (non-IID) data, heterogeneity and out-of-distribution (OOD). While FL has been applied in cybersecurity [11,18], its role in XSS detection remains underexplored. Most prior works focus on network traffic analysis, rather than text-based XSS payloads.

This study presents the first systematic application of federated learning to XSS detection under text-based XSS threat scenarios. Our key contributions are.

• We design a federated learning (FL) framework for XSS detection that simulates structurally non-IID client distributions, incorporating diverse XSS variants (e.g., Reflected, Stored, DOM-based), obfuscation styles, and potential attacks. This setup emulates real-world conditions where specific clients contain mostly partial or ambiguous XSS indicators with low detection rates, while others have clearer attack patterns. Importantly, this structural asymmetry extends beyond positive samples. We find that negative class heterogeneity plays a critical and underexplored role in triggering generalisation failures. Our FL setup thus enables a novel investigation of bidirectional structural OOD, where complex, fragmented negatives induce high false positive rates under mismatched distributions.
• Unlike prior work that interleaves lexical and contextual features across splits, we preserve strict structural separation between training and test datasets. By incorporating an external dataset [57] as a reference OOD domain, we isolate and assess bi-directional distributional shifts between positive and negative samples under federated settings. Our analysis reveals that generalisation failure often arises not from rare or obfuscated attacks, but from structurally diverse benign samples that dominate the negative class. This provides new insight into the limitations of conventional dataset design and the critical role of structure-aware generalisation.
• We conduct a comparative study of three embedding models (GloVe [24], CodeT5 [26], GraphCodeBERT [25]) in centralised and federated settings, revealing that model generalisation is governed less by capacity than by the compatibility between embedding structure and the heterogeneity of both classes. Through divergence metrics (JSD, Wasserstein, MMD, TF-IDF similarity) and ablation studies, we expose how structurally complex negatives, especially those underrepresented during training, can induce severe false positive spikes. We further demonstrate that static embeddings like GloVe exhibit more robust generalisation under structural OOD, suggesting that model stability is tightly linked to representation resilience rather than expressiveness alone.

2. Related Work

Existing research on federated learning (FL) for XSS detection remains scarce. The most relevant work by Jazi & Ben-Gal [2] investigated FL’s privacy-preserving properties using simplified setups and traditional models (e.g., MLP, KNN). Their non-IID configuration assumes an unrealistic “all-malicious vs. all-benign” client split, and evaluation is conducted separately on a handcrafted text-based XSS dataset [57] and the CICIDS2017 intrusion dataset [28]. However, they do not consider data heterogeneity or OOD generalisation. Still, the dataset [57] they selected is structurally rich and thus serves as a suitable OOD test dataset in our experiments (see Section 3.2).

Heterogeneity in datasets remains a significant challenge for XSS detection [14,15,39,61]. The absence of standardized datasets, particularly in terms of class variety and sample volume, can have a substantial impact on the decision boundaries learned by detection models [60,64]. Most existing studies, including [3,4,5,10], attempt to address this issue through labor-intensive manual processing, aiming to ensure strict control over data quality, feature representation, label consistency, and class definitions.

However, we argue that complete reliance on manual curation often fails to reflect real-world conditions. In practical cybersecurity scenarios, data imbalance is both common and inevitable, especially regarding the ratio and diversity of attack versus non-attack samples [60,61,62]. This often results in pronounced structural and categorical divergence between positive and negative classes. For example, commonly used XSS filters frequently over-filter benign inputs [63], indicating a mismatch between curated datasets and actual deployment environments.

In light of these challenges, federated learning demonstrates strong potential. It enables models to share decision boundaries through privacy-preserving aggregation [33,56], offering an effective alternative to centralized data collection and manual intervention.

Meanwhile, we argue that findings from FL research on malicious URL detection [9,37] are partially transferable to XSS detection. Although some malicious URLs may embed XSS payloads, the two tasks differ in semantic granularity, execution contexts, and structural variability. Given their shared challenges like class imbalance, distribution shift, and non-IID data, we think FL techniques proven effective for URL detection offer a reasonable foundation for XSS adaptation.

The high sensitivity of XSS-related information such as emails or session tokens, makes sharing difficult without anonymisation. Yet studies [53,54] show that anonymisation often introduces significant distributional shifts due to strategy-specific biases. Disparities in logging, encoding, and user behaviour further distort data distributions, compromising generalisation [53,54].

For example, strings embedded in polyglot-style payloads are hard to anonymise, as minor changes may affect execution. Consider the following sample:

<javascript:/*-><img/src='x'onerror=eval(unescape(/%61%6c%65%72%74%28%27%45%78%66%69%6c%3A%20%2b%20%27%2b%60test@example.com:1849%60%29/))>

Naively replacing "test@example.com" with an unquoted *** breaks JavaScript syntax, rendering the sample invalid and misleading detectors. While AST-based desensitisation can preserve structure, it is complex, labour-intensive, and lacks scalability [52].

To address these challenges, this study introduces a federated learning (FL) framework to enhance XSS detection while preserving data privacy, especially under an OOD scenario. FL enables collaborative training without exposing raw data [11,56], mitigating distributional divergence and improving robustness [56,59]. More importantly, our approach leverages structurally well-aligned, semantically coherent clients to anchor global decision boundaries, allowing their generalisation capabilities to be implicitly shared across clients with fragmented, noisy, or ambiguous data distributions. In doing so, we avoid the need for centralised, large-scale anonymisation or sanitisation, and instead provide low-quality clients with clearer classification margins without direct data sharing or manual intervention. This decentralised knowledge transfer mechanism forms the basis of our FL framework, detailed in Section 5, and evaluated under dual OOD settings across three embedding models. Section 4 will explain the Centralized OOD testing,

3. Methodology and Experimental Design

3.1. Settings and Rationale

Please see Figure 1 for the project pipeline and Figure 2 for the overall paper logic flow.

3.1.1. Experiment Environment

Our experiments are based on the FLOWER framework [19], an open-source system for simulating federated learning that supports various federated learning (FL) schemes and aggregation algorithms, including FedAvg [21], FedProx [22], and robust methods such as Krum [23]. The experiments were conducted on the JADE2 high-performance computing (HPC) cluster, using a single NVIDIA V100 GPU (32GB) per run (used average ram 16GB for FL training). As JADE2 is a multi-user shared system, Centralized Training time varied between 0.1-0.5 hours and Federated training time varied between 0.5 - 2 hours, depending on system load and job scheduling conditions. (Typical time cost 2882.32s for GloVe with FedAvg, 4614.06s for GraphCodeBERT with FedAvg)

3.1.2. Embedding Selection Rationale

To evaluate the effectiveness of different natural language processing techniques in OOD XSS detection, we selected three representative word embedding paradigms:

• GloVe-6B-300d (static embedding): A word embedding model that maps words to fixed-dimensional vectors based on co-occurrence statistics.
• GraphcodeBERT-base (BERT-based, pre-trained on code): A bidirectional transformer model designed for code and mixed text-code inputs, well-suited for representing structured XSS payloads in web scripts.
• CodeT5-base (sequence-to-sequence, code-aware): A unified encoder-decoder model pre-trained on large-scale code corpora. In our setting, we utilize the encoder component to extract contextual embeddings. CodeT5 captures both local and global structural patterns through its masked span prediction and identifier-aware objectives, making it suitable for modeling fragmented or obfuscated payloads that lack explicit syntax trees.

Unlike GraphCodeBERT, which relies heavily on syntax-level alignment, CodeT5 learns a broader structural abstraction that generalizes better to heterogeneous inputs. This makes it particularly effective in detecting distributional shifts in structurally diverse or OOD payloads commonly seen in federated XSS detection scenarios.

For practical considerations, we adopted mid-sized variants of each model to ensure computational feasibility and compatibility with federated learning environments. Larger-scale state of art (SOTA) models such as GPT-3/3.5/4 [43] and DeepSeek-coder-1B/6.7B [44], while potentially more expressive, are prohibitively expensive in terms of inference cost and memory footprint, even when used solely for frozen embedding. Such overhead renders them unsuitable for decentralised training settings, especially when synchronous inference across heterogeneous clients is required.

In addition, to ensure a fair and interpretable comparison, we intentionally avoided mixing model scale and design improvements. The selected models strike a practical balance between representation power and computational efficiency, enabling a focused evaluation of embedding characteristics without introducing confounding factors or excessive system complexity.

3.1.3. Freeze Emebedding

Despite the potential for improved downstream performance, we intentionally avoid fine-tuning the embedding models (e.g., CodeT5, GraphCodeBERT) in our pipeline. This design choice reflects both practical and privacy-driven considerations.

In typical and classical FL settings, model training must occur on decentralised clients where raw data cannot be aggregated. Fine-tuning pre-trained models typically requires centralised access to data and intensive resources, which contradicts FL’s privacy-preserving assumptions.

Furthermore, recent studies [40,41,45] have demonstrated that fine-tuning can amplify privacy leakage risks by recovering previously “forgotten” personal information from language models (LMs). They will also increase the FL computation cost and complexity [45]. Therefore, we use frozen embedding models to better align with real-world FL deployments, where privacy and generalisation must coexist without heavy centralised retraining, and to reduce the risk of inference attacks [46] that exploit model updates to extract senssitive client information.

3.1.4. Downstream Classifier

The downstream classifier is a unified light transformer model with d_model = 256, nhead = 8, num_encoder_layers = 3, dim_feedforward = 512, dropout = 0.1, learning rate = 0.001. Batch_size = 64. The input dimensions of the three word-embedding models used are 768 for both CodeT5 and GraphcodeBERT, and 300 for GloVe, respectively. We used Cross Entropy Loss for both Centralised and FL tests.

3.1.5. Optimization and Aggregation

We applied FedAvg and FedProx with Focal Loss [34] to address client drift and imbalance in the Non-IID federated learning setting for aggregation. The Focal Loss modification helps mitigate the impact of class imbalance, particularly for rare XSS attack variants. For details, please see section 5.1.

In the overall framework, we avoided overly complex designs like federated domain adaptation [47] to minimise the influence of different factors on the advantages of federated learning. Our experiment design aims to verify the potential role of the federated learning framework in OOD XSS attack detection rather than to validate single models or approaches that have already been extensively studied and repeatedly tested, as mentioned earlier. Many of these models strongly depend on specific datasets and centralised training conditions, making them less applicable to real-world FL scenarios with non-IID, privacy-constrained data distributions. The following sections will explain the dataset preparation, the central aggregation algorithms used for federated learning, and the experimental evaluation results.

3.2. Dataset Design and Explanation

3.2.1. Dataset Construction

Based on our review of recent works [1,2,3,4,5] and several survey studies [15,20,39,57], datasets for XSS detection can be broadly categorised into two types: text-oriented and traffic-oriented. Traffic-oriented datasets (e.g., NF-ToN-IoT [27], CICIDS2017) focus on network-level intrusion features like packet metadata and response delays. In contrast, text-oriented datasets contain raw payloads, JavaScript fragments, and event handlers that more directly reflect the surface layer of XSS attacks. Unlike intrusion detection, XSS text detection lacks standardised, large-scale text benchmarks. Existing datasets are often small, task-specific, and weakly documented [20,29,30]. For example, XSS-Attacks-2019 [5] includes metadata like IP and geolocation, which are not directly relevant to payload analysis.

To support federated learning research, we used two complementary datasets: a manually curated training set (Dataset 1) and an externally sourced test set (Dataset 2) [57] with higher structural completeness in both negative samples and positive samples. Dataset 1 was collected from public sources (e.g., GitHub, OWASP, PortSwigger) and cleaned via heuristics and manual review to remove malformed or mislabeled samples (~5% of positives). It contains diverse XSS types (Reflected, Stored, DOM-based) and various obfuscation styles.

In addition, the composition of negative samples differs significantly across datasets. Dataset 1 contains many ambiguous fragments, including mixed-format inputs such as code snippets or meaningless trace embedded in free text, broken payload traces, and text from unrelated injection contexts. In contrast, Dataset 2's negative samples tend to fall into clearer categories, with harmless full URL (~50%) and plain-text entries more distinctly separated. This asymmetry introduces both lexical and structural imbalance between the two domains, amplifying generalisation difficulty under non-IID settings

No data augmentation or resampling was performed, in order to preserve naturally occurring structural fragmentation, including partial payloads, fuzzing traces, and incomplete injection chains. For example, a sample in Dataset 1 may be a broken script tag or a partial event-handler attribute, likely be captured during scanning or blocked by server filtering.

In contrast, Dataset 2 comprises well-formed, executable XSS payloads, typically embedded in query parameters or HTML contexts with full DOM closure and side effects. This allows us to study structure-level distribution shift. For both dataset’s negative samples are mainly about harmless query, codes segments, and plain text, but the negative samples in Dataset 1 is more distributed and complicated.

Dataset 2 is adopted from the public dataset by Mereani and Howe [57]. While not a benchmark in the conventional sense, its structural consistency and token-level regularity make it a suitable reference for OOD evaluation.

To simulate FL-specific challenges, we:

1. Partition Dataset 1 across five clients using intentionally non-IID splits (e.g., attack type skew, source-specific imbalance);

2. Use Dataset 2 as a structurally distinct OOD test set to evaluate generalisation across distributional shifts.

We released both raw datasets in:

https://github.com/Phillipswangbo/V1.4/tree/26dcf185a412f982cab28f8e113313ffeff565e1

The dataset is undergoing further refinement to completely ensure the observed OOD behaviour stems from data fragmentation and diversity rather than artificial perturbations.

Our dataset design was also inspired by the research of Sun's team [31], along with their formula for evaluating model generalisation errors:

$${\mathsf{\epsilon }}_{\mathrm{gen}}≔E_S{E}_A\left[R\left(A\left(S\right)\right)-\widehat{R_S}\left(A\left(S\right)\right)\right]$$

(1)

3.2.2. Dataset Partitioning and OOD Design

After being minimally cleaned (Dataset 1), our primary training dataset consists of 73,277 samples, among which 39,134 are positive XSS pure payloads or potential XSS injection points. These include a broad mix of the three classical XSS types, mostly short payload fragments or isolated injection points, rather than fully resolved attack URLs containing all necessary features (e.g., domain context, query structure, executable flow). Reflected (~77.35%), Stored (~3.38%), and DOM-based (~18.76%), alongside complex variants such as obfuscated. This diversity helps preserve a realistic distribution of XSS behaviours.

To evaluate generalisation under distributional shift, we used the external test set mentioned earlier (Dataset 2) with 42,514 samples, including 15,137 positive samples. Notably, about 94.69% of these are Reflected XSS, with Stored (~1.90%) and DOM-based (~3.41%) comprising only a tiny fraction. This concentration makes Dataset 2 structurally and semantically narrower, favouring fully formed, and long attack URL (~ 95.7% of positive cases are complete, well-structured samples).

3.2.3. Semantic-Preserving Substitution and Lexical Regularisation

In Dataset 1, we replaced high-frequency canonical payloads such as “alert” with syntactically valid but functionally diverse JavaScript APIs like prompt. See Table 1. These variants, although not strictly equivalent in runtime effect, remain plausible within XSS contexts and preserve executable structure. The substitutions were selected to expand structural diversity and better reflect real-world attack surface variability. Unlike traditional lexical regularisation that aims to preserve semantic identity, our transformation introduces controlled structural perturbations without altering the label or removing executable intent. While Dataset 2 retains conventional alert-style payloads, Dataset 1 exposes the model to more varied expressions. This design enables us to evaluate robustness under structurally diverse but semantically plausible inputs, particularly relevant for fragmented or ambiguous samples in practical deployment scenarios.

3.2.4. Quantitative Lexical-Level Analysis Reveals Distributional Divergence

To quantify lexical-level divergence between Dataset 1 and Dataset 2, we extracted top-100 TF-IDF features from 3,000 sampled samples. In positive samples, 63 features overlapped (Jaccard = 45.98%, Cosine = 0.4988), showing moderate consistency. In contrast, negative samples had only 20 overlaps (Jaccard = 10.5%, Cosine = 0.2230), reflecting greater lexical diversity. While this suggests notable variation in negative samples, We hypothesise that generalisation gaps cannot be solely attributed to this, as structural inconsistencies in positive samples also play a key role. See Table 2. For the formulation, $T_1$ refers to the Top-k TF-IDF features from Dataset 1, same to $T_2$ , the overlap count is defined as $\left| T_1\cap T_2\right|$ where $T_1$, $T_2$ denote the sets of top-k TF-IDF features in each dataset. cosine similarity between aggregated TF-IDF vectors is given by${\displaystyle \frac{\overrightarrow{v_1}\cdot \overrightarrow{v_2}}{|\overrightarrow{v_1}\left||\overrightarrow{v_2}\right|}}$, where $\overrightarrow{v_1}$ $\overrightarrow{v_2}$ represent the mean TF-IDF vectors of each dataset. However, since TF-IDF cannot effectively capture structural differences in positive samples (similar to GloVe, which also lacks structural awareness), we further employed other measurements to visualise such differences in the following paragraphs of section 4.2.

$$Overlap Count=\left| T_1\cap T_2\right|$$

(2)

$$\mathrm{Jaccard} \mathrm{Similarity}={\displaystyle \frac{\left|T_1\cap T_2\right|}{\left|T_1\cup T_2\right|}}$$

(3)

$$\mathrm{Cos}\mathrm{ine} \mathrm{Similarity}={\displaystyle \frac{\overrightarrow{v_1}\cdot \overrightarrow{v_2}}{|\overrightarrow{v_1}\left||\overrightarrow{v_2}\right|}}$$

(4)

3.2.5. Visualisation of Different Datasets’ Positive Samples

While we initially considered multiple projection methods, such as T-SNE [32], we ultimately chose UMAP [58] for this analysis, we used GraphCodeBERT embeddings, as it offers better sensitivity to structural and token-level variation in code-like or script-based inputs, which are common in XSS payloads. We focused on positive samples for visualisation since our dataset mainly contains potential payloads and a relatively minor portion of actual attacks. As shown in Figure 3, Dataset 1 appears fragmented, reflecting obfuscated or diverse payloads, while Dataset 2 forms a more compact and uniform cluster. This structural contrast supports the presence of feature-level drift across datasets.

3.3. Experimental Procedure Overview

We conducted four groups of experiments to evaluate model generalisation, feature sensitivity, and federated learning performance:

• Centralized Embedding Evaluation: We tested three embedding models, GloVe, GraphcodeBERT, and CodeT5 under centralised settings using Dataset 1 for training and Dataset 2 for testing. This setup evaluates each model’s generalisation ability to unseen attack structures in an OOD context.
• Dataset Swap OOD Test: To further explore the impact of feature distribution divergence, we reversed the datasets: training on Dataset 2 and testing on Dataset 1. This demonstrates how models trained on one domain generalise (or fail to generalise) to structurally distinct inputs.
• Federated Learning with Non-IID Clients: We simulated a more realistic extreme horizontal FL setup with five clients. Dataset 1 and Dataset 2 were partitioned across clients to introduce heterogeneous distributions. Each client was trained locally and evaluated on unseen data from the other dataset. We used FedAvg and FedProx for aggregation, evaluating accuracy, false positive rate, recall, and precision.
• Centralized Mixed-Distribution Control Test: As a control experiment, we repeated the training with no data isolation: all clients received mixed samples from both datasets. This scenario helped evaluate whether FL benefits diminish when distributional divergence is removed, shedding light on gradient dilution and homogenisation effects in federated settings.

4. Independent Client Testing with OOD Distributed Data

In the first part of our evaluation, we trained on Dataset 1 (balanced and sufficiently sized) and tested on Dataset 2, then reversed the setup. While both datasets target reflected XSS, they differ in structural and lexical characteristics, as detailed in Section 3.1. This asymmetry, present in both positive and negative samples, led to significant generalisation gaps. In particular, models trained on one dataset exhibited lower precision and increased false positive rates when tested on the other, reflecting the impact of data divergence under OOD settings.

We evaluated all three embedding models under both configurations. Confusion matrices (Figure 4 and Figure 5) illustrate the classification differences when trained on low- versus high-generalisation data, respectively. Before this, we established performance baselines via 20% splits on the original training set to rule out overfitting (Table 3). Figure 6 summarises cross-distribution performance under each model, and Figure 7 highlights the extent of performance shifts under structural OOD. These results confirm that both positive and negative class structures play a critical role in the generalisation performance of XSS detectors

To isolate the impact of positive sample structure, we conducted cross-set training where the training positives originated from the high-generalisation Dataset2 while retaining fragmented negatives from Dataset1 on the most structure sensitive model GraphcodeBERT. Compared to the baseline trained entirely on Dataset1, this setup substantially improved Accuracy (from 56.80% to 71.57%) and precision (from 44.82% to 68.39%), with Recall slightly increased to 99.70%. These findings highlight that structural integrity in positive samples enhances model confidence and generalisability even under noisy negative supervision. Conversely, negatives primarily increase false positives (FPR 68.19%). See Table 4.

4.1. Generalisation Performance Analysis

When We evaluate the generalisation ability of GloVe, GraphCodeBERT, and CodeT5 embeddings by training on the high-generalisation dataset (Dataset 2) and testing on the structurally diverse and fragmented Dataset 1.

All models experience a significant drop in performance, particularly in precision and false positive rate (FPR), indicating high sensitivity to structural shifts across datasets.

GraphCodeBERT shows the most severe performance degradation, with precision dropping from 84.38% to 45.03% (−39.35%), and FPR increasing from 19.16% to 65.62% (+46.46%). Despite maintaining nearly perfect recall (99.63%), it heavily overpredicts positives when faced with unfamiliar structures, suggesting poor robustness to syntactic variance due to its code-centric pretraining.

CodeT5 suffers slightly less, but still significant degradation: precision drops from 84.50% to 46.36% to 38.14%, and FPR rises from 18.47% to 61.95% (+43.48%). This suggests that while its span-masked pretraining aids structural abstraction, it still fails under negative class distribution shift.

GloVe demonstrates the most stable cross-dataset performance, with a precision decline from 90.13% to 51.58% (−38.55%), and FPR increasing from 11.90% to 47.90 (+36.00%). Although static and context-agnostic, GloVe is less vulnerable to structural OOD, likely due to its reliance on global co-occurrence statistics rather than positional or syntactic features.

These results support that structural generalisation failure arises from both positive class fragmentation and negative class dissimilarity. Models relying on local syntax (e.g., GraphCodeBERT) are more prone to false positives, while those leveraging global distributional features (e.g., GloVe) exhibit relatively better robustness under extreme OOD scenarios.

4.1.1. Sensitivity of Embeddings to Regularization Under OOD

Under structural OOD conditions, CodeT5 achieved high recall (≥99%) but suffered from low precision and high FPR, indicating overfitting to local patterns. Stronger regularization (dropout = 0.3, lr = 0.0005) led to improved precision (+4.73%) and reduced FPR (−10.89%), showing modest gains in robustness. GloVe benefited the most from regularization, with FPR dropping to 29.49% and precision rising to 63.41%. In contrast, GraphCodeBERT remained not very sensitive to regularization, with relatively smaller change across settings. These results suggest that structure-sensitive embeddings require tuning to remain effective under structural shift, while static embeddings like GloVe offer more stable performance. See Table 5.

4.2. Embedding Level Analysis

To assess whether embedding similarity correlates with generalisation, we computed pairwise Jensen-Shannon Divergence (JSD) [49] and Wasserstein distances (WD) [50] across models on both datasets. $P$ and $Q$: Probability distributions of two embedding sets, $M$ : Mean distribution. $\mathrm{KL}$ : Kullback–Leibler divergence from one distribution to another. $F_P\left(x\right)-F_Q\left(x\right)$：Cumulative distribution functions. $\mathrm{JSD}\left(P\parallel Q\right)$ reflects a symmetric, smoothed divergence metric capturing the balanced difference between $P$ and $Q$.

As shown in Table 6, the three embedding models respond differently to structural variation. GraphCodeBERT has the lowest JSD (0.2444) but the highest WD (0.0758), suggesting its embeddings shift more sharply in space despite low average token divergence. This sensitivity leads to poor generalisation, with false positive rates exceeding 65% under OOD tests. GloVe shows the highest JSD (0.3402) and moderate WD (0.0562), indicating broader but smoother distribution changes. It performs most stably in OOD scenarios, likely due to better tolerance of structural drift. CodeT5 has the lowest WD (0.0237), meaning its embeddings change little across structure shifts. However, this low sensitivity results in degraded precision, especially for negative-class drift.

$$\mathrm{JSD}\left(P\parallel Q\right)={\displaystyle \frac{1}{2}}\hspace{0.17em}\mathrm{KL}\left(P\parallel M\right)+{\displaystyle \frac{1}{2}}\hspace{0.17em}\mathrm{KL}\left(Q\parallel M\right),\hspace{1em}M={\displaystyle \frac{1}{2}}\left(P+Q\right)$$

(5)

$$W\left(P,Q\right)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\left|F_P\left(x\right)-F_Q\left(x\right)\right|\hspace{0.17em}dx$$

(6)

4.2.1. Kernel-Based Statistical Validation of OOD Divergence

While metrics like JSD and Wasserstein quantify distributional shifts, they do not assess statistical significance. To address this, we compute the Maximum Mean Discrepancy (MMD) between Dataset 1 and Dataset 2 using Random Fourier Features (RFF) for efficiency, with 40,000 samples per set, for details.

• $\mathrm{MMD}$ score scope for different models embedding in all samples: 0.001633 (GraphcodeBERT) - 0.108761 (CodeT5).
• In positive samples: 0.000176 (GloVe) - 0.000853 (CodeT5).
• In negative samples: 0.004105(GraphcodeBERT) - Glove (0.517704).
• All Embeddings’ $P-VALUE$< 0.001(refers to a distinct OOD)

These data confirmed a statistically significant distributional shift and semantic OOD in negative samples. For formulation, please see below. $\mathcal{X}$, $\mathcal{Y}$ refers to the set of different embeddings. $\varphi \left(x_i\right)$ means the kernel feature mapping approximated via Random Fourier Features (RFF). For $P-VALUE$, $s$ is the observed MMD score, $k$ represents the number of permutations, $s_i$ is the MMD value obtained for the i permutation.

$$\mathrm{MM}{\mathrm{D}}^2\left(\mathcal{X},\mathcal{Y}\right)=⃦{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\varphi \left(x_i\right)-{\displaystyle \frac{1}{m}}{\sum }_{j=1}^m\varphi \left(y_j\right) {⃦}^2$$

(7)

$$p={\displaystyle \frac{1+{\sum }_{i=1}^{k}I\left(s_i\ge s\right)}{k+1}}$$

(8)

Although GloVe presents the highest absolute MMD score in the negative class, this corresponds to stronger sensitivity to lexical variation rather than instability. GloVe consistently achieved lower false positive rates and more stable generalisation in downstream evaluation, indicating greater robustness under OOD scenarios. In contrast, CodeT5 and GraphCodeBERT exhibit similar MMD magnitudes and comparable degradation patterns, suggesting limited adaptability to negative-class drift despite their contextual expressiveness.

These results, supported by lexical analysis (Section 3.2) indicate that the observed generalisation gap is attributable to systematic data divergence, particularly in negative sample distributions, rather than random fluctuations.

5. Federated Learning Tests Under Non-IID Scenarios

This paragraph will investigate whether such generalisation holds under decentralised settings, to validate our original idea that Federated learning can enhance the model’s generalisation even under an OOD situation.

5.1. Federated Learning Settings

5.1.1. Dataset Distribution

The rest of the training and test splits were partitioned according to the label and sample categories described in Section 3.2, using fixed random seeds (= 42) to ensure reproducibility. We set three representative non-IID configurations: (1) clients with severe class imbalance (e.g., skewed positive/negative ratios), (2) clients with varied total data quantities and randomly sampled label distributions, and (3) clients with composite distribution skew involving both label imbalance and quantity mismatch, potentially including noisy samples. An example of the composite configuration (3) is illustrated in Figure 8. For the remaining parts of the two datasets, approximately 50% of the labels and sample sizes are evenly distributed among five clients as the test set. However, the test sets for clients 1 to 4 are derived from dataset 2, while the test set for client 5 is from dataset 1. This forms the OOD distribution. This setup reflects a realistic federated setting where label and distributional skews co-occur [48,56].

5.1.2. Federated Learning Setup

We simulated a horizontal federated learning environment consisting of five clients, each holding disjoint subsets of training data with distinct structural characteristics. Clients 1–4 are assigned structurally diverse and imbalanced samples derived from Dataset 1, characterised by syntactic irregularities and complex payload structures. Client 5, however, holds structurally regular and semantically coherent data from Dataset 2, creating a heterogeneous training landscape with inter-client label imbalance and significant structural feature skew.

All clients participate in standard federated global rounds =30, learning rate = 0.005, dropout = 0.1. FedProx with a proximal regularisation of 0.2, and native FedAvg algorithm for server-side aggregation. StepLR with a step size of 5 and a decay factor gamma = 0.5, 10 epochs for client. The current global model is distributed to all clients at the beginning of each round. Clients then perform local training using Focal Loss with alpha = 1.4, gamma = 2.0, optimised by stochastic gradient descent (SGD), subsequently uploading their updated model weights back to the server. The server aggregates these updates into a new global model.

We adopt a cross-structure testing strategy to evaluate model generalisation under structural distribution shifts. Specifically, after each aggregation round, the newly aggregated global model is redistributed to all clients, which evaluate this global model on locally maintained test sets. These test sets exhibit distributions mismatched by the respective client's training data. For instance, while Clients 1–4 train exclusively on Dataset 1, their test sets are derived from Dataset 2, representing the centralised OOD status we mentioned. Conversely, Client 5, trained on Dataset 2, evaluates on a Dataset 1 test set. Although sharing the same label space, this deliberate cross-distributional setup allows us to systematically assess the global model’s capacity to generalise across structurally distinct but semantically consistent scenarios.

Client training and evaluation occur in parallel across the federated system. Evaluation metrics, including accuracy, recall, precision, F1-score, and false positive rate (FPR), are computed locally and aggregated at the server to provide comprehensive performance insights at each federated training round.

5.1.3. Aggregation Algorithms

We adopt two standard aggregation methods to evaluate FL under non-IID settings: FedAvg and FedProx. FedAvg computes the global model as a weighted average of client updates, proportionally based on each client’s local data size. This method ensures that clients with more data significantly influence the global model, which enhances the model’s performance and generalisation ability.

FedAvg Formulas:

$$w_{t+1}={\sum }_{k=1}^K{\displaystyle \frac{n_k}{n}}{w}_t^k$$

(9)

${\mathit{w}}_{\mathit{t}+1}$: The weight of the global model after round t+1.

$\mathit{K}$: The number of participating clients.

${\mathit{n}}_{\mathit{k}}$: The data size of client k

$\mathit{n}$: The total data size across all clients

${\mathit{w}}_{\mathit{t}}^{\mathit{k}}$: The local model weight of client k after round t

FedProx is particularly suitable for non-IID settings, as it stabilises training by reducing local model drift. We include it to evaluate how regularised aggregation affects generalisation under heterogeneous XSS data.

FedProx Formulas:

$${w_{t+1}^k=\mathrm{ar}\mathrm{g}\mathrm{m}\mathrm{in}}_w\left(f_k\left(w\right)+{\displaystyle \frac{\mathsf{\mu }}{2}}|w-w_t{|}^2\right)$$

(10)

${\mathit{w}}_{\mathit{t}+1}^{\mathit{k}}$: The optimised weight of the local model on client k after round t+1.

${\mathit{f}}_{\mathit{k}}\left(\mathit{w}\right)$: The loss function for client k.

$\mathsf{\mu }$: The regularisation parameter (proximal term).

${\mathit{w}}_{\mathit{t}}$: The weight of the global model after round t.

${\mathbf{a}\mathbf{r}\mathbf{g}\mathbf{m}\mathbf{i}\mathbf{n}}_{\mathit{w}}:$ The argument of the minimum indicates that $w_{t+1}^k$ minimises the expression within the parentheses.

5.2. Federated Learning Performance

Firstly, we examined the global classifier’s performance under two different algorithms. We selected GLOVE-6B-300D as an example (the other two models showed different convergence optimisation, while GraphCodeBERT shows the most improvement), as shown in Figure 9.

Under a unified learning rate of 0.005, the global model demonstrates severe oscillation during training, regardless of the optimisation algorithm. Additional experiments show that this instability is caused by the non-adaptiveness of GloVe embeddings under client-wise OOD and non-IID conditions. To mitigate this, significantly smaller learning rates = 0.001 are required to achieve smoother convergence curves. See Figure 10.

We also observed that although the global model trained with GloVe embeddings achieves more stable convergence under a reduced learning rate (e.g., 0.001), this stability comes at the cost of slower convergence and requires more communication rounds to reach comparable performance.

For instance, after 30 rounds of aggregation, under FedProx, the global model using GloVe embeddings under a learning rate of 0.001 achieves an aggregated accuracy of 94.89% and an FPR of 6.7%, which is lower than the performance obtained under 0.005 learning rate (accuracy = 97.14%, FPR = 3.2%).

Despite this, GraphcodeBERT and CodeT5 demonstrated different effects, especially in terms of convergence stability, which contrasted sharply with their extremely poor performance under OOD. we separately record the Global classifier’s aggregated accuracy and FPR convergence curves of three embedding models, under the two aggregation algorithms. See Figure 11 and Figure 12.

FedProx improves training stability across all embeddings but slightly hinders final performance for GloVe and CodeT5. In contrast, GraphCodeBERT benefits from FedProx, showing improved final accuracy, though the CodeT5 still achieved a better performance on single client. This suggests that FedProx better aligns structural variations, which particularly helps structurally sensitive models. Table 7 reports peak global and worst client-side results under FedProx.

5.2.1. Centralised No Data Isolation Testing Baseline

We also tested the classifier performance of three different embedding models, without data isolation, to demonstrate a comparison with federated learning. In this scenario, the train dataset contains data from both Dataset 1 and Dataset 2 (25% from the high generalisation dataset, the test dataset also includes 25% from the original train dataset, dataset1), with balanced negative, positive samples. See Figure 13 and Table 8.

5.3. Federated Learning Result Analysis

This part evaluates the embedding-level performance of GloVe, GraphCodeBERT, and CodeT5 under FedProx, with special focus on convergence stability and generalisation capacity in non-IID federated learning scenarios.

Contrary to prior expectations, for GloVe, despite its strong performance in centralised OOD settings, it exhibited the most unstable training dynamics under FedProx. The global model’s accuracy oscillated sharply, and although final performance (Accuracy = 96.2%, FPR = 5.5%) was competitive, the path to convergence was highly erratic. This indicates a poor tolerance to structural divergence across clients, likely due to GloVe’s static and non-contextual nature.

GraphCodeBERT, on the other hand, delivered the most stable convergence trajectory throughout 30 rounds. Starting from a lower baseline (~77.3%), it steadily improved to reach 96.8% accuracy and 4.7% FPR. This suggests that GraphCodeBERT's structural encoding is well-suited to federated alignment, benefitting from client-specific variability rather than being hindered by it.

CodeT5 demonstrated rapid initial gains, quickly surpassing 90% accuracy in early rounds. However, it later suffered from increased fluctuation, with clear signs of overfitting or instability under client aggregation. While its final performance was strong (Accuracy = 97.6%, FPR = 3.5%), the convergence was less smooth compared to GraphCodeBERT.

We also recorded the client’s best performance improvements, Initial means the metrics of first-time aggregation result tested on single client’s test dataset, see Figure 14.

These findings strongly support our initial assumption that federated learning can achieve significant performance gains even under extreme XSS data heterogeneity (primarily on negative samples and then positive samples) when adopted with better aggregation mechanisms and structure-sensitive embeddings. The contrast between these configurations highlights the importance of architectural compatibility between local feature extraction and global aggregation in non-IID federated settings.

6. Conclusions

This This study explores the feasibility of federated learning (FL) for XSS detection under structural out-of-distribution (OOD) and non-IID conditions. We demonstrate that while FL offers privacy benefits and avoids raw data sharing, its generalisation ability is tightly linked to the behaviour of the embedding model during distributed training.

Among the three evaluated embeddings, GloVe shows the best OOD generalisation in centralized settings but suffers from unstable convergence in FL due to its static nature and high divergence across clients. CodeT5 achieves rapid early convergence but experiences performance drift in later rounds, indicating weaker robustness to client drift. In contrast, GraphCodeBERT, despite poor centralized OOD performance, it benefits most from FL aggregation. Its structure-aware design aligns well with FedProx, resulting in smooth convergence and reduced FPR over rounds.

Contrary to prior assumptions, we find that model generalisation failure arises from both negative-sample heterogeneity and fragmented or incomplete positive examples. Visualisation and distributional metrics (JSD, Wasserstein, MMD) further confirm meaningful embedding-level shifts, particularly in the negative class, validating the design of our dual OOD setup.

Overall, FL enhances global decision boundaries by diffusing stable structural priors from cleaner clients to noisier participants, reducing the impact of structural asymmetry without direct data sharing. These findings establish FL as a practical and privacy-aligned solution for OOD-resilient XSS detection, while also emphasising the critical role of embedding stability and alignment.

7. Limitations and Future Work

• Incorporating Partial Participation with Invariant Learning. Our current setup assumes synchronous client participation per round, whereas real-world FL often involves dropout or intermittent availability. While we do not explicitly simulate asynchronous updates, recent methods such as FEDIIR [55] have shown robustness under partial participation by implicitly aligning inter-client gradients to learn invariant relationships. Extending such approaches to our structure-variant OOD setting may improve robustness in realistic, non-synchronous FL environments.
• Data Quality as a Structural Bottleneck. A key challenge in federated XSS detection lies not in algorithmic optimisation, but in the difficulty of acquiring high-quality, generalisable data across all clients. Our results suggest that if no clients possess substantial structural diversity or sufficient sample representation, the global model’s generalisation ability will be severely impaired, even with robust aggregation. Federated learning in XSS detection contexts fundamentally depends on partial data sufficiency among clients. As part of future work, we plan to expand the dataset to include more structurally complex XSS payloads, especially context-dependent polyglot attacks that combine HTML, CSS, and JavaScript in highly obfuscated forms. Such samples are essential to better simulate real-world, evasive behaviours and stress-test federated models under extreme structural variability.
• Deployment Feasibility and Optimization Needs.
  While the current framework employs a lightweight Transformer classifier, future work may explore further simplification of the downstream classifier through distilled models (e.g., TinyBERT), linear-attention architectures (e.g., Performer), or hybrid convolution-attention designs to reduce computational overhead and improve real-world deplorability.