Enhancing Privacy-Preserving of Heterogeneous Federated Learning Algorithms Using Data-Free Knowledge Distillation

1. Introduction

With the advancement of wireless communication networks enabling the Internet of Things (IoT), edge clients are now equipped with increasingly sophisticated sensing, communication, computing, and analysis capabilities [1]. To address data-driven learning tasks arising from various real-world problems, artificial intelligence (AI), particularly deep learning and deep neural networks [2], has been widely adopted for feature extraction and model building to enhance decision-making. This approach is expected to benefit numerous application fields, including smart healthcare [3], smart manufacturing [4], smart cities [5], and smart agriculture [6]. In the traditional centralized deep learning framework, large volumes of raw data collected from edge clients are directly transmitted to a central server for the development of comprehensive deep learning models. However, this data often contains sensitive information (e.g., healthcare records, private photos), posing significant privacy risks. Consequently, ensuring data security and privacy is one of the most critical concerns in this context.

Federated Learning (FL) [7] is an approach to distributed machine learning that enhances privacy by transmitting gradients or model parameters rather than raw data. This framework exchanges parameters between the server and each client, enabling the server to learn from the clients’ private data without directly accessing it. According to the Data Processing Inequality [8], communicating via parameters— which are deterministic mappings of local data—reduces the risk of compromising data privacy compared to sharing raw data. However, recent research has demonstrated that even parameter-based communication in FL can still lead to privacy breaches. For instance, multiple studies have shown that adversaries can perform membership inference attacks (MIA) using the transmitted model parameters [9,10,11], thereby compromising data privacy and anonymity. Additionally, attackers can fully reconstruct training data through data reconstruction attacks (DRA) [12,13,14,15,16], posing significant privacy threats to FL systems. Several innovative privacy-preserving techniques have been proposed to mitigate these risks. For example, [17] introduces a decentralized FL scheme that combines quality-based aggregation with an extended dynamic contribution broadcast encryption (DConBE) and local differential privacy. [18] proposes an adaptive FL framework that integrates adaptive gradient descent with differential privacy mechanisms. To address the dual challenges of ensuring provable privacy guarantees while minimizing communication and computational overheads in FL, especially for linear regression models, [19] suggests a differential private FL scheme. Lastly, [20] presents the Noising before Model Aggregation FL framework, which enhances differential privacy by adding artificial noise to parameters at the client’s side before aggregation.

In addition to privacy concerns, a significant challenge in FL is data heterogeneity, where clients’ data exhibit Non-Identical and Independent Distributions (Non-IID). Non-IID data refers to substantial variations across clients in terms of statistical properties, such as data distributions, feature representations, and class imbalances. This heterogeneity complicates model training, as models trained on Non-IID data may struggle to generalize effectively across different clients or accurately represent the overall patterns within the entire dataset. Conventional FL algorithms, such as FedAvg [21], have been observed to result in divergent local models and a considerable loss of global knowledge under Non-IID conditions, leading to decreased performance and slower convergence [22,23]. To address this issue, the concept of Knowledge Distillation (KD), as introduced by [24], offers a potential solution by transferring knowledge from teacher (client) models to a student (global) model. For instance, FedDF [25] employs ensemble distillation for model fusion, where the global model is trained using the averaged logits from local models. FedAUX [26] enhances this approach by determining model initialization for local models and assigning weights to local model logits using ($ϵ$, $\delta$)-differential private certainty scoring. FedBE [27] takes a Bayesian approach, generating multiple global models from local models and then merging them into a single global model through ensemble KD. FedFTG, as introduced in [28], addresses the Non-IID challenge by training a GAN generator as a data generation model and fine-tuning the global model with the data produced by the GAN. Similarly, FedGen [29] trains a lightweight generator to mitigate the impact of Non-IID data on the global model’s performance.

Deploying differential privacy mechanisms for preserving privacy while simultaneously utilizing KD methods to address heterogeneity is a challenging task. This difficulty arises because distilling knowledge from noisy clients is not possible, and basically, privacy and model performance are inversely proportional, meaning that improving privacy typically comes at the cost of model performance, such as accuracy. Consequently, centralized learning often outperforms FL applications in terms of overall performance due to the added privacy constraints in FL. In this work, we propose a novel method, FedSGAN, to tackle the Non-IID challenge in heterogeneous systems while maintaining the privacy of clients against DRA concurrently. FedSGAN employs a dual decomposition optimization approach to tackle the privacy challenge and leverages a Generative Adversarial Network (GAN) model to distill knowledge between clients by generating pseudo data. Consequently, FedSGAN effectively addresses the Non-IID challenge while concurrently preserving client privacy. The primary contributions of this work are outlined as follows:

• We present an innovative method for enhancing client model performance through data-free knowledge distillation.
• We modified GAN training procedure to generate pseudo data, effectively facilitating knowledge transfer between clients to tackle Non-IID challenges.
• We utilize dual decomposition optimization technique to protect clients’ private data against DRA.
• FedSGAN simultaneously enhances client model performance and preserves privacy.

2. Preliminaries

In this paper, our methodology tackles Non-IID challenges by distilling knowledge between clients and employing dual decomposition to preserve client parameter privacy against DRA. This section elucidates the characteristics of the methods utilized in the development of the FedSGAN approach.

2.1. Dual Decomposition Optimization

Dual decomposition optimization, as proposed in [30], is a privacy-preserving optimization technique designed to address multi-constraint problems, such as those encountered in FL. In FL, the set of active clients is denoted as A where $\left|A\right|=N$, and each client,$I\in A$, has access to its local private dataset, $D_i$. The goal for each client is to determine the optimal weight vector ${\theta }_i\in {\mathcal{R}}^m,$ solving the following optimization problem:

$$\underset{\theta }{min}\sum _{i=1}^N{\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right).$$

(1)

In this context, C denotes the classifier, $\sigma$ represents the softmax function, and $\mathcal{L}$ refers to the loss function employed for computing the error of the model. Here, ${\theta }_i$ represents the prediction score for $(X,Y)$, where X is the input data and Y is its ground truth label, with $(X,Y)\in D_i$. Additionally, N signifies the total number of clients. For utilizing the Dual Decomposition, the Equation (1) is reformulated into an equivalent constrained optimization problem that mathematically shares a common optimal value which each client aims to solve the following problem:

$$\begin{array}{c}\hfill \underset{\theta }{min}\sum _{i=1}^N{\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right).,\\ \hfill \mathrm{subject}\mathrm{to}{\theta }_i={\theta }_j,\forall i,j\in A\end{array}$$

(2)

In Equation (2), each client i holds a copy of the weight vector ${\theta }_i$, and all these copies are required to agree with the mentioned constraint. However, it can be noticed this constraint in 2 can be replaced with pairs of double-sided inequality constraints. Therefore, the Equation (2) can be rewritten as a standard constrained distributed optimization problem within equality constraints as follows:

$$\begin{array}{c}\hfill \underset{\theta }{min}\sum _{i=1}^N{\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right).,\\ \hfill \mathrm{subject}\mathrm{to}\sum _{i=1}^N{e}_i\left({\theta }_i\right)\le 0,\end{array}$$

(3)

where $e_i(·)\in {\mathcal{R}}^m\to {\mathcal{R}}^{2m(N-1)}$ can be designed in infinitely possible ways to represent the equality constraint in Equation (2). For instance, Algorithm 3 provides a detailed explanation for calculating the output vector of $e_i\left({\theta }_i\right)$.

The $e_i(·)$ function represents one of the infinitely many possible realizations that can satisfy the constraint in Equation (3). In Algorithm 1 , $\left[\mathbf{0}\right]$ denotes the zero vector of size m. Considering Equation (3) and the Lagrange multipliers vector with zero initial value, denoted as ${\lambda }_i$ for client i, along with the ${\nu }_i$ vector calculated as ${\nu }_i=m·e_i\left({\theta }_i\right)$, the training in client i process can be performed as follows:

• Clients randomly initialize ${\theta }_i\in {\mathcal{R}}^m$.
• Clients initialise ${\lambda }_i\in {\mathcal{R}}^{2m(N-1)}$ with zero value and ${\nu }_i\in {\mathcal{R}}^{2m(N-1)}$ with $m·e_i\left(thet{a}_i\right)$ value and send them to the server.
• The $\overline{\lambda }$ and $\overline{\nu }$ are calculated as Equations (4)-(4), transferred to the clients from server in each communication round.

  $$\overline{\lambda }\leftarrow {\displaystyle \frac{1}{N}}.\left(\sum _{i=1}^N{\lambda }_i\right)$$

  (4)

  $$\overline{\nu }\leftarrow {\displaystyle \frac{1}{N}}.\left(\sum _{i=1}^N{\nu }_i\right).$$

  (5)
• Client Calculates the loss value :

  $${f_c}_i\leftarrow {\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right))$$

  (6)
• Clients fine-tune its gradient of loss value as below:

  $$u_i\leftarrow ▿{f_c}_i+▿e_i{\left({\theta }_i\right)}^T\overline{\lambda }.$$

  (7)
• Clients update its weight with learning rate ${\alpha }_c$ as below:

  $${\theta }_{update}\leftarrow {\theta }_i-{\alpha }_c·u_i.$$

  (8)
• The new ${\lambda }_i$ and ${\nu }_i$ calculate with hyperparameter, $\sigma$, form Equations (9)–(11) , and transfer to the server.

  $$q_i\leftarrow \overline{\nu }-\sigma .{\alpha }_c.\overline{\lambda }.$$

  (9)

  $${\lambda }_i\leftarrow [\overline{\lambda }+{\alpha }_c.q_i].$$

  (10)

  $${\nu }_i\leftarrow \overline{\nu }+m(e_i\left({\theta }_{update}\right)-e_i\left({\theta }_i\right).$$

  (11)
• Repeat steps 3 to 7 till the end of the training phase.

Algorithm 1 Calculating the $e_i(·)$.

Inputs: The client local model parameters ${\theta }_i$, m is the size of ${\theta }_i$, N is the total number of clients. Output: Vector with size of $2m(N-1)$ Case 1: $i=1$ $$\left\{\begin{array}{c}{e}_i\left[1\right]={\theta }_i\hfill \\ e_i\left[2\right]=-{\theta }_i\hfill \\ e_i\left[j\right]=\left[\mathbf{0}\right],\hfill & \mathrm{for}j=3\mathrm{to}2(N-1)\hfill \end{array}\right.$$ Case 2: $i=N$ $$\left\{\begin{array}{cc}{e}_i\left[j\right]=\left[\mathbf{0}\right],\hfill & \mathrm{for}j=0\mathrm{to}2(N-2)\hfill \\ e_i[2N-3]=-{\theta }_i\hfill \\ e_i[2N-2]={\theta }_i\hfill \end{array}\right.$$ Case 3: $1<i<N-1$ $$\left\{\begin{array}{cc}{e}_i\left[j\right]=-{\theta }_i,\hfill & \mathrm{if}j=2i-3\hfill \\ e_i\left[j\right]={\theta }_i,\hfill & \mathrm{if}j=2i-2\hfill \\ e_i\left[j\right]={\theta }_i,\hfill & \mathrm{if}j=2i-1\hfill \\ e_i\left[j\right]=-{\theta }_i,\hfill & \mathrm{if}j=2i\hfill \\ e_i\left[j\right]=\left[\mathbf{0}\right],\hfill & \mathrm{otherwise}\hfill \end{array}\right.\left\{\begin{array}{c}\mathrm{Let}\mathrm{index}=2(i-1)\hfill \\ e_i\left[j\right]=\left\{\begin{array}{cc}-{\theta }_i,\hfill & \mathrm{if}j=\mathrm{index}\hfill \\ {\theta }_i,\hfill & \mathrm{if}j=\mathrm{index}+1\mathrm{or}j=\mathrm{index}+2\hfill \\ -{\theta }_i,\hfill & \mathrm{if}j=\mathrm{index}+3\hfill \\ \mathbf{0}],\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \end{array}\right.$$

2.1.1. Assumption.1

For all $i\in A$, we assume that the $e_i(·)$ are not disclosed to the server. In other words, the server is unaware of the exact realization of $e_i(·)$ for all $i\in A$.

2.2. GAN

Generative Adversarial Networks (GANs), introduced by [31], are a class of deep learning models designed to generate new data samples that resemble a given training dataset. GANs have gained significant attention in privacy-preserving data generation due to their ability to generate pseudo data that preserves the statistical properties of the original data without revealing sensitive information. This pseudo data can then be used for the training phase without compromising privacy. A GAN consists of two neural networks, namely a Generator $\mathcal{G}$ with a set of model parameters denoted as $\omega$ and a Discriminator $\mathcal{D}$, which are trained simultaneously. Through an adversarial process, the generator takes as input a random noise vector z sampled from a prior distribution, which typically is a uniform or Gaussian distribution. The generator maps this noise vector to the data space, aiming to produce pseudo data indistinguishable from real data. As illustrated in Figure 1, the discriminator receives either a real data sample $(X,Y)$ or a generated sample $X^{\prime }$ as input and outputs a probability representing the likelihood that the input is real rather than generated.

The GAN framework is mathematically formulated as a minimax optimization problem, where the objective is to find an equilibrium between the generator and the discriminator. The objective function for GANs can be expressed as Equation (12) where $\sim \mathcal{N}(0,1)$ and $Y\sim p\left(Y\right)$:

$$\underset{\omega }{min}\underset{\mathcal{D}}{max}{\mathbb{E}}_x\left[log\mathcal{D}\left(X\right)\right]+{\mathbb{E}}_{z,Y}[1-log\mathcal{D}\left(\mathcal{G}(\omega ,z,Y)\right)]$$

(12)

The generator aims to minimize the loss function by generating samples that maximize the discriminator’s error, while the discriminator tries to maximize the loss function by correctly classifying real and generated samples.

3. Methodology

In this section, we delineate the FedSGAN training phase with Non-IID clients. Figure 2 represents that each client initializes two models: a local model for the FL objective task, denoted as $\theta \in {\mathcal{R}}^m$, and a generator model, denoted as $\mathcal{G}$ with model parameters set $\omega$, aimed at enhancing local model performance under Non-IID conditions.

During each communication round, clients employ a local optimizer loss function, $\mathcal{L}$, to train their local model on their private dataset, $D_i$. Furthermore, the local model at each client is refined using pseudo data generated by $\mathcal{G}$. Unlike conventional FL methods, where clients transfer their local model gradients or parameters, ${\theta }_i$, for aggregation, FedSGAN clients employ the modified dual decomposition optimization method, as explained in the Section 2.1. They transfer only ${\lambda }_i$ and ${\nu }_i$ vectors, and as a result, FedSGAN protects clients from sharing their local model parameters, thereby defending them against DRA. The FedSGAN client-side and server-side processes are further explained in the subsequent subsections.

3.1. Client Side

Clients in FedSGAN utilize the decomposition optimization method described in Section 2.1 for aggregation and to keep their data privacy against DRA. But decomposition optimization causes the performance loss as it shown in Table which compare the decomposition FL mehtod with FedAVG and FedProx, the FedAVG and FedProx have better accuracy on MNIST classification and also decomposition FL method could not address the Non-IID challenges in Heterogeneous systems. Therefore, The proposed FedSGAN utilizes the $\mathcal{G}$ model with GAN architecture to distillate knowledge from other clients and improve the performance. During the training phase, $\mathcal{G}$ gets noise z, and y grand true label for generating pseudo data, $x^{\prime }$, as described in Equation (13).

$$x^{\prime }=\mathcal{G}(z,Y;{\omega }_i).$$

(13)

The ${\omega }_i$ is the $\mathcal{G}$ model parameter for client i, z is Gaussian noise $z\sim \mathcal{N}(0,1)$ and $Y\sim p\left(Y\right)$. The number of grand true data, $(X,Y)\in D_i$, with label Y label is denoted as ${n^i}_Y$ and $p\left(Y\right)$ for generating pseud data is calculated from Equation (14) while it is objective to each client and is not disclosed to the server. In Equation (14), for each class, $R{n}_Y$ represents a random number for each label Y and is significantly greater than $n_Y$.

$$p\left(Y\right)\propto \sum _{j=1}^{{n^i}_Y}{\mathbb{E}}_{(X_j,Y_j)\sim {\mathcal{D}}_i}\left[1_{Y_j=Y}\right]=R{n}_Y.$$

(14)

The pseudo data $x^{\prime }$ generated by $\mathcal{G}$ is employed to fine-tune the client loss function, ${f_c}_i$. By evaluating the loss of the local model, ${\theta }_i$, on this pseudo data, ${F_g}_i$, as detailed in Equation (15), the performance loss induced by decomposition optimization is mitigated. Moreover, the $p\left(Y\right)$ distribution compels ${\omega }_i$ to generate pseudo data that guide the client towards a more uniform data distribution for training. This approach mitigates the drift during aggregation caused by the Non-IID challenge, as demonstrated by the experimental results in the subsequent section.

$${f_c}_i\leftarrow {\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right))+\underset{{f_g}_i}{\underbrace{\mathcal{L}\left(\sigma \right(C({\theta }_i,x^{\prime })}}.$$

(15)

After calculating the ${f_c}_i$, the $u_i$ is computed from Equation (7) and then clients updates their model parameters, ${\theta }_i$, with ${\alpha }_c$ learning rate from Equation (8). In the end, refer to the Equations (9)–(11) clients update the ${\lambda }_i$ and ${\nu }_i$ for transferring them to the server for aggregation.

FedSGAN addresses the privacy performance loss and Non-IID challenge by using ${\omega }_i$ during the training phase, but training the ${\omega }_i$ still remains a critical task. The $\mathcal{G}$ can not use client’s local private dataset, $D_i$, for training ${\omega }_i$ because in the Non-IID situation training ${\omega }_i$ on the clients private dataset caused to generated pseudo data in a similar distribution as client has and it can not help us to address the Non-IID. Therefore, for training the ${\omega }_i$, FedSGAN modified the formal GAN architecture as illustrated in Figure 3. In taring phase of ${\omega }_i$, FedSGAN remove Discriminator $\mathcal{D}$, Figure 3, and utilize ${f_g}_i$ loss value as it describe in Equation (15). ${F_g}_i$ calculates the loss of the local model on the pseudo data generated by ${\omega }_i$. Clients transfer the gradient of ${f_g}_i$ to the server for aggregation and receive the $F_g$ from the server for updating the ${\omega }_i$ with ${\alpha }_g$ learning rates as described in Equation (16). The method for calculating $F_g$ from $▿{f_g}_i$ is detailed in Equation (17) in the server side subsection.

$${\omega }_i\leftarrow {\omega }_i-{\alpha }_g.F_g.$$

(16)

As illustrated in Figure 2, clients in FedSGAN, in addition to transferring the ${\lambda }_i$ and ${\nu }_i$ to a server in the training phase, also transfer the $▿{f_g}_i$ for training their ${\omega }_i$. The step-by-step FedSGAN training phase on the client side has been shown in Algorithm 2.

Algorithm 2 FedSGAN, Client side.

Inputs: Average Lagrange multiplier vector: $\overline{\nu }$, Average Lagrange dual vector: $\overline{\lambda }$ ,Average gradient loss function of generator model: $F_g$, Local model learning rate: ${\alpha }_c$, Generator model learning rate: ${\alpha }_g$, $\sigma$. repeat       for all client i∈N in parallel do             Client $i^{th}$ receive $\overline{\nu }$, $\overline{\lambda }$ and $F_g$ from server.             ${\omega }_i\leftarrow$ from Equation (16).             $(z,y)\leftarrow (z\sim \mathcal{N}(0,1),Y\sim p(Y)$ ).             $x^{\prime }\leftarrow$ from Equation (13).             ${f_g}_i\leftarrow$ from Equation (15).             ${f_c}_i\leftarrow$ from Equation (15).             $u_i\leftarrow ▿{f_c}_i+▿e_i{\left({\theta }_i\right)}^T\overline{\lambda }$             $q_i\leftarrow$ from Equation (9)             ${\theta }_{update}\leftarrow {\theta }_i-{\alpha }_c.u_i$             ${\lambda }_i\leftarrow$ from Equation (10).             ${\nu }_i\leftarrow$ from Equation (11).             ${\theta }_i\leftarrow {\theta }_{update}.$             client sends ${\nu }_i,{\lambda }_i$ and $▿{f_g}_i$ to the server.       end for until training stop

3.1.1. Remark.1

The Assumption. 1 Section 2.1.1 ensures that when executing Algorithm 2, the server is unable to reconstruct ${\theta }_i$, thereby preventing access to information about the clients’ private local dataset $D_i$. It is important to note that the $▿{f_g}_i$ transferred between clients and the server represents the gradient of the loss function of the local model ${\theta }_i$ on the pseudo data $x^{\prime }$ generated from Equation (13). Consequently, state-of-the-art privacy attacks can only reconstruct information about $x^{\prime }$ (the pseudo data) and not the clients’ private data. Additionally, due to ${R_n}_Y>>{n^i}_Y$ in Equation (14), the distribution of the private data $D_i$’s ground truth labels cannot be reconstructed from the pseudo data’s ground truth label distribution. Therefore, Algorithm 2 strengthens the privacy of FedSGAN by safeguarding the clients’ private data.

3.2. Server Side

As previously mentioned, unlike conventional FL methods, FedSGAN protects client DRA by avoiding the transfer of the client’s local model parameters, ${\theta }_i$, to the server. Consequently, there is no global model at the server side for training or aggregation. Additionally, the server lacks any information about the clients’ data distributions and the $e_i(·)$ function used for decomposition optimization, as explained in Remark.1Section 3.1.1, thereby ensuring the privacy of FedSGAN’s clients. Server received the ${\lambda }_i$, ${\nu }_i$ and $▿{f_g}_i$ from clients and process to calculate the average of these parameters as formulated in Equations (4), (5), (17) and send back the $\overline{\lambda }$,$\overline{\nu }$ and $F_g$ to the clients for updates their models. The server-side FedSGAN algorithm has been represented in Algorithm 3.

$$F_g\leftarrow {\displaystyle \frac{1}{N}}.\left(\sum _{i=1}^N▿{f_g}_i\right).$$

(17)

Algorithm 3 FedSGAN, Server side.

Inputs: clients average Lagrange multiplier vector: ${\left({\nu }_i\right)}_{i=1}^N$ , clients average Lagrange dual vector: ${\left({\lambda }_i\right)}_{i=1}^N$ and clients average gradient loss function of generator model:$▿{\left({f_g}_i\right)}_{i=1}^N$, Number of communication round: T. for t=1,...,T do             Collect ${\nu }_i,{\lambda }_i$ and ${f_g}_i$ from clients.             $\overline{\lambda }\leftarrow$ from Equation (4)             $\overline{\nu }\leftarrow$ from Equation (5)             $F_g\leftarrow$ from Equation (17)             sends $\overline{\lambda },\overline{\nu }$ and $F_g$ to the clients. end for

4. Experiments

This section will elucidate the implementation aspects of FedSGAN and provide a comparative analysis of its performance against state-of-the-art methods for addressing Non-IID challenges. Additionally, a comparison of its performance in situations involving DRA with state-of-the-art methods will be conducted in Section 4.2.

4.1. Implementation Details

In this section, we conduct a comparative analysis to evaluate the performance of FedSGAN against other notable related works. Detailed implementation specifics and comprehensive experimental results are discussed in the subsequent subsections. In addition, the Python source code of FedSGAN is accessible via the following link: https://anonymous.4open.science/r/FedSGAN-2816/README.md.

4.1.1. Reference Methods

In this study, we compare FedSGAN with several significant related works, including FedAvg [7], FedProx [32], SCAFFOLD [33], FedFTG [28], FedGen [34], FedRand [35], FedKD [36], and f-differential [37]. Table 1 outlines the features of these state-of-the-art methods and demonstrates that FedGen and FedFTG utilize KD to address the Non-IID challenge. FedFTG employs a classification model for the client’s local model $\sigma \left(C\left({\theta }_i\right)\right)$, utilizing ResNet18 [38], and uses a GAN architecture [31] for its $\mathcal{G}$ model. Similarly, FedGen also employs ResNet18 as the local model and utilizes a simple Autoencoder model for its $\mathcal{G}$ architecture. Notably, FedAvg, FedProx, SCAFFOLD, FedGen, and FedFTG do not mention any techniques in their methodologies to ensure client privacy against DRA. Meanwhile, the FedKD local model employs a Multi-Layer Perceptron (MLP) architecture and utilizes the KD for performance improvement but does not employ any generator model.

4.1.2. FedSGAN Networks Architecture

FedSGAN, similar to FedFTG and FedGen, has ResNet18 architecture for clients local model $\sigma \left(C\left({\theta }_i\right)\right)$. Additionally, The GAN proposed in [39] is employed as the $\mathcal{G}$ model.

4.1.3. Datasets

The effectiveness of FedSGAN is evaluated using the CIFAR-10 [40] and MNIST [41] datasets, which feature heterogeneous dataset partitions. To emulate Non-IID data distribution among clients, a practice adopted by previous studies [28,42,43,44], we utilize the Dirichlet distribution Dir($\psi$) on label ratios. Here, a smaller value of $\psi$ signifies increased data heterogeneity. For FedSGAN implementation, and $\psi =0.3,0.6$.

4.1.4. Differential Privacy

We also compare FedSGAN with reference methods under differential privacy conditions as proposed by [45]. This comparison involves deploying Gaussian noise with varying privacy budgets ($ϵ$) to the clients’ weight models to protect their privacy against DRA.

4.1.5. Hyperparameters

The number of clients $N=100$, number of local training epochs 50, communication rounds $T=1000$, for local training, the batch size is 50. Moreover the ${\alpha }_c$ and ${\alpha }_g$ are both initialized to 0.1 and $\sigma$ is $0.3$ . The dimension of X and $x^{\prime }$ are $32\times 32\times 3$ for CIFAR-10 and $28\times 28\times 1$ for MNIST.

4.2. Performance Comparison

In this section, we first compare the proposed FedSGAN without the $\mathcal{G}$ model, utilizing only decomposition optimization, to the full implementation containing $\mathcal{G}$. This analysis highlights the significance of the $\mathcal{G}$ model and utilizing KD method, particularly in the context of Non-IID clients. Subsequently, we conduct a comparative analysis of FedSGAN’s accuracy against existing state-of-the-art methods in three scenarios: I. with Non-IID clients and without privacy consideration, II. with implemented privacy measures and all clients are IID, and III. with Non-IID clients while considering private training by utilizing the DP method.

4.2.1. Comparison of FedSGAN Model Without Utilizing KD Method

We compare the accuracy of FedSGAN with and without using the KD method. The experimental results of this comparison across different values of $\psi$ are illustrated in Figure 4 for the MNIST and CIFAR-10 datasets. As shown in Figure 4, the accuracy of FedSGAN with the KD method significantly surpasses without KD, particularly in the presence of Non-IID clients. This highlights the importance of the KD method in enhancing FedSGAN’s performance.

4.2.2. I.Performance Without Privacy Consideration

The overall accuracy comparison between FedSGAN and other state-of-the-art methods is presented in Table 2 for the CIFAR-10 and MNIST datasets across two various $\psi$ values. All experiments were conducted using three random seeds. As shown in Table 2, FedSGAN surpasses most of the methods for $\psi =0.3$ and $\psi =0.6$, except for FedFTG. FedFTG demonstrates strong performance in Non-IID client FL applications. However, as illustrated in Figure 5 which presents a comparison of test accuracy across various $\psi$, the accuracy of FedSGAN is comparable to FedFTG when the $\psi$ value is close to 1, indicating clients are IID. For $\psi$ values less than $0.3$, SCAFFOLD’s accuracy exceeds that of FedSGAN, likely due to FedSGAN’s approach of not transferring clients’ local model parameters ${\theta }_i$ Unlike other methods, and utilizing decomposition optimization for privacy considerations. Nonetheless, FedSGAN’s accuracy remains competitive with state-of-the-art methods, particularly for non-IID clients in CIFAR-10 classification applications, and it shows minimal differences from FedFTG.

4.2.3. II.Performance with Privacy Consideration

The strength of FedSGAN lies in its ability to ensure client privacy against DRA while maintaining performance in heterogeneous systems, as it mentioned in Remark.1Section 3.1.1. Differential privacy is a well-known method utilized to address the privacy challenges of DRA in FL applications. Table 3 compares various state-of-the-art FL methods that employ differential privacy techniques with FedSGAN, using different $ϵ$ values on the MNIST dataset. As shown, for lower $ϵ$ values, the accuracy of f-differential slightly surpasses FedSGAN by less than $1\%$. However, for higher $ϵ$ values, FedSGAN outperforms all methods with significant difference and remains independent of $ϵ$ values because it maintains privacy without relying on differential privacy techniques. Like FedSGAN, FedAKD employs the KD method to preserve accuracy, but its performance does not compare favorably to FedSGAN.

Table 4 compares state-of-the-art methods with accuracy levels similar to FedSGAN in MNIST classification, now applied to CIFAR-10 classification across varying $ϵ$ values. This comparison underscores how FedSGAN performs relative to these leading methods in maintaining accuracy while addressing privacy concerns. Unlike in MNIST classification, where FedSGAN’s accuracy is comparable, in CIFAR-10 classification, FedSGAN’s accuracy is notably higher than that of other methods, demonstrating significant performance improvements at larger $ϵ$.

4.2.4. III.Performance with Privacy Consideration and Non-IID Clients

FedSGAN effectively preserves client privacy against DRA while simultaneously addressing the challenges posed by Non-IID clients. As discussed in previous subsections, its performance in independent scenarios, Non-IID environments, and privacy protection against DRA is comparable with state-of-the-art methods. We now compare FedSGAN’s performance with other leading methods that either outperform or match FedSGAN in the previously discussed scenarios. Figure 6 illustrates its accuracy in scenarios where both privacy must be maintained and clients are Non-IID, with a $\psi$ value of 0.5 for CIFAR-10 classification. Based on Remark 1Section 3.1.1, FedSGAN ensures privacy against DRA, whereas other methods utilize differential privacy with varying $ϵ$ values for comparison. As shown in Figure 6, FedSGAN demonstrates a significant advantage over other methods at $ϵ$ values greater than zero, specially in compare to FedFTG that has better performance than FedSGAN when $\psi =0.5$. However, its performance under differential privacy is significantly reduced and less effective than FedSGAN, underscoring FedSGAN’s strength for real-world federated learning applications.

5. Discussion

The primary limitation of FedSGAN lies in the parameters transferred between clients and servers. As detailed in Methodology section, FedSGAN clients transmit ${\lambda }_i$, ${\nu }_i$, and $▿{f_g}_i$ instead of their model parameters. While the size of $▿{f_g}_i$ depends on the client’s model architecture, the parameters ${\lambda }_i$ and ${\nu }_i$ have a size of ${\mathcal{R}}^{2m(N-1)}$, which depends on the number of clients, N. In scenarios with a large number of clients and bandwidth constraints, FedSGAN becomes unsuitable due to the large data size transferred between clients and the server. To address this challenge, one potential solution involves optimizing the parameter communication process with techniques such as parameter compression, or quantization, which can significantly reduce the data volume without compromising model performance.

6. Conclusion

We have introduced a novel data-free knowledge distillation approach in FL that effectively balances accuracy and privacy preservation. By transmitting alternative parameters (${\lambda }_i$, ${\nu }_i$, and $▿{f_g}_i$) instead of direct model parameters, FedSGAN maintains privacy against DRA while addressing the challenges posed by Non-IID client distributions. Leveraging a GAN model, FedSGAN efficiently transfers knowledge between clients, overcoming the discrepancies of Non-IID data. FedSGAN achieves over $50\%$ better accuracy compared to the FedFTG method, which uses the same data-free knowledge distillation approach, when $\psi =0.5$ and $ϵ=10$ in CIFAR-10 classification, demonstrating significant improvements in overall accuracy. Comprehensive evaluations across two benchmark datasets confirm the effectiveness of FedSGAN, positioning it as a promising solution for privacy preservation with Non-IID clients in FL applications.

Appendix A.1. Optimization Problem Formulation

The optimization problem for FedSGAN is formulated as follows:

$$\begin{array}{c}\hfill \underset{\theta }{min}\sum _{i=1}^N{\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right).,\\ \hfill \mathrm{subject}\mathrm{to}{\theta }_i={\theta }_j,\forall i,j\in A\end{array}$$

(A1)

This can be reformulated into a set of inequalities:

$$\begin{array}{c}\hfill {\theta }_i={\theta }_j\to \left\{\begin{array}{cc}{\theta }_i-{\theta }_j\le 0& \forall i,j\in A\\ -{\theta }_i+{\theta }_j\le 0& \forall i,j\in A\end{array}\right.\end{array}$$

(A2)

By using Equation (A2) we could defined the $e_i(·)$ function as explained in Equation (A3) which had describe in Algorithm1. Additionally, from Equation (A3) we impose the following constraint:

$$\begin{array}{c}\hfill \sum _{i=1}^N{e}_i\left({\theta }_i\right)\le 0,\end{array}$$

Appendix A.2. Lagrangian Formulation

With relies on [30], the Lagrangian for the optimization problem 3 could formulate given by:

$$\begin{array}{c}\hfill \mathbf{L}(\theta ,\lambda )=\sum _{i=1}^N\left({\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right)\right)+{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(e_i\left({\theta }_i\right){\lambda }_i\right)\end{array}$$

where ${\lambda }_i$ are the dual variables associated with the constraint ${\sum }_{i=1}^N\left(e_i\left({\theta }_i\right)\right)\le 0$.

Appendix A.3. Gradient Descent Update

The parameter update rule using gradient descent is:

$$\begin{array}{c}\hfill {\theta }_{update}\leftarrow {\theta }_i-{\alpha }_c(▿\mathbf{Loss})\end{array}$$

Expanding this for the specific Primal update, we have:

$$\begin{array}{c}\hfill {\theta }_{update}\leftarrow {\theta }_i-{\alpha }_c\underset{u_i}{\underbrace{(▿{\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right)+▿e_i{\left({\theta }_i\right)}^T\underset{\overline{\lambda }}{\underbrace{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\lambda }_i}})}}\end{array}$$

where ${\alpha }_c$ is the learning rate, ${\mathcal{L}}_{X\sim D_i}$ is the loss function for the data distribution $D_i$.

Moreover Let ${\theta }^{*}$ denote the optimal solution of 3. We define the regret functions as follows:

$$\begin{array}{c}\hfill Re{g}^{\mathcal{L}}\left(T\right)\stackrel{△}{=}\sum _{i=1}^N\left(\sum _{k=1}^T(\mathcal{L}{\left(\sigma \left(C({\theta }_i,X)\right)\right)}_k-\mathcal{L}\left(\sigma \left(C({\theta }^{*},X)\right)\right))\right)\end{array}$$

$$\begin{array}{c}\hfill Re{g}^e\left(T\right)\stackrel{△}{=}\left|\right|\left[\sum _{i=1}^N\sum _{k=1}^T{e}_i{\left({\theta }_i\right)}_k\right]\left|\right|\end{array}$$

where T represents the number of communications. A successful FL algorithm must ensure that:

$$\begin{array}{c}\hfill \underset{T\to \infty }{lim}{\displaystyle \frac{Re{g}^{\mathcal{L}}\left(T\right)}{T}}=0\end{array}$$

$$\begin{array}{c}\hfill \underset{T\to \infty }{lim}{\displaystyle \frac{Re{g}^e\left(T\right)}{T}}=0\end{array}$$

The success criteria, as expressed through , articulate the convergence of regrets over time, indicating the effectiveness and convergence properties of the FL algorithm. The following theorem guarantees the desired performance.

Appendix A.3.1. Theorem.1

Suppose for all clients $i\in A$, we assume that the respective gradients $\mathcal{L}(·)$ and $e_i(·)$ are convex and uniformly bounded. In other words, there exists a positive constant C such that $\left|\right|▿{\mathcal{L}}_{X\sim D_i}\left(\sigma \left(C({\theta }_i,X)\right)\right)\le C$ and $\left|\right|▿e_i\left({\theta }_i\right)\left|\right|\le C$, where $\left|\right|▿e_i\left({\theta }_i\right)\left|\right|$ represents a Jacobian matrix at ${\theta }_i$. Consider that FedSGAN Algorithm 2 is executed by all clients with $\alpha ={\sigma }^{-1}{T}^{-\beta }$, where $\sigma =2N(N{C}^2+1)$ and $\beta \in (0,1)$.Then the limits introduced in tend to 0.

Appendix A.3.2. Proof:

The aggregation process in the server during Algorithm3 closely resembles the scenario where clients communicate in a complete graph, assigning ${\displaystyle \frac{1}{N}}$ to their incoming edges. In this context, the graph’s adjacency matrix is doubly stochastic and strongly connected. Consequently, noticing the proposed values for ${\alpha }_c$ and $\sigma$, the fulfillment of above assumption satisfies all the essential conditions outlined in Theorem.1 and as a result, it holds concluding the proof.

$$\begin{array}{c}\hfill \underset{T\to \infty }{lim}{\displaystyle \frac{Re{g}^{\mathcal{L}}\left(T\right)}{T}}=0\end{array}$$

$$\begin{array}{c}\hfill \underset{T\to \infty }{lim}{\displaystyle \frac{Re{g}^e\left(T\right)}{T}}=0\end{array}$$

$$\begin{array}{c}\hfill \underset{e_1\left({\theta }_1\right)}{\underbrace{{\left[\begin{array}{c}{\theta }_1\\ -{\theta }_1\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ ⋮\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\end{array}\right]}_{2m(N-1)}}}+\underset{e_2\left({\theta }_2\right)}{\underbrace{{\left[\begin{array}{c}-{\theta }_2\\ {\theta }_2\\ {\theta }_2\\ -{\theta }_2\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\end{array}\right]}_{2m(N-1)}}}+\underset{e_3\left({\theta }_3\right)}{\underbrace{{\left[\begin{array}{c}\left[\mathbf{0}\right]\\ \mathbf{0}]\\ -{\theta }_3\\ {\theta }_3\\ {\theta }_3\\ -{\theta }_3\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\end{array}\right]}_{2m(N-1)}}}+\underset{e_4\left({\theta }_4\right)}{\underbrace{{\left[\begin{array}{c}\left[\mathbf{0}\right]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ -{\theta }_4\\ {\theta }_4\\ {\theta }_4\\ -{\theta }_4\\ ⋮\\ \mathbf{0}]\end{array}\right]}_{2m(N-1)}}}+\cdots +\underset{e_{N-1}\left({\theta }_{N-1}\right)}{\underbrace{{\left[\begin{array}{c}\left[\mathbf{0}\right]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ ⋮\\ -{\theta }_{N-1}\\ {\theta }_{N-1}\\ {\theta }_{N-1}\\ -{\theta }_{N-1}\end{array}\right]}_{2m(N-1)}}}+\underset{e_N\left({\theta }_N\right)}{\underbrace{{\left[\begin{array}{c}\left[\mathbf{0}\right]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ \mathbf{0}]\\ ⋮\\ -{\theta }_N\\ {\theta }_N\end{array}\right]}_{2m(N-1)}}}\le 0\end{array}$$

(A3)