Convergence Rate Analysis of Non-I.I.D. SplitFed Learning with Partial Worker Participation and Auxiliary Networks

1. Introduction

In the conventional Federated Learning (FL), several clients in parallel, train a model jointly particularly leading to speed-up in the learning process under the supervision of a server [1]. Hence, given a central server and N clients as participants in the training, an optimization problem of the below form is solved by FL:

$$\begin{array}{c}\hfill \underset{\tilde{\mathbf{x}}\in {\mathbb{R}}^{\tilde{d}}}{min}f\left(\tilde{\mathbf{x}}\right)\stackrel{\Delta }{=}\underset{\tilde{\mathbf{x}}\in {\mathbb{R}}^{\tilde{d}}}{min}{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{F}_i\left(\tilde{\mathbf{x}}\right)\end{array}$$

(1)

In which, $F_i\left(\tilde{\mathbf{x}}\right)\stackrel{\Delta }{=}{\mathbb{E}}_{\xi \sim D_i}\left[F_i(\tilde{\mathbf{x}},\xi )\right]$ can be a non-convex loss function and $\xi$ corresponds to a random sample of local dataset of the client i, $D_i$. In the FL training, there are m clients training on their local datasets. However, FL encounters the challenge that clients must train the entire model, placing a considerable computational burden, particularly with complex and large-scale models. Additionally, gathering all client data and broadcasting the aggregated model at each round can result in substantial communication overhead. While one of the principal aims of FL is to safeguard clients’ privacy, the server retains access to both the client’s local and global models, prompting security concerns [2]. To address the computational limitations and further safeguard the privacy of the client-side model, [3] pioneered SL, dividing the ML model into two parts. The client trains one portion of the model, while the server trains the remaining portion. However, according to [2], this method incurs notable training time overhead, as only one client can engage in split learning (SL) at any given time, leaving others idle. To address this issue, they proposed SplitFed Learning (SFL), which integrates both the parallel computational capabilities of clients from FL and the benefits of split models from SL. In particular, the convergence theory of the SFL framework has not been thoroughly explored in the existing literature. Our primary contribution lies in establishing the theoretical underpinnings for SFL, incorporating the general assumptions of traditional FL, non-convex loss functions, non-identically independently distributed (non-i.i.d.) datasets, and addressing both full and partial client participation in the SFL training process. Our proof is for the state-of-the-art algorithm for SFL developed by [4] based on conventional assumptions in FL settings. We demonstrate that the SFL can still recover the linear convergence rate of conventional FL. However, changes in the number of clients and local steps cannot speed up the convergence.

2. Related work

• SL and FL
  The reference [5] introduces a personalized SL framework to address issues like data leakage and non-iid datasets in decentralized learning. It proposes an optimal cut layer selection method using multiplayer bargaining and the Kalai-Smorodinsky bargaining solution (KSBS). This approach efficiently balances the time of training, usage of energy, and privacy of data. Each device tailors its model for non-i.i.d. datasets while they have a common server-side model which ensures robustness by generalization. Simulation results validate the framework’s effectiveness in achieving optimal utility and addressing decentralized learning challenges. However, they do not address the communication overhead caused by transmitting the forward-propagation results at each local step. The reference [6] provides convergence analysis for Sequential Split Learning (SSL), a variant of SL in which the model training process is conducted sequentially, with each client trained one after the other, on heterogeneous data. It compares SSL with Federated Averaging (FedAvg) showing SSL’s superiority on extremely heterogeneous data. However, in practice, if the heterogeneity of data is mild, FedAvg outperforms SSL. Also, SSL still suffers from large communication overheads between the server and clients.
• SplitFed learning
  The reference [7] presents AdaSFL, a method designed to optimize model training efficiency by controlling local update frequency and batch size. The theoretical analysis demonstrates convergence rates, which facilitate the creation of an adaptive algorithm for adjusting update frequency and batch sizes tailored to heterogeneous workers. However, clients must obtain back-propagation results from the server at each local update. Meanwhile, [8] recommends updating client and server-side models concurrently, utilizing local-loss-based training and auxiliary networks designed specifically for split learning. This parallel training approach effectively reduces latency and eliminates the need for server-to-client communication. The paper includes latency analysis for optimal model partitioning and offers guidelines for model splitting. Specifically, [4] developed a communication and storage-efficient SFL approach. In this method, each client trains a portion of the model and calculates its local loss function using an auxiliary network, leading to reduced communication overhead. Furthermore, the server model is trained based on the sequence of forward propagation results from the clients, ensuring that only one copy of the server model is maintained at any given time. Additionally, [8] suggested a similar framework, albeit with a key difference that each client possesses its separate server model, and these models are aggregated to construct the global server model.
• Auxiliary networks
  Neural network training with back-propagation is hindered by inefficiencies arising from the update locking issue, where layers must await the complete propagation of signals through the network before updating [9]. To address this, [9] proposed Decoupled Greedy Learning (DGL), a more straightforward training approach that relaxes the joint training objective greedily, showing significant effectiveness for CNNs in large-scale image classification. This method optimizes the training objective using auxiliary modules or replay buffers to reduce communication delays caused by waiting for backward propagation. [10] addressed the backward update lock constraint by introducing a model that decouples modules through predictions of future computations within the network graph. These models use local information to predict the outcomes of subgraphs, particularly focusing on error gradients. By using synthetic gradients instead of true backpropagated gradients, subgraphs can update independently and asynchronously, realizing decoupled neural interfaces. A similar approach has been adopted for training in SFL by [4,8]. Indeed, they use an auxiliary model to replace the server model. The mentioned research demonstrates that an auxiliary model with a relatively smaller dimension compared to the server model performs sufficiently well in serving as a replacement.

3. SplitFed Learning Scenario

In this section, we introduce the SFL framework, encompassing both client and server-side models. Additionally, we present the CSE-SFL algorithm designed by [4] to mitigate communication overhead. Accordingly, we split the model as $\tilde{\mathbf{x}}({\mathbf{x}}_C,{\mathbf{x}}_S)$ where ${\mathbf{x}}_C$ denotes the client-side model, and ${\mathbf{x}}_S$ indicates the server-side model. We introduce $\mathbf{x}({\mathbf{x}}_C,{\mathbf{x}}_A)$ as a client-side model including the auxiliary network where ${\mathbf{x}}_A$ indicates the model for the auxiliary network.

The client-side non-convex loss function in the SFL setting is given by:

$$\begin{array}{c}\hfill F_i^c\left(\mathbf{x}\right)\stackrel{\Delta }{=}{\mathbb{E}}_{\xi \sim D_i}\left[F_i^c(\mathbf{x};\xi )\right]\end{array}$$

(2)

Also, the non-convex loss function in the SFL setting is defined by:

$$\begin{array}{c}\hfill F^s({\mathbf{x}}_S;{\mathbf{z}}_f,\mathbf{y})\stackrel{\Delta }{=}{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{F}_i^s({\mathbf{x}}_S;{\mathbf{z}}_{f,i},{\mathbf{y}}_i)\end{array}$$

(3)

We denote ${\mathbf{z}}_{f,i}({\mathbf{x}}_C;\xi )$ as the output of the forward propagation of the client i’s model, ${\mathbf{x}}_{C,i}$, on its local random data sample, $\xi \in D_i$, which is intended to be transmitted to the server at specific intervals including the true labels ${\mathbf{y}}_i$ corresponding to the local random data sample. Note that the sampled data at the client is not shared with the server but the true labels. Similarly, ${\mathbf{z}}_{b,i}({\mathbf{x}}_S;{\mathbf{z}}_{f,i},{\mathbf{y}}_i)$ indicates the backward propagation model of the server for client i. Accordingly, ${\widehat{\mathbf{z}}}_{b,i}({\mathbf{x}}_A;{\mathbf{z}}_{f,i},{\mathbf{y}}_i)$ corresponds to the backward propagation results obtained by the auxiliary network. In more detail, the client i performs forward propagation up to the splitting layer and transmits the output of this layer, along with the true labels, to the server. The server then continues forward propagation through to the final layer and computes the loss function. Subsequently, the server performs backward propagation of the error and sends the gradients of its first layer back to the client. We consider ${\overline{\mathbf{x}}}_C^t$ as the aggregated model at each global round $t\in \left[T\right]$ where $\left[T\right]=\{0,...,T-1\}$ and ${\overline{\mathbf{x}}}_C^t={\displaystyle \frac{1}{m}}{\sum }_i{\mathbf{x}}_C^t$. Throughout this paper, $\left[S\right]=\{0,...,m-1\}$ identifies the clients’ set which is indexed by i. We employ two strategies for client participation. The first strategy entails all clients participating in the learning process. The second strategy involves the server randomly sampling a subset of size n of clients with replacement, $\left[S_t\right]$, following a uniform distribution. We assume that $D_i$s are non-i.i.d. The derivative of local loss function of client i in SFL setting with respect to ${\mathbf{x}}_C$ and ${\mathbf{x}}_A$ are indicated by $\nabla F_i^c\left({\mathbf{x}}_C\right)$ and $\nabla F_i^c\left({\mathbf{x}}_A\right)$ respectively. As for the server-side model, the derivative of the loss function is $\nabla F^s\left({\mathbf{x}}_S\right)$ which is with respect to ${\mathbf{x}}_S$. The stochastic gradients of each of the aforementioned gradients will be distinguished by a $\tilde{\nabla }$ sign, e.g., $\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_C\right)=\nabla F_i^c({\mathbf{x}}_C;\xi )$ where $\xi \sim D_i$ is a random sample from client i dataset. Note that ${\mu }_L,$ and $\mu$ are the learning rates of client-side and server-side models respectively. Client i trains ${\mathbf{x}}_{C,i}$ on its local dataset and renders the forward propagation results, ${\mathbf{z}}_{f,i}$, to the auxiliary network at each local step k and it receives the ${\widehat{\mathbf{z}}}_{b,i}$ in response. Note that $k\in \left[K\right]$ indexes the local steps. Additionally, the client sends the ${\mathbf{z}}_{f,i}$ to the server at each global round t such that $t\equiv 0modl$ where l is a parameter determining the frequency of this process. We have one server performing the model aggregation at each global round, completing the forward propagation of clients, and updating the server model at specific global rounds. Algorithm 1 illustrates the proposed procedure by [4] in detail.

Algorithm 1 CSE-SFL [4]

1: At Server  2:     Initialize ${\mathbf{x}}_C^0$, ${\mathbf{x}}_A^0$ and ${\mathbf{x}}_S^0$  3:     for $t=0,1,...,T-1$ do  4:         Sample a subset $S_t$ of n clients out of m clients  5:         Receive ${\mathbf{x}}_{C,i}^t,{\mathbf{x}}_{A,i}^t\forall i\in \left[S_t\right]$   6:         Let ${\overline{\mathbf{x}}}_C^t\leftarrow {\displaystyle \frac{1}{m}}{\sum }_{i\in \left[S_t\right]}{\mathbf{x}}_{C,i}^t$ and ${\overline{\mathbf{x}}}_A^t\leftarrow {\displaystyle \frac{1}{m}}{\sum }_{i\in \left[S_t\right]}{\mathbf{x}}_{A,i}^t$  7:         Broadcast ${\overline{\mathbf{x}}}_C^t$ and ${\overline{\mathbf{x}}}_A^t$ to clients  8:         if $t\equiv 0modl,\mathit{and}t\ne 0$ then  9:            for each client $i\in \left[S_t\right]$ in sequence do  10:                ${\mathbf{z}}_{f,i},{\mathbf{y}}_i\leftarrow$ Client($i,{\mathbf{z}}_f,\mathbf{y}$)  11:                Complete forward propagation with ${\mathbf{z}}_{f,i}$, and ${\mathbf{x}}_S^0$   12:                Compute ${\widehat{\mathbf{y}}}_i$, the prediction of ${\mathbf{y}}_i$  13:                Compute loss function $F_i^s({\mathbf{x}}_S^0;{\mathbf{z}}_{f,i},{\mathbf{y}}_i)$  14:                Complete backward-propagation  15:                Send ${\mathbf{z}}_{b,i}$ to the client  16:                Update server model: ${\mathbf{x}}_S^0\leftarrow {\mathbf{x}}_S^0-{\displaystyle \frac{\mu }{m}}\nabla F_i^s({\mathbf{x}}_S^0;{\mathbf{z}}_{f,i},{\mathbf{y}}_i)$  17:            end for  18:         end if  19:     end for  20:     Concatenate ${\mathbf{x}}_C$ and ${\mathbf{x}}_S$  21: At Clients :  22:     for all clients $i\in \left[S_t\right]$ in parallel at round t do  23:         ${\mathbf{x}}_{C,i}^0$, $\leftarrow \mathrm{Server}\left({\overline{\mathbf{x}}}_C^t\right)$  24:         if $t\equiv 0modl,\mathit{and}t\ne 0$ then  25:            ${\mathbf{z}}_{f,i}\leftarrow \mathrm{ForwardPass}({\mathbf{x}}_{C,i}^0;\xi )$  26:            Send ${\mathbf{z}}_{f,i}$ and ${\mathbf{y}}_i$ to the server   27:            ${\mathbf{z}}_{b,i}^t$, $\leftarrow \mathrm{Server}\left({\mathbf{z}}_b^t\right)$  28:            Complete backward-propagation with ${\mathbf{z}}_{b,i}^t$  29:            Client update: ${\mathbf{x}}_{C,i}^1\leftarrow {\mathbf{x}}_{C,i}^0-{\mu }_L\nabla F_i^c\left({\mathbf{x}}_{C,i}^0\right)$  30:            Auxiliary update: ${\mathbf{x}}_{A,i}^1\leftarrow {\mathbf{x}}_{A,i}^0$  31:            for local step $k=1,..,K-1$ do  32:                Compute forward propagation with ${\mathbf{x}}_{C,i}^k$ and ${\mathbf{x}}_A^t$   33:                Compute local loss $F_i^c({\mathbf{x}}_i^k;{\xi }^k)$  34:                Client update: ${\mathbf{x}}_{C,i}^{k+1}\leftarrow {\mathbf{x}}_{C,i}^k-{\mu }_L\nabla F_i^c\left({\mathbf{x}}_{C,i}^k\right)$  35:                Auxiliary update: ${\mathbf{x}}_{A,i}^{k+1}\leftarrow {\mathbf{x}}_{A,i}^k-{\mu }_L\nabla F_i^c\left({\mathbf{x}}_{A,i}^k\right)$  36:            end for  37:         else  38:            for local step $k=0,..,K-1$ do  39:                Compute forward propagation with ${\mathbf{x}}_{C,i}^k$ and ${\mathbf{x}}_A^t$   40:                Compute local loss $F_i^c({\mathbf{x}}_i^k;{\xi }^k)$  41:                Client update: ${\mathbf{x}}_{C,i}^{k+1}\leftarrow {\mathbf{x}}_{C,i}^k-{\mu }_L\nabla F_i^c\left({\mathbf{x}}_{C,i}^k\right)$  42:                Auxiliary update: ${\mathbf{x}}_{A,i}^{k+1}\leftarrow {\mathbf{x}}_{A,i}^k-{\mu }_L\nabla F_i^c\left({\mathbf{x}}_{A,i}^k\right)$  43:            end for  44:         end if  45:         Return ${\mathbf{x}}_{C,i}^K$ to the server   46:     end for

4. Convergence rate analysis

The following assumptions for the convergence rate evaluation have been made:

Assumption 1.(L-Lipschitz continuous gradient) Both client and server-side models are $L-$smooth non-convex functions, i.e., there is a constant $L>0$ such that $\forall {\mathbf{x}}_C,{\mathbf{y}}_C\in {\mathbb{R}}^{d_c}$, and $\forall {\mathbf{x}}_S,{\mathbf{y}}_S\in {\mathbb{R}}^{d_s}:$

$$\begin{array}{c}\hfill \parallel \nabla F^c\left({\mathbf{x}}_C\right)-\nabla F^c\left({\mathbf{y}}_C\right)\parallel \le L\parallel {\mathbf{x}}_C-{\mathbf{y}}_C\parallel \mathit{and}\parallel \nabla F^s\left({\mathbf{x}}_S\right)-\nabla F^s\left({\mathbf{y}}_S\right)\parallel \le L\parallel {\mathbf{x}}_S-{\mathbf{y}}_S\parallel \end{array}$$

Assumption 2.(Unbiased local gradient estimator) We assume that $\forall i\in \left[S\right]$,

$$\begin{array}{c}\hfill {\mathbb{E}}_{\xi \in D_i}\left[\nabla F_i^c({\mathbf{x}}_C;\xi )\right]=\nabla F_i^c\left({\mathbf{x}}_C\right)\end{array}$$

that is the local gradient estimator of the client-side model is unbiased. The expectation is over all the local datasets of the client. Note that we have a similar assumption for the server-side model as follows $\forall i\in \left[S\right]$:

$$\begin{array}{c}\hfill {\mathbb{E}}_{\xi \in D_i}\left[\nabla F_i^s({\mathbf{x}}_S;{\mathbf{z}}_{f,i}({\mathbf{x}}_C;\xi ))\right]=\nabla F_i^s\left({\mathbf{x}}_S\right)\end{array}$$

Assumption 3.(Bounded local and global variance) We have bounded variance of the stochastic gradients locally and globally for both server-side and client-side models, i.e., there exist positive constants ${\sigma }_L$ and ${\sigma }_G$ such that

$$\begin{array}{cc}\hfill & \mathbb{E}\left[\parallel {\nabla F_i^c({\mathbf{x}}_C;\xi )-\nabla F_i^c\left({\mathbf{x}}_C\right)}^2\parallel \right]\le {\sigma }_L^2\mathit{and}\mathbb{E}\left[\parallel {\nabla F_i^c\left({\mathbf{x}}_C\right)-\nabla F^c\left({\mathbf{x}}_C\right)}^2\parallel \right]\le {\sigma }_G^2\hfill \\ \hfill & \mathbb{E}\left[\parallel {\nabla F_i^s({\mathbf{x}}_S;\xi )-\nabla F_i^s\left({\mathbf{x}}_S\right)}^2\parallel \right]\le {\sigma }_L^2\mathit{and}\mathbb{E}\left[\parallel {\nabla F_i^s\left({\mathbf{x}}_S\right)-\nabla F^s\left({\mathbf{x}}_S\right)}^2\parallel \right]\le {\sigma }_G^2\hfill \end{array}$$

Assumptions 1, 2, and 3 are natural assumptions applied in non-convex optimization and FL, e.g., see [7,11,12,13,14,15]. Figure 1 gives an overview of the communication and storage of efficient federated split learning (CSE-FSL) algorithm in an illustrative way.

4.1. Client-Side Model Convergence

We examine the convergence rate when $t\equiv 0modl$ because it is during these rounds that the server-side model is also updated. This will let us study the impact of l on the convergence rate and communication overhead.

Theorem 1. 

Under Assumptions 1, 2, 3, and full participation of clients, if ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}}}$, and $t\equiv 0modl$, in Algorithm 1, the convergence rate of client model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le {\displaystyle \frac{2\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)}{(1-\Gamma {\lambda }_2){\mu }_{L}KT}}+{\Phi }_1+{\displaystyle \frac{\Gamma {\lambda }_1}{1-\Gamma {\lambda }_2}}\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \mathrm{Where},\hfill \\ \hfill & {\Phi }_1={\displaystyle \frac{5K{\mu }_L^2{L}^2({\sigma }_L^2+6K{\sigma }_G^2))}{1-\Gamma {\lambda }_2}}+\hfill \\ \hfill & {\displaystyle \frac{4L{\mu }_L+4{\mu }_L^{2}K{L}^2(l-1)}{1-\Gamma {\lambda }_2}}\left(\left(Kl+5L^2{K}^{2}l{\mu }_L^2\right){\sigma }_L^2+\left(Kl+30L^2{K}^{3}l{\mu }_L^2\right){\sigma }_G^2\right)\hfill \\ \hfill & {\lambda }_1=B\sum _{j=0}^{l-1}{\displaystyle \frac{A^j-1}{A-1}},{\lambda }_2={\displaystyle \frac{A^l-1}{A-1}},\hfill \\ \hfill & B=8L^2{\mu }_L^2\left(\left(K^2+5L^2{K}^3{\mu }_L^2\right){\sigma }_L^2+\left(K^2+30L^2{K}^4{\mu }_L^2\right){\sigma }_G^2\right),\hfill \\ \hfill & A=8L^2{\mu }_L^2\left(K^2+30L^2{K}^3{\mu }_L^2\right)+2,\hfill \\ \hfill & \Gamma =4\left(L{\mu }_L+{\mu }_L^{2}K{L}^2(l-1)\right)\left(K+30L^2{K}^2{\mu }_L^2\right)+30K^2{\mu }_L^2{L}^2),\mathit{and}\hfill \\ \hfill & {\overline{\mathbf{x}}}_C^{*}=\underset{{\overline{\mathbf{x}}}_C^t,t\in \left[T\right]}{argmin}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\hfill \end{array}$$

Corollary 1. 

Let ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}\sqrt{T}}}$. Then, the convergence rate of the client-side model in Algorithm 1 is

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla f^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}+{\displaystyle \frac{1}{T\sqrt{T}}}\right).\end{array}$$

(4)

Theorem 2. 

Under Assumptions 1, 2, 3, and partial participation of clients due to strategy one, if ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}}}$, and $t\equiv 0modl$, in Algorithm 1, the convergence rate of client model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \le {\displaystyle \frac{2\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)}{{\mu }_{L}KT}}+\hfill \\ \hfill & \left(5K{\mu }_L^2{L}^2+4{\mu }_L^2{L}^2\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right)+L{\mu }_L\left({\displaystyle \frac{1}{n}}+15K^2{L}^2{\mu }_L^2\right)\right){\sigma }_L^2+\hfill \\ \hfill & \left(30K^2{\mu }_L^2{L}^2+L{\mu }_L\left(90K^3{L}^2{\mu }_L^2+3K\right)+4{\mu }_L^2{L}^2\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right)\right){\sigma }_G^2+\hfill \\ \hfill & {\displaystyle \frac{{\Gamma }^{\prime }{\lambda }_1}{1-{\Gamma }^{\prime }{\lambda }_2}}\hfill \end{array}$$

Where

$$\begin{array}{cc}\hfill & {\Gamma }^{\prime }=4{\mu }_L^2{L}^2\left(K^{2}l+30L^2{K}^{3}l{\mu }_L^2\right)+{\displaystyle \frac{L{\mu }_L}{l}}\left(90l{K}^3{L}^2{\mu }_L^2+3K\right)+30K^2{\mu }_L^2{L}^2\hfill \\ \hfill & \mathit{and},\hfill \\ \hfill & {\overline{\mathbf{x}}}_C^{*}=\underset{{\overline{\mathbf{x}}}_C^t,t\in \left[T\right]}{argmin}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\hfill \end{array}$$

Corollary 2. 

Let ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}\sqrt{T}}}$. Then, the convergence rate of the client-side model in Algorithm 1 is

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla f^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}+{\displaystyle \frac{1}{T\sqrt{T}}}\right).\end{array}$$

(5)

4.2. Server-Side Model Convergence

Theorem 3. 

Under Assumptions 1, 2, 3, and full participation of clients, if $\mu \le {\displaystyle \frac{1}{2L}}$, and $t\equiv 0modl$, the convergence rate of the server model of Algorithm 1 satisfies:

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{2l\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (2m-3)T}}+{\displaystyle \frac{L\mu m^2}{2m-3}}\left(9.2{\sigma }_L^2+13.2{\sigma }_G^2\right)\end{array}$$

Where${\mathbf{x}}_S^{*}={argmin}_{{\mathbf{x}}_S^t,t\in \left[T\right]}\mathbb{E}\parallel {\nabla F^s({\mathbf{x}}_S^t,{\mathbf{z}}_f^t)}^2\parallel$.

Corollary 3. 

Let $\mu \le {\displaystyle \frac{1}{2L\sqrt{T}}}$, then the convergence rate of the server-side model is:

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}\right)\end{array}$$

Theorem 4. 

Under Assumptions 1, 2, 3, and partial participation of clients due to strategy one, if $\mu \le {\displaystyle \frac{1}{8L^2{m}^2}}$, and $t\equiv 0modl$, the convergence rate of the server model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{l\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (m-2)T}}+{\displaystyle \frac{L\mu m^2}{m-2}}\left(7{\sigma }_L^2+7{\sigma }_G^2\right)\hfill \end{array}$$

Where ${\mathbf{x}}_S^{*}={argmin}_{{\mathbf{x}}_S^t}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2$.

$$\begin{array}{c}\hfill \parallel \end{array}$$

Corollary 4. 

Let $\mu \le {\displaystyle \frac{1}{L^2{m}^2\sqrt{T}}}$, then the convergence rate of the server-side model is:

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}\right)\end{array}$$

5. Discussion and Conclusions

In this paper, we proposed theoretical convergence proofs for the state-of-the-art SplitFed Learning algorithm, CSE-FSL, which is designed to improve the convergence rates of both client-side and server-side models leveraging parallelism power of Federated Learning (FL) and reduce the storage at the server by keeping one copy at a time policy. Our approach leverages several key assumptions that are conventional in FL to underpin the theoretical foundations for CSE-FSL convergence. We prove the convergence for the cases where we have non-i.i.d. datasets, and non-convex loss functions given full and partial client participation scenarios.

5.1. Summary of Contributions

• Convergence Analysis: We clearly formulated the CSE-FSL algorithm developed by [4]. We conducted a comprehensive convergence rate analysis under both full and partial client participation scenarios given the non-i.i.d. dataset and non-convex loss function. The convergence guarantees are derived under several assumptions, including L-smoothness of the objective functions, unbiased gradient estimators, and bounded gradient variances which are natural in conventional FL convergence analysis.
• Key Results:

  −

  Client-Side Model: We demonstrated that, under full client participation, the client-side model converges with a rate of $\mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}+{\displaystyle \frac{1}{T\sqrt{T}}}\right)$. This result highlights the effectiveness of the algorithm in achieving linear convergence rates while accommodating the federated setting’s constraints and sequential update of the server model. An increase in l, causes a longer convergence time which is obvious as it means the server model will be updated after more global rounds.

  −

  Server-Side Model: For the server-side model, we established convergence rates of $\mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}\right)$ under both full and partial client participation scenarios. This result underscores the robustness of the algorithm in ensuring effective learning even when clients participate partially. This also demonstrates that the number of clients and their local steps are not effective in speeding up the convergence in contrast to FL settings.

5.2. Implications

Our findings underscore the importance of efficient communication and gradient estimation (auxiliary networks) techniques in SplitFed Learning (SFL). The derived convergence rates demonstrate that the CSE-FSL algorithm achieves a balance between computational efficiency and convergence performance, making it a viable solution for practical federated learning applications where the privacy of clients is of high importance.

The theoretical guarantees provided by our convergence analysis offer valuable insights into how the algorithm performs under various conditions, thus guiding practitioners in optimizing federated learning systems. Future work could extend these results to explore more complex scenarios and refine the algorithm further for enhanced performance in real-world applications. For example, considering stragglers, elimination of label sharing by clients, and determining the optimal cut layer seem to be promising avenues for further research.

In summary, the CSE-FSL algorithm represents a significant advancement in FL, providing a robust framework for effective model training leveraging the parallelism power of FL, auxiliary networks, and sequential updates of the server-side model which helps reduce storage on the server side. It recovers the linear convergence speed of FL while providing more privacy by only forward-propagation and label transition between clients and servers instead of trained parameters.

Appendix A.1. Client-Side Model Convergence

We examine the convergence rate when $t\equiv 0modl$ because it is during these rounds that the server-side model is also updated. This analysis allows us to investigate the influence of l on both the convergence rate and communication overhead.

Theorem A1. 

Under Assumptions 1, 2, 3, and full participation of clients, if ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}}}$, and $t\equiv 0modl$, in Algorithm 1, the convergence rate of client model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le {\displaystyle \frac{2\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)}{(1-\Gamma {\lambda }_2){\mu }_{L}KT}}+{\Phi }_1+{\displaystyle \frac{\Gamma {\lambda }_1}{1-\Gamma {\lambda }_2}}\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \mathrm{Where},\hfill \\ \hfill & {\Phi }_1={\displaystyle \frac{5K{\mu }_L^2{L}^2({\sigma }_L^2+6K{\sigma }_G^2))}{1-\Gamma {\lambda }_2}}+\hfill \\ \hfill & {\displaystyle \frac{4L{\mu }_L+4{\mu }_L^{2}K{L}^2(l-1)}{1-\Gamma {\lambda }_2}}\left(\left(Kl+5L^2{K}^{2}l{\mu }_L^2\right){\sigma }_L^2+\left(Kl+30L^2{K}^{3}l{\mu }_L^2\right){\sigma }_G^2\right)\hfill \\ \hfill & {\lambda }_1=B\sum _{j=0}^{l-1}{\displaystyle \frac{A^j-1}{A-1}},{\lambda }_2={\displaystyle \frac{A^l-1}{A-1}},\hfill \\ \hfill & B=8L^2{\mu }_L^2\left(\left(K^2+5L^2{K}^3{\mu }_L^2\right){\sigma }_L^2+\left(K^2+30L^2{K}^4{\mu }_L^2\right){\sigma }_G^2\right),\hfill \\ \hfill & A=8L^2{\mu }_L^2\left(K^2+30L^2{K}^3{\mu }_L^2\right)+2,\hfill \\ \hfill & \Gamma =4\left(L{\mu }_L+{\mu }_L^{2}K{L}^2(l-1)\right)\left(K+30L^2{K}^2{\mu }_L^2\right)+30K^2{\mu }_L^2{L}^2),\mathrm{and}\hfill \\ \hfill & {\overline{\mathbf{x}}}_C^{*}=\underset{{\overline{\mathbf{x}}}_C^t,t\in \left[T\right]}{argmin}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\hfill \end{array}$$

Corollary A1. 

Let ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}\sqrt{T}}}$. Then, the convergence rate of the client-side model in Algorithm 1 is

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla f^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}+{\displaystyle \frac{1}{T\sqrt{T}}}\right).\end{array}$$

(A1)

Proof. 

In this proof, all the gradients are w.r.t. ${\mathbf{x}}_C$. Due to Assumption 1, for any ${\overline{\mathbf{x}}}_C^{t+l}$ and ${\overline{\mathbf{x}}}_C^t$ such that $t\in \left[T\right]$, we can write:

$$\begin{array}{c}\hfill F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\le F^c\left({\overline{\mathbf{x}}}_C^t\right)+\nabla F^c{\left({\overline{\mathbf{x}}}_C^t\right)}^{\top }({\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t)+{\displaystyle \frac{L}{2}}\parallel {{\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t}^2\parallel \end{array}$$

(A2)

Particularly, we consider the case when $t\equiv 0modl$ from now on.

Also, note the global aggregation and client update rule in the Algorithm 1,

$$\begin{array}{c}{\overline{\mathbf{x}}}_C^{t+l}={\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{\mathbf{x}}_{C,i}^{t+l}={\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\left({\mathbf{x}}_{C,i}^t-{\mu }_L\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)\right)=\hfill \\ {\overline{\mathbf{x}}}_C^t-{\displaystyle \frac{{\mu }_L}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)\hfill \end{array}$$

(A3)

Thus,

$${\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t=-{\displaystyle \frac{{\mu }_L}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)$$

(A4)

Taking expectation of $F^c\left({\mathbf{x}}_C^{t+1}\right)$ with respect to randomness at round $t+l-1$, i.e., ${\xi }^{[t+l-1]}\stackrel{\Delta }{=}{\left[{\xi }_i^{\tau }\right]}_{i\in \left[N\right],\tau \in [t+l-1]}$, and plugging A3 into A2 note that:

$$\begin{array}{c}\mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]\le F^c\left({\overline{\mathbf{x}}}_C^t\right)+〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t\right]〉+{\displaystyle \frac{L}{2}}\mathbb{E}\parallel {{\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t}^2\parallel \hfill \\ \mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]\le F^c\left({\overline{\mathbf{x}}}_C^t\right)+{\mu }_L\underset{A_1}{\underbrace{〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\displaystyle \frac{-1}{m}}{\sum }_{i=0}^{m-1}{\sum }_{j=0}^{l-1}{\sum }_{k=0}^{K-1}\left(\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)\right]〉}}\hfill \\ \underset{A_2}{\underbrace{-{\mu }_L〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\displaystyle \frac{K}{m}}{\sum }_{i=0}^{m-1}{\sum }_{j=0}^{l-1}\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right]〉}}+{\displaystyle \frac{L{\mu }_L^2}{2}}\underset{A_3}{\underbrace{\mathbb{E}\left[\parallel {{\displaystyle \frac{1}{m}}{\sum }_{i=0}^{m-1}{\sum }_{j=0}^{l-1}{\sum }_{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel \right]}}\hfill \end{array}$$

(A5)

We bound the term $A_1$ as follows:

$$\begin{array}{cc}\hfill & A_1=〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\displaystyle \frac{-1}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\left(\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)\right]〉\hfill \\ \hfill & =〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[\underset{y_1}{\underbrace{{\displaystyle \frac{-1}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\left(\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)}}\right]〉\hfill \\ \hfill & \stackrel{\left(a_1\right)}{=}{\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{1}{2K{m}^2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\left(\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)}^2\parallel \hfill \\ \hfill & -{\displaystyle \frac{1}{2K{m}^2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\left(\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)+K\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_2\right)}{\le }{\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2+{\displaystyle \frac{1}{2K{m}^2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\left(\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_3\right)}{\le }{\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{l{L}^2}{2m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\mathbb{E}\left[\parallel {\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \stackrel{\left(a_4\right)}{\le }{\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{l{L}^2}{2m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\underset{y_2}{\underbrace{\mathbb{E}\left[\parallel {{\mathbf{x}}_{C,i}^{t+j,k}-{\overline{\mathbf{x}}}_C^{t+j}}^2\parallel \right]}}\hfill \end{array}$$

(A6)

$$\begin{array}{cc}\hfill & \mathrm{We}\mathrm{have}〈a,b〉={\displaystyle \frac{1}{2}}\left(\parallel a^2\parallel +\parallel b^2\parallel -\parallel {a-b}^2\parallel \right)\mathrm{for}\mathrm{any}\mathrm{two}\mathrm{vectors}a\mathit{and}b.\hfill \end{array}$$

(A7)

$$\begin{array}{cc}\hfill & \mathrm{Thus},\mathrm{if}\mathrm{we}\mathrm{put}a=\sqrt{K}\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathit{and}b={\displaystyle \frac{1}{\sqrt{K}}}{y}_1,\mathrm{it}\mathrm{yields}\mathrm{equality}\left(a_1\right).\mathrm{Inequality}\left(a_2\right)\hfill \\ \hfill & \mathrm{follows}\mathrm{from}\mathrm{eliminating}\mathrm{a}\mathrm{strictly}\mathrm{negative}\mathrm{term}.\mathrm{Now},\mathrm{due}\mathrm{to}\mathbb{E}\parallel {\sum _i^n{z}_i}^2\parallel \le n\sum _i\mathbb{E}\left[\parallel {z_i}^2\parallel \right]\hfill \\ \hfill & \mathrm{for}\mathrm{any}\mathrm{random}\mathrm{variables}{z}_i,\mathrm{inequality}\left(a_3\right)\mathrm{holds}.\hfill \end{array}$$

(A8)

$$\begin{array}{cc}\hfill & \mathrm{The}\mathrm{inequality}{a}_4\mathrm{follows}\mathrm{from}\mathrm{Assumption}1.\mathrm{There}\mathrm{is}\mathrm{an}\mathrm{upper}\mathrm{bound}\mathrm{for}\mathrm{the}\mathrm{term}{y}_2\hfill \\ \hfill & \mathrm{provided}\mathrm{by}\left[13\right].\mathrm{To} \mathrm{preserve} \mathrm{the} \mathrm{integrity} \mathrm{of} \mathrm{the} \mathrm{work}, \mathrm{we} \mathrm{include} \mathrm{it} \mathrm{here} \mathrm{as} \mathrm{well}.\hfill \\ \hfill & \mathbb{E}\left[\parallel {{\mathbf{x}}_{C,i}^{t+j,k}-{\overline{\mathbf{x}}}_C^{t+j}}^2\parallel \right]=\mathbb{E}\left[\parallel {{\mathbf{x}}_{C,i}^{t+j,k-1}-{\overline{\mathbf{x}}}_C^{t+j}-{\mu }_L\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)}^2\parallel \right]\hfill \\ \hfill & =\mathbb{E}\parallel {\mathbf{x}}_{C,i}^{t+j,k-1}-{\overline{\mathbf{x}}}_C^{t+j}-{\mu }_L(\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)-\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)+\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)\hfill \\ \hfill & -\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)+\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)+\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)){\parallel }^2\hfill \\ \hfill & \stackrel{\left(a_5\right)}{\le }(1+{\displaystyle \frac{1}{2K-1}})\mathbb{E}\parallel {{\mathbf{x}}_{C,i}^{t+j,k-1}-{\overline{\mathbf{x}}}_C^{t+j}}^2\parallel +\mathbb{E}\parallel {{\mu }_L\left(\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)-\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)\right)}^2\parallel +\hfill \\ \hfill & 6K\mathbb{E}\parallel {{\mu }_L\left(\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k-1}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)}^2\parallel +6K\mathbb{E}\parallel {{\mu }_L\left(\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)}^2\parallel \hfill \\ \hfill & +6K\mathbb{E}\parallel {{\mu }_L\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_6\right)}{\le }(1+{\displaystyle \frac{1}{K-1}})\mathbb{E}\parallel {{\mathbf{x}}_{C,i}^{t+j,k-1}-{\overline{\mathbf{x}}}_C^{t+j}}^2\parallel +{\mu }_L^2{\sigma }_L^2+6K{\mu }_L^2{\sigma }_G^2+6K{\mu }_L^2\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \\ \hfill \end{array}$$

(A9)

The inequality $\left(a_5\right)$ follows from the fact that $\mathbb{E}\parallel {{\sum }_i{z}_i}^2\parallel \le \mathbb{E}\left[{\sum }_i\parallel {z_i}^2\parallel \right]$ holds true for independent random variables $z_i$ with zero mean.

$$$$

(A10)

The term $\left(a_6\right)$ is due to Assumption 3. Finally, by unrolling recursion and some simplification, we have:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{C,i}^{t+j,k}-{\overline{\mathbf{x}}}_C^{t+j}}^2\parallel \le 5K{\mu }_L^2\left({\sigma }_L^2+6K{\sigma }_G^2\right)+30K^2{\mu }_L^2\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \\ \hfill & \mathrm{Now},\mathrm{we}\mathrm{continue}\mathrm{by}\mathrm{substituting}\mathrm{A}11\mathrm{into}\mathrm{A}6,\hfill \end{array}$$

(A11)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{5K^2{\mu }_L^{2}l{L}^2}{2}}({\sigma }_L^2+6K{\sigma }_G^2)+15K^3{\mu }_L^{2}l{L}^2\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \end{array}$$

(A12)

The above inequality, A12, is an upper bound for the term $A_1$. We continue with bounding $A_2$ as follows.

$$\begin{array}{cc}\hfill & A_2=-{\mu }_L〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\displaystyle \frac{K}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right]〉\hfill \\ \hfill & \stackrel{\left(a_7\right)}{=}-{\mu }_{L}K〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[\sum _{j=0}^{l-1}\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right]〉\hfill \\ \hfill & =-{\mu }_{L}K\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel -{\mu }_{L}K〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[\sum _{j=1}^{l-1}\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right]〉\hfill \\ \hfill & \stackrel{\left(a_8\right)}{\le }-{\mu }_{L}K\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel -{\mu }_{L}K〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[(l-1)\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j^{*}}\right)\right]〉\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \stackrel{\left(a_9\right)}{=}-{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \underset{y_3}{\underbrace{-{\displaystyle \frac{{\mu }_{L}K(l-1)}{2}}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j^{*}}\right)}^2}}\parallel \hfill \\ \hfill & +{\displaystyle \frac{{\mu }_{L}K(l-1)}{2}}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j^{*}}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_{10}\right)}{\le }-{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{{\mu }_{L}K(l-1)}{2}}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j^{*}}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_{11}\right)}{\le }-{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{{\mu }_{L}K{L}^2(l-1)}{2}}\mathbb{E}\parallel {{\overline{\mathbf{x}}}_C^{t+j^{*}}-{\overline{\mathbf{x}}}_C^t}^2\parallel \hfill \end{array}$$

$$\begin{array}{cc}\hfill & \stackrel{\left(a_{12}\right)}{\le }-{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{{\mu }_L^{3}K{L}^2(l-1)}{2}}\underset{y_4}{\underbrace{\mathbb{E}\parallel {{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{j^{*}}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel }}\hfill \end{array}$$

(A13)

$$\begin{array}{cc}\hfill & \stackrel{\left(a_{13}\right)}{\le }-{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{{\mu }_L^{3}K{L}^2(l-1)}{2}}\underset{A_3}{\underbrace{\mathbb{E}\parallel {{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2}}\parallel \hfill \end{array}$$

(A14)

The equality $\left(a_7\right)$ follows from definition of the global aggregation in Algorithm 1. In inequality $\left(a_8\right)$, we assume there exists a $j^{*}$ such that $j^{*}={argmin}_{1\le j\le l-1}〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right]〉$ . The equality $\left(a_9\right)$ follows from A7, where $a=\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)$ and $b=\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j^{*}}\right)$. The inequality $\left(a_{10}\right)$ is due to the fact that the term $y_3$ is negative. Thus, it can be eliminated safely. Due to Assumption 1, we have inequality $\left(a_{11}\right)$. The inequality $\left(a_{12}\right)$ is due to equation A3. For the term $y_4$ in A13, there is an upper bound when $j^{*}=l-1$ due to A15. Hence, with $j^{*}=l-1$, inequality $\left(a_{13}\right)$ is achieved. We proceed with bounding $A_3$ as follows.

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel {{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel \hfill \\ \hfill & ={\displaystyle \frac{1}{m^2}}\mathbb{E}\parallel \sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}(\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)+\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\hfill \\ \hfill & +\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)+\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)){\parallel }^2\hfill \\ \hfill & \stackrel{\left(a_{14}\right)}{\le }{\displaystyle \frac{4Kl}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}(\mathbb{E}\parallel {\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel +\mathbb{E}\parallel {\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel +\hfill \\ \hfill & \mathbb{E}\parallel {\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel +\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel )\hfill \\ \hfill & \stackrel{\left(a_{15}\right)}{\le }{\displaystyle \frac{4Kl}{m}}\sum _{i=0}^{m-1}\sum _{j=0}^{l-1}\sum _{k=0}^{K-1}\left({\sigma }_L^2+L^2\mathbb{E}\parallel {{\mathbf{x}}_{C,i}^{t+j,k}-{\overline{\mathbf{x}}}_C^{t+j}}^2\parallel +{\sigma }_G^2+\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \right)\hfill \\ \hfill & \stackrel{\left(a_{16}\right)}{\le }4\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right){\sigma }_L^2+4\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right){\sigma }_G^2+4(K^{2}l+\hfill \\ \hfill & 30L^2{K}^{3}l{\mu }_L^2)\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \end{array}$$

(A15)

The inequality $\left(a_{14}\right)$ holds due to A8, and inequality $\left(a_{15}\right)$ follows from Assumptions 2 and A3. Due to the bound on client drift, A11, note the inequality $\left(a_{16}\right)$. Substituting A12, A14, and A15 into A5, observe that:

$$\begin{array}{c}\mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]\le F^c\left({\overline{\mathbf{x}}}_C^t\right)\hfill \\ +{\mu }_L\left({\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{5K^2{\mu }_L^{2}l{L}^2}{2}}({\sigma }_L^2+6K{\sigma }_G^2)+15K^3{\mu }_L^{2}l{L}^2\sum _{j=0}^{l-1}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \right)\hfill \\ -{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +2\left(L{\mu }_L^2+{\mu }_L^{3}K{L}^2(l-1)\right)(\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right){\sigma }_L^2+\hfill \\ \left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right){\sigma }_G^2+\left(K^{2}l+30L^2{K}^{3}l{\mu }_L^2\right)\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel )\hfill \\ \mathrm{Rearranging}\mathrm{and}\mathrm{simplifying}\mathrm{the}\mathrm{terms},\hfill \\ \mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]\le F^c\left({\overline{\mathbf{x}}}_C^t\right)-{\displaystyle \frac{{\mu }_{L}Kl}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\mu }_L\left({\displaystyle \frac{5K^2{\mu }_L^{2}l{L}^2}{2}}({\sigma }_L^2+6K{\sigma }_G^2)\right)+\hfill \\ 2\left(L{\mu }_L^2+{\mu }_L^{3}K{L}^2(l-1)\right)\left(\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right){\sigma }_L^2+\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right){\sigma }_G^2\right)+\hfill \\ \left(2\left(L{\mu }_L^2+{\mu }_L^{3}K{L}^2(l-1)\right)\left(K^{2}l+30L^2{K}^{3}l{\mu }_L^2\right)+15K^3{\mu }_L^{3}l{L}^2\right)\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \\ \mathrm{By}\mathrm{iterating}\mathrm{over}t,\mathrm{note}\mathrm{that},\hfill \\ \sum _{t\in \left[T\right]}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le {\displaystyle \frac{2}{{\mu }_{L}Kl}}\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)+{\displaystyle \frac{T}{l}}\left(5K{\mu }_L^2{L}^2({\sigma }_L^2+6K{\sigma }_G^2)\right)+\hfill \\ {\displaystyle \frac{4T}{l}}\left(L{\mu }_L+{\mu }_L^{2}K{L}^2(l-1)\right)\left(\left(Kl+5L^2{K}^{2}l{\mu }_L^2\right){\sigma }_L^2+\left(Kl+30L^2{K}^{3}l{\mu }_L^2\right){\sigma }_G^2\right)+\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \underset{\Gamma }{\underbrace{\left(4\left(L{\mu }_L+{\mu }_L^{2}K{L}^2(l-1)\right)\left(K+30L^2{K}^2{\mu }_L^2\right)+30K^2{\mu }_L^2{L}^2\right)}}\underset{y_5}{\underbrace{\sum _t\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2}}\parallel \hfill \\ \hfill & \mathrm{We}\mathrm{bound}\mathrm{the}\mathrm{term}{y}_5\mathrm{as}\mathrm{follows}.\mathrm{We}\mathrm{start}\mathrm{with}\mathrm{bounding}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \mathrm{for}\mathrm{a}\mathrm{particular}t\mathit{and}j:\hfill \end{array}$$

(A16)

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel =\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel -\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel +\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_{17}\right)}{\le }2\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)-\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel +2\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_{18}\right)}{\le }2L^2\mathbb{E}\parallel {{\overline{\mathbf{x}}}_C^{t+j}-{\overline{\mathbf{x}}}_C^{t+j-1}}^2\parallel +2\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(a_{19}\right)}{\le }\underset{B}{\underbrace{8L^2{\mu }_L^2\left(\left(K^2+5L^2{K}^3{\mu }_L^2\right){\sigma }_L^2+\left(K^2+30L^2{K}^4{\mu }_L^2\right){\sigma }_G^2\right)}}+\hfill \\ \hfill & \underset{A}{\underbrace{\left(8L^2{\mu }_L^2\left(K^2+30L^2{K}^3{\mu }_L^2\right)+2\right)}}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel \hfill \end{array}$$

(A17)

$$$$

(A18)

The inequality $\left(a_{17}\right)$ is written as a consequence of having $\mathbb{E}\parallel a^2\parallel -\mathbb{E}\parallel b^2\parallel \le 2\mathbb{E}\parallel {a-b}^2\parallel +\mathbb{E}\parallel b^2\parallel$ for any random variables a and b where $a=\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)$ and $b=\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)$ in the inequality. The term $\left(a_{18}\right)$ is written based on Assumption 1. Due to A3, A15, and that $l=1$ in this case, inequality $\left(a_{19}\right)$ was yielded. Thus:

$$\begin{array}{c}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \le B+A\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j-1}\right)}^2\parallel \hfill \\ \mathrm{Unrolling}\mathrm{recursion}\mathrm{on}j,\mathrm{we}\mathrm{achieve}\mathrm{the}\mathrm{following}:\hfill \\ \mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \le B\left({\displaystyle \frac{A^j-1}{A-1}}\right)+A^j\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \hfill \\ \mathrm{Iterating}\mathrm{over}j,\mathrm{we}\mathrm{have}:\hfill \\ \sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \le \underset{{\lambda }_1}{\underbrace{B{\sum }_{j=0}^{l-1}\left({\displaystyle \frac{A^j-1}{A-1}}\right)}}+\underset{{\lambda }_2}{\underbrace{\left({\displaystyle \frac{A^l-1}{A-1}}\right)}}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \hfill \end{array}$$

(A19)

$$\begin{array}{cc}\hfill & \mathrm{Substituting}\mathrm{A}19\mathrm{into}\mathrm{A}16:\hfill \\ \hfill & \sum _{t\in \left[T\right]}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le {\displaystyle \frac{2}{{\mu }_{L}Kl}}\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)+{\displaystyle \frac{T}{l}}\left(5K{\mu }_L^2{L}^2({\sigma }_L^2+6K{\sigma }_G^2)\right)+\hfill \\ \hfill & {\displaystyle \frac{4T}{l}}\left(L{\mu }_L+{\mu }_L^{2}K{L}^2(l-1)\right)\left(\left(Kl+5L^2{K}^{2}l{\mu }_L^2\right){\sigma }_L^2+\left(Kl+30L^2{K}^{3}l{\mu }_L^2\right){\sigma }_G^2\right)+\hfill \\ \hfill & {\displaystyle \frac{T}{l}}\Gamma {\lambda }_1+\Gamma {\lambda }_2\sum _t\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\hfill \end{array}$$

(A20)

$$\begin{array}{cc}\hfill & \mathrm{Choosing}\mathrm{a}\mathrm{proper}{\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}}},\mathrm{we}\mathrm{have}\Gamma {\lambda }_2<1.\mathrm{Thus}:\hfill \\ \hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le {\displaystyle \frac{2\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)}{(1-\Gamma {\lambda }_2){\mu }_{L}KT}}+{\displaystyle \frac{5K{\mu }_L^2{L}^2({\sigma }_L^2+6K{\sigma }_G^2))}{1-\Gamma {\lambda }_2}}+\hfill \\ \hfill & {\displaystyle \frac{4L{\mu }_L+4{\mu }_L^{2}K{L}^2(l-1)}{1-\Gamma {\lambda }_2}}\left(\left(Kl+5L^2{K}^{2}l{\mu }_L^2\right){\sigma }_L^2+\left(Kl+30L^2{K}^{3}l{\mu }_L^2\right){\sigma }_G^2\right)+{\displaystyle \frac{\Gamma {\lambda }_1}{1-\Gamma {\lambda }_2}}\hfill \end{array}$$

(A21)

□

Theorem A2. 

Under Assumptions 1, 2, 3, and partial participation of clients due to strategy one, if ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}}}$, and $t\equiv 0modl$ in Algorithm 1, the convergence rate of client model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \le {\displaystyle \frac{2\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)}{{\mu }_{L}KT}}+\hfill \\ \hfill & \left(5K{\mu }_L^2{L}^2+4{\mu }_L^2{L}^2\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right)+L{\mu }_L\left({\displaystyle \frac{1}{n}}+15K^2{L}^2{\mu }_L^2\right)\right){\sigma }_L^2+\hfill \\ \hfill & \left(30K^2{\mu }_L^2{L}^2+L{\mu }_L\left(90K^3{L}^2{\mu }_L^2+3K\right)+4{\mu }_L^2{L}^2\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right)\right){\sigma }_G^2+\hfill \\ \hfill & {\displaystyle \frac{{\Gamma }^{\prime }{\lambda }_1}{1-{\Gamma }^{\prime }{\lambda }_2}}\hfill \end{array}$$

Where

$$\begin{array}{cc}\hfill & {\Gamma }^{\prime }=4{\mu }_L^2{L}^2\left(K^{2}l+30L^2{K}^{3}l{\mu }_L^2\right)+{\displaystyle \frac{L{\mu }_L}{l}}\left(90l{K}^3{L}^2{\mu }_L^2+3K\right)+30K^2{\mu }_L^2{L}^2\hfill \\ \hfill & \mathit{and},\hfill \\ \hfill & {\overline{\mathbf{x}}}_C^{*}=\underset{{\overline{\mathbf{x}}}_C^t,t\in \left[T\right]}{argmin}\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\hfill \end{array}$$

Corollary A2. 

Let ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}\sqrt{T}}}$. Then, the convergence rate of the client-side model in Algorithm 1 is

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\left[\parallel {\nabla f^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}+{\displaystyle \frac{1}{T\sqrt{T}}}\right).\end{array}$$

(A22)

Proof. 

In this proof, all the gradients are w.r.t. ${\mathbf{x}}_C$. Due to Assumption 1, for any ${\overline{\mathbf{x}}}_C^{t+l}$ and ${\overline{\mathbf{x}}}_C^t$ such that $t\in \left[T\right]$, we can write:

$$\begin{array}{c}\hfill F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\le F^c\left({\overline{\mathbf{x}}}_C^t\right)+\nabla F^c{\left({\overline{\mathbf{x}}}_C^t\right)}^{\top }({\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t)+{\displaystyle \frac{L}{2}}\parallel {{\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t}^2\parallel \end{array}$$

(A23)

Particularly, we consider the case when $t\equiv 0modl$ from now on.

Also, note the global aggregation and client update rule in the Algorithm 1 with partial worker participation,

$$\begin{array}{cc}\hfill & {\overline{\mathbf{x}}}_C^{t+l}={\displaystyle \frac{1}{n}}\sum _{i\in \left[S_{t+l}\right]}{\mathbf{x}}_{C,i}^{t+l}={\displaystyle \frac{1}{n}}\sum _{i\in \left[S_t\right]}{\mathbf{x}}_{C,i}^t-{\displaystyle \frac{{\mu }_L}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right))=\hfill \\ \hfill & {\overline{\mathbf{x}}}_C^t-{\displaystyle \frac{{\mu }_L}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)\hfill \\ \hfill \end{array}$$

(A24)

Thus,

$${\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t=-{\displaystyle \frac{{\mu }_L}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)$$

(A25)

In the case of partial worker participation, there are two sources of randomness. One stems from the stochastic gradient computation, while the other arises from randomly sampling the clients at round t.

Taking expectation of $F^c\left({\mathbf{x}}_C^{t+1}\right)$ w.r.t. both types of randomness at round $t+l-1$, and plugging A24 into A23 note that:

$$\begin{array}{c}\mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]\le F^c\left({\overline{\mathbf{x}}}_C^t\right)+〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t\right]〉+{\displaystyle \frac{L}{2}}\mathbb{E}\parallel {{\overline{\mathbf{x}}}_C^{t+l}-{\overline{\mathbf{x}}}_C^t}^2\parallel \hfill \\ \mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]\le F^c\left({\overline{\mathbf{x}}}_C^t\right)+{\mu }_L\underset{A_1^{\prime }}{\underbrace{〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\displaystyle \frac{-1}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\sum _{k=0}^{K-1}\left(\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right)\right]〉}}\hfill \\ \underset{A_2^{\prime }}{\underbrace{-{\mu }_L〈\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right),\mathbb{E}\left[{\displaystyle \frac{K}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\nabla F_i^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)\right]〉}}+{\displaystyle \frac{L{\mu }_L^2}{2}}\underset{A_3^{\prime }}{\underbrace{\mathbb{E}\left[\parallel {{\displaystyle \frac{1}{n}}\sum _{j=0}^{l-1}\sum _{i\in [S_{t+j}\parallel ]}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\right]}}\hfill \end{array}$$

(A26)

According to [15, Lemma 1], terms $A_1^{\prime }$ and $A_2^{\prime }$ will possess the same bounds as those of $A_1$ and $A_2$. Thus:

$$\begin{array}{cc}\hfill & A_1^{\prime }\le {\displaystyle \frac{K}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{5K^2{\mu }_L^{2}l{L}^2}{2}}({\sigma }_L^2+6K{\sigma }_G^2)+15K^3{\mu }_L^{2}l{L}^2\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \\ \hfill & A_2^{\prime }\le -{\displaystyle \frac{{\mu }_{L}K(l+1)}{2}}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel +{\displaystyle \frac{{\mu }_L^{3}K{L}^2(l-1)}{2}}(4\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right){\sigma }_L^2\hfill \end{array}$$

(A27)

$$\begin{array}{cc}\hfill & +4\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right){\sigma }_G^2+4\left(K^{2}l+30L^2{K}^{3}l{\mu }_L^2\right)\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel )\hfill \end{array}$$

(A28)

Note that $\left[S_t\right]=\{q_1^t,...,q_n^t\}$ is the index set demonstrating the sampled clients, which might contain duplicate elements, as the sampling is with replacement. We now proceed to bound the term $A_3^{\prime }$ following [15]:

$$\begin{array}{cc}\hfill A_3^{\prime }& =\mathbb{E}\left[\parallel {{\displaystyle \frac{1}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel \right]\hfill \\ \hfill & ={\displaystyle \frac{1}{n^2}}\mathbb{E}\left[\parallel {\sum _{j=0}^{l-1}\sum _{i=1}^n\sum _{k=0}^{K-1}\tilde{\nabla }{F}_{q_i^{t+j}}^c\left({\mathbf{x}}_{C,q_i^{t+j}}^{t+j,k}\right)}^2\parallel \right]\hfill \\ \hfill & \stackrel{a_1^{\prime }}{=}{\displaystyle \frac{1}{n^2}}\mathbb{E}\left[\parallel {\sum _{j=0}^{l-1}\sum _{i=1}^n\sum _{k=0}^{K-1}\left(\tilde{\nabla }{F}_{q_i^{t+j}}^c\left({\mathbf{x}}_{C,q_i^{t+j}}^{t+j,k}\right)-\nabla F_{q_i^{t+j}}^c\left({\mathbf{x}}_{C,q_i^{t+j}}^{t+j,k}\right)\right)}^2\parallel \right]\hfill \\ \hfill & +{\displaystyle \frac{1}{n^2}}\mathbb{E}\left[\parallel {\sum _{j=0}^{l-1}\sum _{i=1}^n\sum _{k=0}^{K-1}\nabla F_{q_i^{t+j}}^c\left({\mathbf{x}}_{C,q_i^{t+j}}^{t+j,k}\right)}^2\parallel \right]\hfill \\ \hfill & \stackrel{a_2^{\prime }}{\le }{\displaystyle \frac{lK{\sigma }_L^2}{n}}+{\displaystyle \frac{1}{n^2}}\mathbb{E}\left[\parallel {\sum _{j=0}^{l-1}\sum _{i=1}^n\sum _{k=0}^{K-1}\nabla F_{q_i^{t+j}}^c\left({\mathbf{x}}_{C,q_i^{t+j}}^{t+j,k}\right)}^2\parallel \right]\hfill \end{array}$$

(A29)

The equality $a_1^{\prime }$ follows from the fact that $\mathbb{E}\left[\parallel z^2\parallel \right]=\mathbb{E}\left[\parallel {z-\mathbb{E}\left[z\right]}^2\parallel \right]+\parallel {\mathbb{E}\left[z\right]}^2\parallel$ and the inequality $a_2^{\prime }$ is due to assumption 3 and the explanation provided in A10. Now, let’s consider ${\mathbf{t}}_i^j={\sum }_{k=0}^{K-1}\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)$, then:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[\parallel {\sum _{j=0}^{l-1}\sum _{i=1}^n\sum _{k=0}^{K-1}\nabla F_{q_i^{t+j}}^c\left({\mathbf{x}}_{C,q_i^{t+j}}^{t+j,k}\right)}^2\parallel \right]=\mathbb{E}\left[\parallel {\sum _{j=0}^{l-1}\sum _{i=1}^n{\mathbf{t}}_{q_i^{t+j}}^j}^2\parallel \right]\hfill \\ \hfill & =\mathbb{E}\left[\sum _{j=0}^{l-1}\sum _{i=1}^n\parallel {{\mathbf{t}}_{q_i^{t+j}}^j}^2\parallel +\sum _{j=0}^{l-1}\sum _{i\ne z,q_i^{t+j},q_z^{t+j}\in \left[S_{t+j}\right]}\langle {\mathbf{t}}_{q_i^{t+j}}^j,{\mathbf{t}}_{q_z^{t+j}}^j\rangle \right]\hfill \\ \hfill & \stackrel{a_3^{\prime }}{=}\mathbb{E}\left[n\sum _{j=0}^{l-1}\parallel {{\mathbf{t}}_{q_1^{t+j}}^j}^2\parallel +n(n-1)\sum _{j=0}^{l-1}\langle {\mathbf{t}}_{q_1^{t+j}}^j,{\mathbf{t}}_{q_2^{t+j}}^j\rangle \right]\hfill \\ \hfill & ={\displaystyle \frac{n}{m}}\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i^j}^2\parallel +{\displaystyle \frac{n(n-1)}{m^2}}\sum _{j=0}^{l-1}\sum _{i,z\in \left[S\right]}\langle {\mathbf{t}}_i^j,{\mathbf{t}}_z^j\rangle \hfill \\ \hfill & ={\displaystyle \frac{n}{m}}\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i^j}^2\parallel +{\displaystyle \frac{n(n-1)}{m^2}}\sum _{j=0}^{l-1}\parallel {\sum _{i\in \left[S\right]}{\mathbf{t}}_i^j}^2\parallel \hfill \\ \hfill & \stackrel{a_4^{\prime }}{\le }{\displaystyle \frac{n^2}{m}}\underset{A_4^{\prime }}{\underbrace{\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i^j}^2\parallel }}\hfill \end{array}$$

(A30)

Note that the equality $a_3^{\prime }$ is due to independent sampling with replacement as outlined by [15] and $a_4^{\prime }$ follows from A8. Now, we bound the term $A_4^{\prime }$ as follows:

$$\begin{array}{cc}\hfill & \sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i^j}^2\parallel =\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\parallel {\sum _{k=0}^{K-1}\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel \stackrel{a_5^{\prime }}{=}K\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\sum _{k=0}^{K-1}\parallel {\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel \hfill \\ \hfill & =K\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\sum _{k=0}^{K-1}\parallel {\nabla F_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)-\nabla F_i^c\left({\mathbf{x}}_C^{t+j}\right)+\nabla F_i^c\left({\mathbf{x}}_C^{t+j}\right)-\nabla F^c\left({\mathbf{x}}_C^{t+j}\right)+\nabla F^c\left({\mathbf{x}}_C^{t+j}\right)}^2\parallel \hfill \\ \hfill & \stackrel{a_6^{\prime }}{\le }3K{L}^2\sum _{j=0}^{l-1}\sum _{i\in \left[S\right]}\sum _{k=0}^{K-1}\parallel {{\mathbf{x}}_{C,i}^{t+j,k}-{\mathbf{x}}_C^{t+j}}^2\parallel +3ml{K}^2{\sigma }_G^2+3m{K}^2\sum _{j=0}^{l-1}\parallel {\nabla F^c\left({\mathbf{x}}_C^{t+j}\right)}^2\parallel \hfill \\ \hfill & \stackrel{a_7^{\prime }}{\le }15ml{K}^3{L}^2{\mu }_L^2\left({\sigma }_L^2+6K{\sigma }_G^2\right)+3ml{K}^2{\sigma }_G^2\hfill \\ \hfill & +\left(90ml{K}^4{L}^2{\mu }_L^2+3m{K}^2\right)\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \end{array}$$

(A31)

The term $a_5^{\prime }$ follows from the fact A8, the term $a_6^{\prime }$ stems from the fact A8 and the assumption 1. The term $a_7^{\prime }$ is due to A11. Now, plugging A31 into A30 and A30 into A29, we have the following bound on $A_3^{\prime }$:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[\parallel {{\displaystyle \frac{1}{n}}\sum _{j=0}^{l-1}\sum _{i\in \left[S_{t+j}\right]}\sum _{k=0}^{K-1}\tilde{\nabla }{F}_i^c\left({\mathbf{x}}_{C,i}^{t+j,k}\right)}^2\parallel \right]\hfill \\ \hfill & \le \left({\displaystyle \frac{lK}{n}}+15l{K}^3{L}^2{\mu }_L^2\right){\sigma }_L^2+\left(90l{K}^4{L}^2{\mu }_L^2+3l{K}^2\right){\sigma }_G^2\hfill \\ \hfill & +\left(90l{K}^4{L}^2{\mu }_L^2+3K^2\right)\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \end{array}$$

(A32)

Plugging A27, A28 and A32 into A23, with rearrangement and simplification, observe that:

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \le {\displaystyle \frac{2}{{\mu }_{L}Kl}}\left(-\mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{t+l}\right)\right]+F^c\left({\overline{\mathbf{x}}}_C^t\right)\right)+\hfill \\ \hfill & \left(5K{\mu }_L^2{L}^2+4{\mu }_L^2{L}^2\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right)+L{\mu }_L\left({\displaystyle \frac{1}{n}}+15K^2{L}^2{\mu }_L^2\right)\right){\sigma }_L^2+\hfill \\ \hfill & \left(30K^2{\mu }_L^2{L}^2+L{\mu }_L\left(90K^3{L}^2{\mu }_L^2+3K\right)+4{\mu }_L^2{L}^2\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right)\right){\sigma }_G^2+\hfill \\ \hfill & \underset{{\Gamma }^{\prime }}{\underbrace{\left(4{\mu }_L^2{L}^2\left(K^{2}l+30L^2{K}^{3}l{\mu }_L^2\right)+{\displaystyle \frac{L{\mu }_L}{l}}\left(90l{K}^3{L}^2{\mu }_L^2+3K\right)+30K^2{\mu }_L^2{L}^2\right)}}\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \end{array}$$

(A33)

By iterating over t, note that,

$$\begin{array}{cc}\hfill & \sum _{t\in \left[T\right]}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \le {\displaystyle \frac{2}{{\mu }_{L}Kl}}\left(-\mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right]+F^c\left({\overline{\mathbf{x}}}_C^0\right)\right)+\hfill \\ \hfill & {\displaystyle \frac{T}{l}}\left(5K{\mu }_L^2{L}^2+4{\mu }_L^2{L}^2\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right)+L{\mu }_L\left({\displaystyle \frac{1}{n}}+15K^2{L}^2{\mu }_L^2\right)\right){\sigma }_L^2+\hfill \\ \hfill & {\displaystyle \frac{T}{l}}\left(30K^2{\mu }_L^2{L}^2+L{\mu }_L\left(90K^3{L}^2{\mu }_L^2+3K\right)+4{\mu }_L^2{L}^2\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right)\right){\sigma }_G^2+\hfill \\ \hfill & {\Gamma }^{\prime }\sum _t\sum _{j=0}^{l-1}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^{t+j}\right)}^2\parallel \hfill \end{array}$$

(A34)

Due to A19, observe that:

$$\begin{array}{cc}\hfill & \sum _{t\in \left[T\right]}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \le {\displaystyle \frac{2}{{\mu }_{L}Kl}}\left(-\mathbb{E}\left[F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right]+F^c\left({\overline{\mathbf{x}}}_C^0\right)\right)+\hfill \\ \hfill & {\displaystyle \frac{T}{l}}\left(5K{\mu }_L^2{L}^2+4{\mu }_L^2{L}^2\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right)+L{\mu }_L\left({\displaystyle \frac{1}{n}}+15K^2{L}^2{\mu }_L^2\right)\right){\sigma }_L^2+\hfill \\ \hfill & {\displaystyle \frac{T}{l}}\left(30K^2{\mu }_L^2{L}^2+L{\mu }_L\left(90K^3{L}^2{\mu }_L^2+3K\right)+4{\mu }_L^2{L}^2\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right)\right){\sigma }_G^2+\hfill \\ \hfill & {\displaystyle \frac{T}{l}}{\Gamma }^{\prime }{\lambda }_1+{\Gamma }^{\prime }{\lambda }_2\sum _t\mathbb{E}\left[\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \right]\hfill \end{array}$$

(A35)

We let ${\mu }_L\le {\displaystyle \frac{1}{lLK{2}^{1.15l+1.85}}}$, thus:

$$\begin{array}{cc}\hfill & \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^c\left({\overline{\mathbf{x}}}_C^t\right)}^2\parallel \le {\displaystyle \frac{2\left(F^c\left({\overline{\mathbf{x}}}_C^0\right)-F^c\left({\overline{\mathbf{x}}}_C^{*}\right)\right)}{{\mu }_{L}KT}}+\hfill \\ \hfill & \left(5K{\mu }_L^2{L}^2+4{\mu }_L^2{L}^2\left(K^2{l}^2+5L^2{K}^3{l}^2{\mu }_L^2\right)+L{\mu }_L\left({\displaystyle \frac{1}{n}}+15K^2{L}^2{\mu }_L^2\right)\right){\sigma }_L^2+\hfill \\ \hfill & \left(30K^2{\mu }_L^2{L}^2+L{\mu }_L\left(90K^3{L}^2{\mu }_L^2+3K\right)+4{\mu }_L^2{L}^2\left(K^2{l}^2+30L^2{K}^4{l}^2{\mu }_L^2\right)\right){\sigma }_G^2+\hfill \\ \hfill & {\displaystyle \frac{{\Gamma }^{\prime }{\lambda }_1}{1-{\Gamma }^{\prime }{\lambda }_2}}\hfill \end{array}$$

(A36)

This completes the proof.

□

Appendix A.2. Server-Side Model Convergence

Theorem A3. 

Under Assumptions 1, 2, 3, and full participation of clients, if $\mu \le {\displaystyle \frac{1}{8L{m}^2}}$, $t\equiv 0modl$, the convergence rate of the server model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{2l\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (2m-3)T}}+{\displaystyle \frac{L\mu m^2}{2m-3}}\left(9.2{\sigma }_L^2+13.2{\sigma }_G^2\right)\hfill \end{array}$$

Corollary A3. 

Let $\mu \le {\displaystyle \frac{1}{L{m}^2\sqrt{T}}}$, then the convergence rate of the server-side model is:

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}\right)\end{array}$$

Proof. 

Due to Assumption 1, for any ${\mathbf{x}}_S^{t+l}$ and ${\mathbf{x}}_S^t$, it can be written that:

$$\begin{array}{c}\hfill F^s\left({\mathbf{x}}_S^{t+l}\right)\le F^s\left({\mathbf{x}}_S^t\right)+\nabla F^s{\left({\mathbf{x}}_S^t\right)}^{\top }({\mathbf{x}}_S^{t+l}-{\mathbf{x}}_S^t)+{\displaystyle \frac{L}{2}}\parallel {{\mathbf{x}}_S^{t+l}-{\mathbf{x}}_S^t}^2\parallel \end{array}$$

(A37)

Also, note the client forward-propagation and server model update rules in the Algorithm 1,

$$\begin{array}{cc}\hfill & {\mathbf{x}}_{S,i+1}^t={\mathbf{x}}_{S,i}^t-\mu \tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)\hfill \end{array}$$

(A38)

Thus, putting ${\mathbf{x}}_S^t={\mathbf{x}}_{S,0}^t$ and ${\mathbf{x}}_S^{t+l}={\mathbf{x}}_{S,m}^t$, note that:

$$\begin{array}{c}\hfill 1-1\\ \hfill & {\mathbf{x}}_S^{t+l}-{\mathbf{x}}_S^t=-\mu \sum _{i=0}^{m-1}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)\hfill \end{array}$$

(A39)

Taking expectation with respect to randomness at round t, i.e., ${\xi }^{\left[t\right]}\stackrel{\Delta }{=}{\left[{\xi }_i^{\tau }\right]}_{i\in \left[N\right],\tau \in \left[t\right]}$, and plugging A38 into A37 note that:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[F^s\left({\mathbf{x}}_S^{t+l}\right)\right]\le F^s\left({\mathbf{x}}_S^t\right)-\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i=0}^{m-1}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)\right]〉+{\displaystyle \frac{L{\mu }^2}{2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \hfill \\ \hfill & \mathbb{E}\left[F^s\left({\mathbf{x}}_S^{t+l}\right)\right]\le F^s\left({\mathbf{x}}_S^t\right)\underset{B_1}{\underbrace{-\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i=0}^{m-1}\left(\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)\right]〉}}\hfill \\ \hfill & \underset{B_2}{\underbrace{-\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i=0}^{m-1}\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right]〉}}+{\displaystyle \frac{L{\mu }^2}{2}}\underset{B_3}{\underbrace{\mathbb{E}\parallel {\sum _{i=0}^{m-1}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel }}\hfill \end{array}$$

(A40)

The term $B_1$ will be bounded as follows:

$$\begin{array}{cc}\hfill & -\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\mathbb{E}\left[\sum _{i=0}^{m-1}\left(\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)|\xi \right]\right]〉\stackrel{\left(b_1\right)}{=}{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +\hfill \\ \hfill & {\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)}^2\parallel -{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)+\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_2\right)}{\le }{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i=0}^{m-1}\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_3\right)}{\le }{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{m\mu }{2}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_4\right)}{\le }{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{m\mu L^2}{2}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \hfill \end{array}$$

(A41)

The equality $\left(b_1\right)$ is due to $-\langle a,b\rangle ={\displaystyle \frac{1}{2}}\left(\parallel a^2\parallel +\parallel b^2\parallel -\parallel {a+b}^2\parallel \right)$ for any two vectors a and b. The inequality $\left(b_2\right)$ is clear as we dropped a negative term, inequality $\left(b_3\right)$ stems from the fact that $\mathbb{E}\parallel {{\sum }_i^n{z}_i}^2\parallel \le n{\sum }_i\mathbb{E}\left[\parallel {z_i}^2\parallel \right]$ holds for any random variable $z_i$, and the inequality $\left(b_4\right)$ is due to 1.

$$\begin{array}{cc}\hfill & \mathrm{The}\mathrm{term}{B}_2\mathrm{will}\mathrm{be}\mathrm{bounded}\mathrm{as}\mathrm{follows}:\hfill \end{array}$$

$$\begin{array}{cc}\hfill & -\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i=0}^{m-1}\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right]〉\stackrel{\left(b_5\right)}{=}-m\mu \mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A42)

$$\begin{array}{c}\mathrm{Note}\mathrm{that}\mathrm{the}\mathrm{equality}\left(b_5\right)\mathrm{holds}\mathrm{based}\mathrm{on}\mathrm{the}\mathrm{definition}3.\hfill \\ \mathrm{The}\mathrm{term}{B}_3\mathrm{will}\mathrm{be}\mathrm{bounded}\mathrm{as}\mathrm{below}:\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel {\sum _{i=0}^{m-1}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \stackrel{\left(b_6\right)}{\le }m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \hfill \\ \hfill & =m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)+\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_7\right)}{\le }2m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel +2m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_8\right)}{\le }2m^2{\sigma }_L^2+2m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)+\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_S^t\right)+\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_9\right)}{\le }2m^2{\sigma }_L^2+6m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel +6m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +\hfill \\ \hfill & 6m\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_{10}\right)}{\le }2m^2{\sigma }_L^2+6m^2{\sigma }_G^2+6m{L}^2\underset{B_4}{\underbrace{\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2}}\parallel +6m^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A43)

The inequalities $\left(b_6\right),\left(b_7\right),$ and $\left(b_9\right)$ due to the same reason $\left(b_3\right)$ holds above. The inequalities $\left(b_8\right)$ and $\left(b_{10}\right)$ hold due to Assumptions 3. The term $B_4$ is bounded similar to [13, Lemma 3]:

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \stackrel{\left(b_{11}\right)}{\le }\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t\parallel -{\mathbf{x}}_S^t-\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\hfill \\ \hfill & =\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t\parallel -{\mathbf{x}}_S^t}^2+\mathbb{E}\parallel {\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel +2〈{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t,-\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)〉\hfill \\ \hfill & \stackrel{\left(b_{12}\right)}{\le }(1+{\displaystyle \frac{1}{2m-1}})\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +(1+2m)\mathbb{E}\parallel {\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel \hfill \\ \hfill & =(1+{\displaystyle \frac{1}{2m-1}})\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +(1+2m){\mu }^2\mathbb{E}\parallel \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)+\hfill \\ \hfill & \nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_S^t\right)+\nabla F_{i-1}^s\left({\mathbf{x}}_S^t\right)-\nabla F^s\left({\mathbf{x}}_S^t\right)+\nabla F^s\left({\mathbf{x}}_S^t\right){\parallel }^2\hfill \\ \hfill & \stackrel{\left(b_{13}\right)}{\le }(1+{\displaystyle \frac{1}{2m-1}})\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2\mathbb{E}\parallel {\tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel +\hfill \\ \hfill & 4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_S^t\right)}^2\parallel +4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F_{i-1}^s({\mathbf{x}}_S^t\parallel )-\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\hfill \\ \hfill & +4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_{14}\right)}{\le }(1+{\displaystyle \frac{1}{2m-1}}+4(1+2m){\mu }^2{L}^2)\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+\hfill \\ \hfill & 4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

The A38 implies the inequality $\left(b_{11}\right)$. The inequality $\left(b_{12}\right)$ holds true based on $2\langle a,b\rangle \le {\displaystyle \frac{1}{n-1}}\parallel a^2\parallel +n\parallel b^2\parallel$ for any two vectors a, b and positive number n. The inequality $\left(b_{13}\right)$ follows from the previously mentioned fact at the inequality $\left(b_3\right)$, and the inequality $\left(b_{14}\right)$ is based on Assumptions 3 and 1. Given $\mu \le {\displaystyle \frac{1}{2L(2m+1)}}$ and by averaging over the clients, observe that:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \le (1+{\displaystyle \frac{4m}{4m^2-1}}){\displaystyle \frac{1}{m}}\sum _{i=1}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+\hfill \\ \hfill & 4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \le (1+{\displaystyle \frac{1}{m-1}}){\displaystyle \frac{1}{m}}\sum _{i=1}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

Unrolling the recursion, following [13, Lemma 3], it is inferred that:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \hfill \\ \hfill & \le \sum _{j=0}^{m-1}{(1+{\displaystyle \frac{1}{m-1}})}^j\left(4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \right)\hfill \\ \hfill & \le (m-1)\times \left[{\left(1+{\displaystyle \frac{1}{m-1}}\right)}^m-1\right]\times \left[4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \right]\hfill \\ \hfill & \le 16(m+2m^2){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+16(m+2m^2){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A44)

Note that in the above inequality, ${\left(1+{\displaystyle \frac{1}{m-1}}\right)}^m-1\le 4$ for $m>1$.

$$\begin{array}{cc}\hfill & \mathrm{Plugging}A41,A42,A43\mathrm{and}A44\mathrm{into}A40,\mathrm{observe}\mathrm{that}:\hfill \\ \hfill & \mathbb{E}\left[F^s\left({\mathbf{x}}_S^{t+l}\right)\right]\le F^s\left({\mathbf{x}}_S^t\right)+{\displaystyle \frac{1}{2}}\left(16L^2{\mu }^3{m}^3(1+2m)+2L{\mu }^2{m}^2+96L^3{\mu }^4{m}^3(1+2m)\right){\sigma }_L^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\left(16L^2{\mu }^3{m}^3(1+2m)+6L{\mu }^2{m}^2+96L^3{\mu }^4{m}^3(1+2m)\right){\sigma }_G^2+\hfill \\ \hfill & {\displaystyle \frac{1}{2}}\left(-2\mu m+6L{m}^2{\mu }^2+\mu +96L^3{\mu }^4{m}^3(1+2m)+16L^2{\mu }^3{m}^3(1+2m)\right)\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_{15}\right)}{\le }{F}^s\left({\mathbf{x}}_S^t\right)+{\displaystyle \frac{\mu (3-2m)}{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{L{\mu }^2{m}^2}{2}}\left(9.2{\sigma }_L^2+13.2{\sigma }_G^2\right)\hfill \\ \hfill & \mathrm{Rearranging}\mathrm{the}\mathrm{terms},\mathit{and}\mathrm{summing}\mathrm{over}t,\mathrm{observe}\mathrm{that}:\hfill \\ \hfill & \sum _t\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{2\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (2m-3)}}+\sum _t{\displaystyle \frac{L\mu m^2}{2m-3}}\left(9.2{\sigma }_L^2+13.2{\sigma }_G^2\right)\hfill \end{array}$$

(A45)

$$\begin{array}{cc}\hfill & \mathrm{Assuming}\mathrm{there}\mathrm{are}T\mathrm{global}\mathrm{rounds}\mathrm{overall},\hfill \\ \hfill & \underset{t}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{2l\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (2m-3)T}}+{\displaystyle \frac{L\mu m^2}{2m-3}}\left(9.2{\sigma }_L^2+13.2{\sigma }_G^2\right)\hfill \end{array}$$

(A46)

□

Theorem A4. 

Under Assumptions 1, 2, 3, and full participation of clients, if $\mu \le {\displaystyle \frac{1}{8L^2{m}^2}}$, $t\equiv 0modl$, the convergence rate of the server model of Algorithm 1 satisfies:

$$\begin{array}{cc}\hfill & \underset{t}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{l\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (m-2)T}}+{\displaystyle \frac{L\mu m^2}{m-2}}\left(7{\sigma }_L^2+7{\sigma }_G^2\right)\hfill \end{array}$$

Corollary A4. 

Let $\mu \le {\displaystyle \frac{1}{L^2{m}^2\sqrt{T}}}$, then the convergence rate of the server-side model is:

$$\begin{array}{c}\hfill \underset{t\in \left[T\right]}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le \mathcal{O}\left({\displaystyle \frac{l}{\sqrt{T}}}\right)\end{array}$$

Proof. 

Due to Assumption 1, for any ${\mathbf{x}}_S^{t+l}$ and ${\mathbf{x}}_S^t$, it can be written that:

$$\begin{array}{c}\hfill F^s\left({\mathbf{x}}_S^{t+l}\right)\le F^s\left({\mathbf{x}}_S^t\right)+\nabla F^s{\left({\mathbf{x}}_S^t\right)}^{\top }({\mathbf{x}}_S^{t+l}-{\mathbf{x}}_S^t)+{\displaystyle \frac{L}{2}}\parallel {{\mathbf{x}}_S^{t+l}-{\mathbf{x}}_S^t}^2\parallel \end{array}$$

(A47)

Also, note the client forward-propagation and server model update rules in the Algorithm 1,

$$\begin{array}{cc}\hfill & {\mathbf{x}}_{S,i+1}^t={\mathbf{x}}_{S,i}^t-\mu \tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)\hfill \end{array}$$

(A48)

Thus, putting ${\mathbf{x}}_S^t={\mathbf{x}}_{S,0}^t$ and ${\mathbf{x}}_S^{t+l}={\mathbf{x}}_{S,m}^t$, note that:

$$\begin{array}{c}\hfill 1-1\\ \hfill & {\mathbf{x}}_S^{t+l}-{\mathbf{x}}_S^t=-\mu \sum _{i=0}^{m-1}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)\hfill \end{array}$$

(A49)

Taking expectation for both types of randomness, i.e., randomness due to stochastic gradients and due to sampling of clients, at round t, and plugging A48 into A47 note that:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[F^s\left({\mathbf{x}}_S^{t+l}\right)\right]\le F^s\left({\mathbf{x}}_S^t\right)-\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i\in \left[S_t\right]}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)\right]〉+{\displaystyle \frac{L{\mu }^2}{2}}\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \hfill \\ \hfill & \mathbb{E}\left[F^s\left({\mathbf{x}}_S^{t+l}\right)\right]\le F^s\left({\mathbf{x}}_S^t\right)\underset{B_1}{\underbrace{-\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i\in \left[S_t\right]}\left(\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)\right]〉}}\hfill \\ \hfill & \underset{B_2}{\underbrace{-\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i\in \left[S_t\right]}\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right]〉}}+{\displaystyle \frac{L{\mu }^2}{2}}\underset{B_3}{\underbrace{\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel }}\hfill \end{array}$$

(A50)

Note that $\left[S_t\right]=\{q_1^t,...,q_n^t\}$ is the index set demonstrating the sampled clients, which might contain duplicate elements, as the sampling is with replacement. The term $B_1$ will be bounded as follows:

$$\begin{array}{cc}\hfill & -\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\mathbb{E}\left[\sum _{i\in \left[S_t\right]}\left(\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)|\xi \right]\right]〉\stackrel{\left(b_1\right)}{=}{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +\hfill \\ \hfill & {\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)}^2\parallel -{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)+\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_2\right)}{\le }{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\left(\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_3\right)}{\le }{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\sum _{i=1}^n\left(\nabla F_{q_i^t}^s\left({\mathbf{x}}_{S,q_i^t}^t\right)-\nabla F_{q_i^t}^s\left({\mathbf{x}}_S^t\right)\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_4\right)}{\le }{\displaystyle \frac{\mu }{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{n\mu L^2}{2}}\sum _{i=1}^n\mathbb{E}\parallel {{\mathbf{x}}_{S,q_i^t}^t-{\mathbf{x}}_S^t}^2\parallel \hfill \end{array}$$

(A51)

The equality $\left(b_1\right)$ is due to $-\langle a,b\rangle ={\displaystyle \frac{1}{2}}\left(\parallel a^2\parallel +\parallel b^2\parallel -\parallel {a+b}^2\parallel \right)$ for any two vectors a and b. The inequality $\left(b_2\right)$ is clear as we dropped a negative term, inequality $\left(b_4\right)$ stems from the fact that $\mathbb{E}\parallel {{\sum }_i^n{z}_i}^2\parallel \le n{\sum }_i\mathbb{E}\left[\parallel {z_i}^2\parallel \right]$ holds for any random variable $z_i$, and 1.

The term $B_2$ will be bounded as follows:

$$\begin{array}{cc}\hfill & -\mu 〈\nabla F^s\left({\mathbf{x}}_S^t\right),\mathbb{E}\left[\sum _{i\in \left[S_t\right]}\nabla F_i^s\left({\mathbf{x}}_S^t\right)\right]〉\stackrel{\left(b_5\right)}{=}-n\mu \mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A52)

$$\begin{array}{cc}\hfill & \mathrm{Note}\mathrm{that}\mathrm{equality}\left(b_5\right)\mathrm{holds}\mathrm{based}\mathrm{on}\mathrm{the}\mathrm{definition}3.\hfill \\ \hfill & \mathrm{The}\mathrm{term}{B}_3\mathrm{will}\mathrm{be}\mathrm{bounded}\mathrm{as}\mathrm{below}:\hfill \\ \hfill & \mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \stackrel{\left(b_6\right)}{\le }\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel +\mathbb{E}\parallel {\sum _{i\in \left[S_t\right]}\left(\tilde{\nabla }{F}_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_7\right)}{\le }n{\sigma }_L^2+\mathbb{E}\parallel {\sum _{i=1}^n\nabla F_{q_i^t}^s\left({\mathbf{x}}_{S,q_i^t}^t\right)}^2\parallel \hfill \end{array}$$

(A53)

Now, let’s consider ${\mathbf{t}}_i=\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)$ following the ideas of [15] for this part:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[\parallel {\sum _{i=1}^n\nabla F_{q_i^t}^s\left({\mathbf{x}}_{S,q_i^t}^t\right)}^2\parallel \right]=\mathbb{E}\left[\parallel {\sum _{i=1}^n{\mathbf{t}}_{q_i^t}}^2\parallel \right]\hfill \\ \hfill & =\mathbb{E}\left[\sum _{i=1}^n\parallel {{\mathbf{t}}_{q_i^t}}^2\parallel +\sum _{i\ne z,q_i^t,q_z^t\in \left[S_t\right]}\langle {\mathbf{t}}_{q_i^t},{\mathbf{t}}_{q_z^t}\rangle \right]\hfill \\ \hfill & \stackrel{b_8}{=}\mathbb{E}\left[n\parallel {{\mathbf{t}}_{q_1^t}}^2\parallel +n(n-1)\langle {\mathbf{t}}_{q_1^t},{\mathbf{t}}_{q_2^t}\rangle \right]\hfill \\ \hfill & ={\displaystyle \frac{n}{m}}\sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i}^2\parallel +{\displaystyle \frac{n(n-1)}{m^2}}\sum _{i,z\in \left[S\right]}\langle {\mathbf{t}}_i,{\mathbf{t}}_z\rangle \hfill \\ \hfill & ={\displaystyle \frac{n}{m}}\sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i}^2\parallel +{\displaystyle \frac{n(n-1)}{m^2}}\parallel {\sum _{i\in \left[S\right]}{\mathbf{t}}_i}^2\parallel \hfill \\ \hfill & \stackrel{b_9}{\le }{\displaystyle \frac{n^2}{m}}\underset{B_4}{\underbrace{{\sum }_{i\in \left[S\right]}\parallel {{\mathbf{t}}_i}^2\parallel }}\hfill \end{array}$$

(A54)

Note that the equality $b_8$ is due to independent sampling with replacement as outlined by [15]. The inequality $b_9$ follows from A8. We bound the term $B_4$ as follows:

$$\begin{array}{cc}\hfill & \sum _{i\in \left[S\right]}\parallel {{\mathbf{t}}_i}^2\parallel =\hfill \\ \hfill & =\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel \hfill \\ \hfill & =\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)+\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_S^t\right)+\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \le 3\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F_i^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)}^2\parallel +3\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_{S,i}^t\right)-\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +\hfill \\ \hfill & 3\sum _{i=0}^{m-1}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_{10}\right)}{\le }3m{\sigma }_G^2+3L^2\underset{B_5}{\underbrace{{\sum }_{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel }}+3m\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A55)

The inequality $\left(b_{10}\right)$ holds due to Assumptions 3.

The term $B_5$ is bounded similar to [13, Lemma 3]:

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \stackrel{\left(b_{11}\right)}{\le }\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t-\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel \hfill \\ \hfill & =\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +\mathbb{E}\parallel {\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel +2〈{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t,-\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)〉\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \stackrel{\left(b_{12}\right)}{\le }(1+{\displaystyle \frac{1}{2m-1}})\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +(1+2m)\mathbb{E}\parallel {\mu \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel \hfill \\ \hfill & =(1+{\displaystyle \frac{1}{2m-1}})\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +(1+2m){\mu }^2\mathbb{E}\parallel \tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)+\hfill \\ \hfill & \nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_S^t\right)+\nabla F_{i-1}^s\left({\mathbf{x}}_S^t\right)-\nabla F^s\left({\mathbf{x}}_S^t\right)+\nabla F^s\left({\mathbf{x}}_S^t\right){\parallel }^2\hfill \\ \hfill & \stackrel{\left(b_{13}\right)}{\le }(1+{\displaystyle \frac{1}{2m-1}})\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2\mathbb{E}\parallel {\tilde{\nabla }{F}_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)}^2\parallel +\hfill \\ \hfill & 4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F_{i-1}^s\left({\mathbf{x}}_{S,i-1}^t\right)-\nabla F_{i-1}^s\left({\mathbf{x}}_S^t\right)}^2\parallel +4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F_{i-1}^s({\mathbf{x}}_S^t\parallel )-\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\hfill \\ \hfill & +4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_{14}\right)}{\le }(1+{\displaystyle \frac{1}{2m-1}}+4(1+2m){\mu }^2{L}^2)\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+\hfill \\ \hfill & 4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

The A48 implies the inequality $\left(b_{11}\right)$. The inequality $\left(b_{12}\right)$ holds true based on $2\langle a,b\rangle \le {\displaystyle \frac{1}{n-1}}\parallel a^2\parallel +n\parallel b^2\parallel$ for any two vectors a, b and positive number n. The inequality $\left(b_{13}\right)$ follows from the previously mentioned fact at the inequality $\left(b_3\right)$, and the inequality $\left(b_{14}\right)$ is based on Assumptions 3 and 1.

Given $\mu \le {\displaystyle \frac{1}{2L(2m+1)}}$ and by averaging over the clients, observe that:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \le (1+{\displaystyle \frac{4m}{4m^2-1}}){\displaystyle \frac{1}{m}}\sum _{i=1}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+\hfill \\ \hfill & 4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \le (1+{\displaystyle \frac{1}{m-1}}){\displaystyle \frac{1}{m}}\sum _{i=1}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i-1}^t-{\mathbf{x}}_S^t}^2\parallel +4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A56)

Unrolling the recursion, following [13, Lemma 3], it is inferred that:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\mathbb{E}\parallel {{\mathbf{x}}_{S,i}^t-{\mathbf{x}}_S^t}^2\parallel \hfill \\ \hfill & \le \sum _{j=0}^{m-1}{(1+{\displaystyle \frac{1}{m-1}})}^j\left(4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \right)\hfill \\ \hfill & \le (m-1)\times \left[{\left(1+{\displaystyle \frac{1}{m-1}}\right)}^m-1\right]\times \left[4(1+2m){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+4(1+2m){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \right]\hfill \\ \hfill & \le 16(m+2m^2){\mu }^2({\sigma }_L^2+{\sigma }_G^2)+16(m+2m^2){\mu }^2\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \end{array}$$

(A57)

Note that in the above inequality, ${\left(1+{\displaystyle \frac{1}{m-1}}\right)}^m-1\le 4$ for $m>1$. Plugging A51, A52, A55 and A57 into A50, observe that

$$\begin{array}{cc}\hfill & \mathbb{E}\left[F^s\left({\mathbf{x}}_S^{t+l}\right)\right]\le F^s\left({\mathbf{x}}_S^t\right)+{\displaystyle \frac{1}{2}}\left(16L^4(n^3+2n^4){\mu }^3+Ln{\mu }^2+48L^3{n}^2{\mu }^4(m+2m^2)\right){\sigma }_L^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\left(16L^4(n^3+2n^4){\mu }^3+3L{\mu }^2{n}^2+48L^3{n}^2{\mu }^4(m+2m^2)\right){\sigma }_G^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\left(\mu -2n\mu +16L^4(n^3+2n^4){\mu }^3+3L{\mu }^2{n}^2+48L^3{n}^2{\mu }^4(m+2m^2)\right)\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \hfill \\ \hfill & \stackrel{\left(b_{15}\right)}{\le }{F}^s\left({\mathbf{x}}_S^t\right)+{\displaystyle \frac{\mu (4-2m)}{2}}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel +{\displaystyle \frac{L\mu m^2}{2}}\left(14{\sigma }_L^2+14{\sigma }_G^2\right)\hfill \end{array}$$

(A58)

After simplifications, the inequality $b_{15}$ holds as $\mu \le {\displaystyle \frac{1}{8L^2{m}^2}}$. Rearranging the terms, and summing over t, observe that:

$$\begin{array}{cc}\hfill & \sum _t\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)}{\mu (m-2)}}+\sum _t{\displaystyle \frac{L\mu m^2}{m-2}}\left(7{\sigma }_L^2+7{\sigma }_G^2\right)\hfill \end{array}$$

(A59)

Assuming there are T global rounds overall,

$$\begin{array}{cc}\hfill & \underset{t}{min}\mathbb{E}\parallel {\nabla F^s\left({\mathbf{x}}_S^t\right)}^2\parallel \le {\displaystyle \frac{l\left(F^s\left({\mathbf{x}}_S^0\right)-F^s\left({\mathbf{x}}_S^{*}\right)\right)}{\mu (m-2)T}}+{\displaystyle \frac{L\mu m^2}{m-2}}\left(7{\sigma }_L^2+7{\sigma }_G^2\right)\hfill \end{array}$$

(A60)

This concludes the proof. □