Comparative Performance Evaluation of Federated Learning Frameworks for Edge-Oriented 6G Applications

1. Introduction

Sixth-generation (6G) wireless networks—standardized internationally as IMT-2030 [2] and studied in parallel by 3GPP Release 20 [3]—will connect massive numbers of intelligent edge devices—smartphones, vehicles, industrial sensors, and wearables—that produce data locally but benefit from collaborative model training without centralizing raw data [4,5]. Unlike cloud-centric machine learning, edge intelligence pushes inference and increasingly training toward the network periphery [6,7], where uplink bandwidth, latency, and device availability are constrained [8,9]. Federated learning (FL) addresses this setting by keeping private datasets ${\mathcal{D}}_k$ on client devices while a coordinator aggregates model updates according to a shared protocol such as FedAvg [10,11,12].

1.1. Motivation: Framework Selection at the 6G Edge

Deploying FL in 6G scenarios involves more than choosing an aggregation algorithm. Systems engineers must select a software framework that orchestrates client–server communication, local training loops, checkpointing, and (optionally) privacy or security extensions [13,14,15]. Candidate stacks include Flower [16], TensorFlow Federated (TFF) [17], FedML [13], NVIDIA FLARE [14], OpenFL [18], and PySyft [15]. Each targets a different ecosystem, yet all must ultimately execute the same federated training contract: broadcast ${\mathbf{w}}^{\left(t\right)}$, train locally, upload updates, aggregate, and evaluate.

Framework documentation and tutorials rarely answer a practical deployment question: when the same model, optimizer, data partition, and number of rounds are fixed, how much do accuracy, wall-clock time, and communication bytes differ across frameworks? This question matters for 6G service orchestration because edge training jobs must fit within finite scheduling windows and uplink budgets [19,20]. A framework that is correct in principle but slow or heavy on the wire may be unsuitable for URLLC-style deadlines or mMTC-scale deployments even if its API is convenient.

1.2. Evaluation Dimensions

We evaluate frameworks along three axes that map directly to Equations (5)–(6) in Section 3:

• Model quality — global test accuracy ${\mathrm{Acc}}^{\left(t\right)}$ and cross-entropy loss after each of $T=10$ communication rounds;
• Computational efficiency — total wall-clock time $T_{\mathrm{wall}}$ on a documented Proxmox containerized testbed;
• Communication cost — simulated parameter traffic $C_{\mathrm{total}}$ for reference PyTorch runners (baseline before 6G bandwidth profiles are applied).

We deliberately hold the FL algorithm constant (FedAvg, IID partition, seed 42) so observed differences reflect runtime stacks, orchestration overhead, and numerical training paths rather than algorithmic diversity [21]. We further distinguish native framework integrations from a shared reference PyTorch FedAvg loop (Section 3), reporting both in every result file.

1.3. Research Questions and Paper Organization

This paper addresses three research questions:

• RQ1: Under identical FedAvg settings and IID data, how do six FL frameworks compare in accuracy and wall time on MNIST?
• RQ2: When task difficulty increases (CIFAR-10 edge vision), do framework rankings hold or do stack-specific dynamics widen gaps?
• RQ3: What communication volume does FedAvg incur, establishing a baseline before 6G bandwidth constraints?

Our study follows a two-phase design: Phase 1 (MNIST) verifies protocol correctness quickly; Phase 2 (CIFAR-10) stresses the same frameworks on a harder vision task representative of camera and sensing workloads at the edge [6,22]. Section 2 reviews prior work; Section 2.4 positions our contribution against surveys, scale benchmarks, and vendor documentation. Section 3 formalizes FedAvg and describes the testbed. Section 4 presents accuracy, loss, timing, and communication results with nine figures. Section 5Section 6 interpret findings and outline 6G/security extensions.

2. Related Work

2.1. Federated Learning at the 6G Edge

Federated learning was introduced as a communication-efficient approach for training deep models on decentralized data [10,19]. Comprehensive surveys document algorithms, privacy mechanisms, and open problems [11,21,23]. For wireless and mobile networks, FL surveys emphasize latency, bandwidth, and heterogeneity constraints at the edge [8,12]. Sixth-generation network visions and FL-specific 6G surveys explicitly motivate distributed machine learning close to users and devices [2,3,4,5,20], while edge-computing literature argues that intelligence should be pushed toward the network periphery [6,7,9]. Our work sits at this intersection: we do not yet model full 6G radio channels, but we establish reproducible framework baselines that future work can extend with bandwidth and latency profiles aligned to IMT-2030 usage scenarios [2].

Non-IID data, stragglers, and heterogeneous compute remain active research topics [24,25,26,27]. Security and privacy extensions—secure aggregation [28], differential privacy [29], and poisoning attacks [30,31]—are orthogonal to our Phase 1–2 ideal-network study but motivate the benchmark harness as a platform for later security chapters.

2.2. FL Frameworks and Tooling

Multiple open-source FL frameworks target different ecosystems. Flower provides a framework-agnostic client–server abstraction widely used in research [16]. TensorFlow Federated (TFF) integrates tightly with TensorFlow and supports simulation at scale [17]. FedML targets cross-device and cross-silo production workflows with a unified research library [13]. NVIDIA FLARE focuses on enterprise and regulated deployments, including clinical use cases [14]. OpenFL supports Intel-based federated research clusters [18]. PySyft emphasizes privacy-preserving and encrypted computation over distributed data [15,32].

Each project publishes tutorials and performance claims for its own runtime. However, framework documentation typically demonstrates a single stack under settings chosen by the vendor. Cross-framework comparisons are rarely conducted with matched models, optimizers, random seeds, and aggregation semantics. Because frameworks are not interoperable in one federated round, fair comparison requires a disciplined experimental protocol—the gap our study addresses.

2.3. FL Benchmarks and Empirical Studies

Several benchmark suites study federated learning at scale or in realistic domains. LEAF provides federated datasets and reference implementations for natural user partitions [33]. FedScale benchmarks system and model performance with thousands of clients, focusing on scalability and scheduling [34]. FLBench proposes a broader benchmark suite across tasks and algorithms [35]. FLamby targets healthcare cross-silo settings with domain-realistic datasets [36]. These contributions advance FL methodology and scale but generally emphasize algorithms, datasets, or system scheduling rather than answering: given the same FedAvg job, how do today’s popular Python FL frameworks differ in accuracy, time, and bytes?

2.4. Research Gap and Added Value

Table 1 contrasts our study with representative prior lines of work. Existing resources are complementary; none simultaneously (i) compare six mainstream frameworks under locked hyperparameters and model topology, (ii) formalize the exact FedAvg and communication equations implemented, (iii) publish per-round benchmark metrics and global model checkpoints alongside a containerized evaluation protocol [1], and (iv) structure experiments as a two-phase MNIST→CIFAR-10 progression aimed at 6G edge vision workloads.

Added value. This paper makes four concrete contributions beyond prior surveys and benchmarks:

1.

Controlled multi-framework comparison. We execute Flower, TFF, FedML, NVFlare, OpenFL, and PySyft under one protocol (six IID clients, ten rounds, $\eta =0.01$, $B=32$, seed 42), isolating framework/runtime effects from algorithmic diversity.

2.

Evidence that task difficulty matters. MNIST shows near-identical accuracy ($\le 0.13$ pp spread) but 80% wall-time variation; CIFAR-10 exposes a $\sim 3.4$ pp accuracy gap (TFF vs. PyTorch runners). This supports the methodological claim that easy datasets alone are insufficient for framework selection in edge vision scenarios.

3.

Documented 6G-oriented testbed. Proxmox virtual machines, per-framework container images, and a unified evaluation harness with per-round metric logging provide a transparent experimental record; benchmark metrics and global checkpoints are published in [1].

4.

Foundation for thesis extensions. Communication baselines (∼798 MB MNIST, ∼281 MB CIFAR-10) and the documented evaluation pipeline provide a starting point for 6G network profiles (eMBB, URLLC, mMTC), non-IID Dirichlet partitions [24], and security experiments [28,30] without redesigning the study design.

To our knowledge, this is among the first studies to report head-to-head FedAvg metrics for six widely cited Python FL frameworks with explicit native-versus-reference implementation labels, a formal communication-cost model, and publicly released result artifacts [1].

3. Materials and Methods

3.1. Problem Formulation

We consider a cross-device federated learning setting with K clients indexed by $k\in \{1,\dots ,K\}$ [10,11]. Client k holds a private dataset ${\mathcal{D}}_k={\left\{(x_i,y_i)\right\}}_{i=1}^{n_k}$ drawn from an unknown distribution over the input–label space $\mathcal{X}\times \mathcal{Y}$. Let $N={\sum }_{k=1}^K{n}_k$ denote the total number of training samples. Each client trains a model parameterized by $\mathbf{w}\in {\mathbb{R}}^d$, where d is the number of learnable parameters.

The local empirical risk at client k is

$$F_k\left(\mathbf{w}\right)={\displaystyle \frac{1}{n_k}}\sum _{(x,y)\in {\mathcal{D}}_k}\ell \left(f_{\mathbf{w}}\left(x\right),y\right),$$

(1)

where $f_{\mathbf{w}}:\mathcal{X}\to {\mathbb{R}}^{\left|\mathcal{Y}\right|}$ is the neural network mapping, $\ell (·,·)$ is the cross-entropy loss, and $\left|\mathcal{Y}\right|=10$ for both MNIST and CIFAR-10. The global federated objective is the sample-size weighted average

$$F\left(\mathbf{w}\right)={\displaystyle \sum _{k=1}^K}{\displaystyle \frac{n_k}{N}}{F}_k\left(\mathbf{w}\right).$$

(2)

In the ideal case, FL seeks ${\mathbf{w}}^{★}\in arg{min}_{\mathbf{w}}F\left(\mathbf{w}\right)$ without exchanging raw data ${\mathcal{D}}_k$ [12].

Table 2 summarizes the notation used throughout this paper.

3.2. Federated Averaging (FedAvg)

We employ Federated Averaging (FedAvg) [10], the de facto baseline algorithm for cross-framework comparison [19,21]. At communication round t, the server broadcasts the current global model ${\mathbf{w}}^{\left(t\right)}$ to all participating clients. Each client k performs E epochs of mini-batch stochastic gradient descent (SGD) on ${\mathcal{D}}_k$, initialized at ${\mathbf{w}}^{\left(t\right)}$. Denoting the local optimizer update by $\mathrm{LocalUpdate}(·)$, the client model after local training is

$${\mathbf{w}}_k^{(t+1)}=\mathrm{LocalUpdate}\left({\mathbf{w}}^{\left(t\right)};{\mathcal{D}}_k,\eta ,E,B\right).$$

(3)

The server aggregates client models by sample-size weighted averaging:

$${\mathbf{w}}^{(t+1)}={\displaystyle \sum _{k=1}^K}{\displaystyle \frac{n_k}{N}}{\mathbf{w}}_k^{(t+1)}.$$

(4)

Equations (3)–(4) reduce to single-step local SGD followed by averaging when $E=1$ [27]. All six frameworks in this study implement Equation (4); differences arise from runtime stacks (TensorFlow vs. PyTorch) and orchestration overhead, not from the aggregation formula itself.

For non-IID data, convergence of FedAvg can degrade because local objectives $F_k$ diverge from the global objective F [24]. Our experiments use IID partitions (uniform random split with fixed seed) to isolate framework effects; extensions to Dirichlet non-IID splits are planned [25,26].

3.3. Evaluation Metrics

After each round t, the global model ${\mathbf{w}}^{\left(t\right)}$ is evaluated on a held-out test set ${\mathcal{D}}_{\mathrm{test}}$:

$${\mathrm{Acc}}^{\left(t\right)}={\displaystyle \frac{1}{|{\mathcal{D}}_{\mathrm{test}}|}}\sum _{(x,y)\in {\mathcal{D}}_{\mathrm{test}}}\mathbb{I}\left[arg\underset{c}{max}{f}_{{\mathbf{w}}^{\left(t\right)}}{\left(x\right)}_c=y\right],$$

(5)

where $\mathbb{I}[·]$ is the indicator function and $f_{{\mathbf{w}}^{\left(t\right)}}{\left(x\right)}_c$ is the logit for class c. We also report the average cross-entropy loss on ${\mathcal{D}}_{\mathrm{test}}$. System-level metrics include total wall-clock time $T_{\mathrm{wall}}$ and per-round communication volume. For the reference PyTorch runner, the simulated bytes per round are

$$C_{\mathrm{round}}={\displaystyle \sum _{k=1}^K}\left({\left|\mathbf{w}\right|}_{\mathrm{bytes}}+{\left|\mathbf{w}\right|}_{\mathrm{bytes}}\right)=2K{\left|\mathbf{w}\right|}_{\mathrm{bytes}},$$

(6)

where ${\left|\mathbf{w}\right|}_{\mathrm{bytes}}$ is the serialized model size (upload of client weights plus download of the global model). Over T rounds, $C_{\mathrm{total}}=T·C_{\mathrm{round}}$. Under constrained 6G uplinks, $C_{\mathrm{total}}$ bounds the minimum transfer time given bandwidth $B_{\mathrm{uplink}}$ via $T_{\mathrm{comm}}\ge C_{\mathrm{total}}/B_{\mathrm{uplink}}$ [8,19].

3.4. Two-Phase Experimental Design

We evaluate frameworks in two phases of increasing difficulty while holding FL hyperparameters constant (Table 3).

Phase 1 uses MNIST [37] (28×28 grayscale digits) for fast framework screening. Phase 2 uses CIFAR-10 [22] (32×32 RGB images) as a closer proxy for edge vision workloads [6,38].

3.5. Models

Each dataset uses a lightweight convolutional neural network sized for edge feasibility [38]. MNIST employs two convolutional layers with pooling followed by two fully connected layers (512 units, 10 outputs). CIFAR-10 adds a third convolutional block. TFF implements an equivalent Keras model; PyTorch runners (Flower and the shared reference) share the same topology. The parameter count d determines ${\left|\mathbf{w}\right|}_{\mathrm{bytes}}$ in Equation (6).

3.6. Framework Implementations

Table 4 lists the runner used per framework. Flower (v1.31.0) [16] and TFF (v0.85.0) [17] use native integrations. FedML [13], NVFlare [14], OpenFL [18], and PySyft [15] execute a shared PyTorch FedAvg reference loop implementing Equations (3)–(4) with identical CNN, data split, and hyperparameters. Differences among reference frameworks reflect measurement noise, not distinct SDK behaviour. Results are labelled as native (framework SDK) or reference (shared PyTorch loop) throughout tables and figures.

3.7. Benchmark Harness and Infrastructure

We developed a custom benchmark harness to enforce protocol parity across frameworks. The harness (i) loads MNIST and CIFAR-10 with a fixed IID partition (seed 42), (ii) instantiates the dataset-specific CNN, (iii) orchestrates ten FedAvg rounds with the hyperparameters in Table 3, (iv) evaluates the global model on the held-out test set after each round, and (v) records accuracy, loss, wall time, and (for reference runners) simulated communication volume. For frameworks without a native FedAvg path under our constraints, the same harness invokes a shared PyTorch reference loop so that local training, aggregation, and evaluation follow Equations (3)–(5) exactly.

Experiments run on a Proxmox hypervisor [39] with two Ubuntu 22.04.05 LTS Server virtual machines [40]: a coordinator host (reserved for optional distributed aggregation; idle in the standalone configuration) and a worker host that executes one framework container at a time. The worker was allocated 4 vCPUs and 8 GB RAM. Each framework is packaged in an isolated container image with pinned SDK dependencies to avoid cross-contamination between runs.

For each framework $f\in \{\mathrm{flower},\mathrm{tff},\mathrm{fedml},\mathrm{nvflare},\mathrm{openfl},\mathrm{pysyft}\}$ and dataset $d\in \{\mathrm{MNIST},\mathrm{CIFAR}-10\}$, the workflow comprises three steps: build the framework-specific container image, execute the federated benchmark on the worker host, and archive per-round metrics together with the final global model state. All runs use standalone simulation: six virtual clients co-located inside a single container, under an ideal network profile (zero latency, unlimited bandwidth). This isolates framework runtime and training-stack effects from distributed networking variability. Per-round JSON summaries and global model checkpoints are published in [1]; benchmark source code and container images are not released.

4. Results

This section reports empirical answers to RQ1–RQ3. All figures use the per-round metric logs from Phase 1 (MNIST) and Phase 2 (CIFAR-10). Solid lines and hatched bars denote native runners (Flower, TFF); dashed lines denote the shared PyTorch reference loop (FedML, NVFlare, OpenFL, PySyft).

4.1. Overview

Figure 1 provides a high-level snapshot of round-10 test accuracy across both datasets. MNIST accuracies cluster above 99.2% for every framework, while CIFAR-10 separates native TFF (∼71.7%) from the PyTorch cluster (∼68%). Figure 2 quantifies how much each framework improves from round 1 to round 10: CIFAR-10 gains are an order of magnitude larger (∼25–31 pp) than MNIST (∼1.6–1.9 pp), confirming that Phase 2 is the discriminative benchmark.

4.2. Phase 1: MNIST

Table 5 summarizes results after ten FedAvg rounds. All frameworks converge to 99.22–99.35% accuracy (spread 0.13 percentage points), validating that Equations (4)–(5) were implemented consistently. Wall time spans 23.4–42.1 minutes: TFF is fastest among native runners; Flower is slowest on MNIST despite comparable accuracy.

Figure 3 shows per-round accuracy; all curves rise monotonically with no divergence. Figure 4 reports cross-entropy loss, which mirrors the accuracy ranking: Flower achieves the lowest final loss (0.0199) while remaining within 0.002 of all other frameworks. The near-overlap of reference-runner loss curves (≤0.002 spread at round 10) confirms that FedML, NVFlare, OpenFL, and PySyft execute the same underlying training loop on MNIST.

RQ1 (MNIST). Accuracy is framework-agnostic under IID MNIST (≤0.13 pp spread), but $T_{\mathrm{wall}}$ is not: Flower requires 1.8× the time of TFF (42.1 vs. 23.4 min). Reference runners occupy a narrow 27.8–28.6 min band. For 6G planners, this implies that orchestration overhead alone can dominate edge training schedules even when model quality is identical.

Reference runners transfer 39.9 MB upload and 39.9 MB download per round (∼798 MB over ten rounds), consistent with Equation (6) for $K=6$ and the MNIST CNN parameter size (Figure 9).

4.3. Phase 2: CIFAR-10

Table 6 and Figure 5Figure 6 report Phase 2 results. Accuracy spread widens to 3.90 percentage points (67.84–71.74%). TFF leads at 71.74%; PyTorch reference runners cluster near 68.2%. Native Flower (68.84%) aligns with references, not TFF.

TFF’s advantage appears from round 1 onward (46.05% vs. ∼38% for PyTorch runners) and persists in both accuracy and loss throughout training—suggesting a training-dynamics difference in the equivalent CNN rather than late-round aggregation effects. Loss at round 10 ranges from 0.823 (TFF) to 0.933 (OpenFL reference), correlating with the accuracy spread.

Wall times on CIFAR-10 converge to 22.7–24.2 minutes across frameworks, unlike MNIST where Flower required 42.1 minutes. One Flower CIFAR run (∼164 min wall time, 68.23% accuracy) is excluded as an operational outlier; we report the retained Flower run (24.2 min, 68.84%).

RQ2 (CIFAR-10). Framework rankings change relative to MNIST: TFF leads on accuracy (+2.90 pp over native Flower), while wall-time rankings collapse to a ∼1.5 min band. Phase 2 is therefore essential for framework discrimination in edge vision scenarios. Reference communication totals ∼281 MB over ten rounds (Figure 9), lower than MNIST despite richer inputs because the CIFAR CNN has fewer serialized parameters in our implementation.

4.4. Cross-Phase Analysis: Time, Communication, and Heatmap

Figure 7 compares $T_{\mathrm{wall}}$ across datasets. Flower is a pronounced MNIST outlier (42.1 min) but not on CIFAR-10 (24.2 min), indicating run-configuration sensitivity rather than a universal penalty. Figure 8 summarizes final accuracies side-by-side: the MNIST column is uniformly green (≥99.2%), whereas the CIFAR-10 column reveals TFF as the clear leader.

Figure 7. End-to-end wall time by framework and dataset. Hatched bars: native runners.

Figure 7. End-to-end wall time by framework and dataset. Hatched bars: native runners.

Figure 8. Final test accuracy (round 10) across datasets and frameworks.

Figure 8. Final test accuracy (round 10) across datasets and frameworks.

Figure 9. Total simulated FedAvg communication over ten rounds (reference runners only).

Figure 9. Total simulated FedAvg communication over ten rounds (reference runners only).

Figure 9 plots $C_{\mathrm{total}}$ for reference runners (RQ3). MNIST requires ∼2.8× more bytes than CIFAR-10 (798 vs. 281 MB) because the MNIST CNN yields larger ${\left|\mathbf{w}\right|}_{\mathrm{bytes}}$ in our PyTorch implementation. These values anchor future 6G profile experiments: under a 5 Mbps edge uplink, 281 MB alone implies at least ∼7.5 minutes of pure transfer time before compute is counted.

Table 7 summarizes answers to the research questions.

5. Discussion

The narrow MNIST accuracy band (±0.13 pp) and tight reference clustering on CIFAR-10 (∼68.2%) confirm consistent FedAvg (Equation (4)), partitioning, and seed 42 across runners. The TFF advantage on CIFAR-10 likely reflects TensorFlow/Keras optimization paths and numerical differences in the equivalent architecture [41]—not a different aggregation rule. Similar accuracy variance across deep-learning frameworks has been reported in centralized benchmarks [34].

For 6G edge deployment [4,8,20]: (1) use at least one non-trivial vision dataset before framework selection; (2) budget scheduling windows from measured $T_{\mathrm{wall}}$; (3) plan uplink capacity using $C_{\mathrm{total}}$ from Equation (6) before applying bandwidth-limited or mMTC profiles [2,9,20].

Four frameworks share one PyTorch loop; comparing FedML, NVFlare, OpenFL, and PySyft on accuracy or bytes tests the runner, not product SDKs [13,14,15,18]. Flower and TFF remain the primary native comparison until reference runners are replaced. Future work will evaluate secure aggregation overhead [28,29] and poisoning resilience [30,31].

5.1. Limitations

Single run per framework (no confidence intervals); ideal network only; IID partition; missing communication logs for native Flower/TFF; standalone simulation without distributed TLS/gRPC stress testing [28]; one hardware configuration; one excluded Flower CIFAR outlier. Broader FL benchmarks such as FedScale [34] and LEAF [33] address different axes (scale, realism) and are complementary to our framework-centric study.

6. Conclusions

We presented a documented two-phase comparison of six federated learning frameworks on MNIST and CIFAR-10 with six IID clients, ten FedAvg rounds (Equations (3)–(4)), and an ideal network profile on a Proxmox containerized testbed. MNIST confirms fairness (∼99.3% across frameworks) but hides stack differences. CIFAR-10 reveals a meaningful accuracy lead for TFF (71.74% vs. ∼68% for PyTorch runners) while wall times converge. Communication baselines (∼798 MB MNIST, ∼281 MB CIFAR-10) support upcoming 6G constraint and security studies [8,31] including non-IID data [24,25], radio profiles (eMBB, URLLC, mMTC) [4,5], and adversarial evaluations [30].