A Practical Implementation of Post-Quantum Cryptography for Secure Wireless Communication

1. Introduction

The development of quantum computing is advancing rapidly, transforming theoretical concepts into reality and raising concerns about the security of classical cryptographic systems. Among recent breakthroughs, Equal1 has launched Bell-1: The First Quantum System Purpose-Built for the HPC Era [1], complementing advancements by companies such as IBM and Microsoft. Recent studies, such as Esmailiyan et al.’s work on CMOS position-based charge qubits [2] and Staszewski et al.’s development of cryogenic controllers for quantum dots [3], provide critical insights into foundational technologies driving this revolution. It is anticipated that fully developed quantum computers will render classical cryptography insecure, potentially compromising all data encrypted using classical methods, including currently encrypted data. Therefore, addressing the imminent risks posed to traditional cryptographic algorithms has become increasingly urgent.

To address this issue and provide quantum-resistant cryptographic algorithms, the National Institute of Standards and Technology (NIST) introduced PQC algorithms [4,5,6] which will ensure that applications and network communications are secure against quantum attacks. However, it is not clear what impact or overhead migrating applications to be PQC compliant will introduce, as PQC algorithms come with large ciphertext and key sizes. It is important to evaluate the practical implementation of PQC algorithms in real-world applications. Therefore, there is a need to investigate the performance trade-offs associated with the practical use of PQC algorithms and to address the unique requirements of emerging technologies such as 5G/6G and IoT. Furthermore, it remains uncertain how to ensure and emphasize their stringent quality-of-service (QoS) requirements.

NIST has selected four PQC algorithms [7], and also released the standards for the selected PQC algorithms, which include one key encapsulation mechanism (KEM), CRYSTALS-KYBER (renamed to Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM) in the standards) [8], and three digital signature algorithms: CRYSTALS-Dilithium (Module-Lattice-Based Digital Signature Algorithm (ML-DSA)) [9], FALCON (Fast-Fourier Transform over NTRU-Lattice-Based Digital Signature Algorithm (FN-DSA)), whose draft will be released in FIP206, and SPHINCS+ (Stateless Hash-Based Digital Signature Algorithm (SLH-DSA)) [10]. In addition, as a backup to the ML-KEM, NIST selected HQC [11] for the next PQC KEM algorithm to be standardized, out of the four PQC algorithms in Round 4: BIKE, Classic McEliece, HQC, and SIKE. The draft standard for HQC is expected to be released later.

Additionally, while the NIST standard [8] provides detailed specifications for implementing the ML-KEM algorithm, including key generation, key encapsulation, and decapsulation, it does not finalize specific methods for securing application data. Instead, it recommends the application of established symmetric-key cryptographic techniques, as outlined in other NIST standards, for encrypting and decrypting data to protect against unauthorized access. Therefore, to fully understand the implications of integrating PQC algorithms into various applications, particularly those in IoT, 5G, and 6G systems with stringent QoS requirements, a dedicated PQC-based protocols for securing data transmission is essential.

This paper introduces new and improved PQC-based protocols that extend the PQC standalone and PQC-AES hybrid PKE protocols to secure the communication of application data transmitted between two parties and mitigate common cybersecurity attack vectors. Specifically, we introduce a sequence number-based mechanism to the PQC-based protocols to prevent replay attacks and man-in-the-middle (MITM), providing a practical application of PQC algorithms in real-world scenarios. This offers a tangible example of how these quantum-resistant cryptographic methods can be applied to secure communication and their impacts on applications.

In the PQC standalone mode, secure communication channel is established between sender and receiver through a PQC key encapsulation algorithm. The exchanged public key is used for encryption, while each party uses its secret key for decryption (asymmetric key cryptography). On the other hand, in the hybrid mode, a two-way shared secret key between the client and server is used to generate a symmetric key (symmetric key cryptography) as the encryption and decryption keys using a Key Derivation Function (KDF). The encryption and decryption keys are then used by the client and server to encrypt and decrypt their files.

As stated in our prior work [12], the client generates a pair of PQC public and private keys, and the server generates its own keys in a similar manner. Using the PQC KEM, the public key is exchanged, establishing secure communication. Subsequently, the client encrypts its file using the encryption key, based on the protocol being used (PQC standalone or PQC-AES hybrid). Upon receiving the encrypted file, the server decrypts it using the decryption key to access the content. This process ensures end-to-end secure communication between the two parties (i.e., the client and the server). Furthermore, we devise various use-case scenarios for implementing the proposed PQC-based protocols in real-world applications. To integrate these protocols with pre-existing applications, they are designed to mitigate replay and MITM attacks, enhancing connection security. This ensures a secure and seamless end-to-end communication between the two parties by using the process defined by the proposed protocols. This improvement demonstrates the feasibility of these protocols for real-world secure communications over wireless networks while minimizing their impact on application use cases and network performance. The goal of our proposed protocols is to enable the practical adoption of the PQC algorithm in real-world applications, balancing its effects on use cases and network performance. Our approach could also benefit 5G and 6G development by guiding the transition to PQC, particularly as there is increasing consideration of shifting 5G to a PKI-based trust model.

This paper significantly extends our prior work [12], where we proposed the first PQC-based and PQC-AES hybrid PKE protocols for wireless file transfer and conducted initial performance benchmarking. In this paper, we expand on that work by introducing a new framework that includes a sequence number-based mechanism to defend against replay and MITM attacks, along with an expanded security evaluation against common cybersecurity threats. Moreover, we extend the protocols to support a wider range of applications, including chat-based communication, video streaming, and live streaming. We also evaluate system performance under more diverse and realistic conditions and analyze the impact of these protocols on QoS and operational efficiency. This detailed analysis, along with the additional protocol advancements, provides critical insights beyond what was reported in [12] and offers practical guidance for deploying PQC in next-generation communication systems.

The rest of this paper is organized as follows. Section 2 reviews related work on PKE and PQC. Section 3 describes the proposed PQC-based protocols, while Section 4 details the use cases and implementation of the protocol in applications. In Section 5, we discuss the security analysis of the proposed PQC standalone and PQC-AES hybrid PKE protocols. Section 6 presents the performance evaluation of the proposed method, and Section 7 concludes the paper.

2. Related Work

There have been an extensive work and several ongoing work on the implementation of PQC algorithm and its variants. In this section, we review some of the existing works and explain the major difference between our paper and such research.

Numerous studies have explored the operation [13,14] and implementation of the PQC KEM algorithm across different hardware platforms. Additionally, most current implementations of the PQC algorithm have focused on its integration with Secure Sockets Layer (SSL), Transport Layer Security (TLS) [15,16,17], and Secure Shell (SSH). Furthermore, several works on PKE have been conducted, and there is ongoing work on the implementation of PQC algorithms and their variants.

In [18], Singh et al. propose a new multivariate public key cryptosystem based on permutation p-polynomials over finite fields, which focuses on the need for secure and efficient public key systems that can withstand increasing computing power and quantum computing. Marco et al. [19] propose a generic transformation that achieves post-quantum security for classical signature schemes by hiding the public key with a one-time use of the key pair. Meanwhile, da Silva Lima et al. [20] evaluated the efficiency of the Kyber KEM algorithm in a mobile application, specifically analyzing its performance on x86 and ARM architectures.

In addition to this, surveys and reviews that highlighted the issues and constraints of migrating applications to be PQC compliant have been carried out. Giron [21] discussed the challenges and research efforts required for migrating applications to PQC due to the vulnerability of current public-key cryptography schemes against quantum computers. Liu et al. [22] surveyed the performance and optimization of PQC algorithms for Internet of Things (IoT) systems. They highlighted the computational cost challenges of PQC algorithms for resource-constrained IoT devices and reviewed recent proposals for optimization. Similarly, Asif [23] presented a comprehensive review of PQC for IoT, covering both theoretical and practical aspects. Li et al. [24] reviewed PQC algorithms, focusing on the Kyber algorithm, and discussed the challenges and opportunities of post-quantum security. Balamurugan et al. [25] discussed the need for PQC in light of the threat quantum computing poses to classical encryption schemes.

Fakhruldeen et al. [26] provides a comprehensive introduction to the emerging field of quantum-resistant cryptography, emphasizing its critical importance in safeguarding wireless networks against quantum computing threats. In their paper, they highlights the vulnerabilities of current cryptographic standards to quantum attacks and explores solutions such as PQC and Quantum Key Distribution (QKD), alongside practical migration strategies like hybrid approaches. They further underscores the role of global standardization efforts, particularly by NIST, in advancing PQC algorithms, while addressing challenges in integrating these techniques into existing infrastructures.

Despite these varied approaches, the practical application of the PQC algorithm remains confined to the aforementioned protocols and has not expanded to other commonly used applications. Moreover, directly applying PQC algorithms to various applications and systems may lead to performance issues, as many of these have stringent QoS requirements. Therefore, to fully assess the impact of direct PQC algorithm usage on application scenarios within a wireless network, a protocol that allows a PQC algorithm or hybrid mode to be used for securing application data is essential.

3. PQC-Based PKE Protocols for Using PQC Algorithm in Application

According to the protocols described in [12], NIST recommends using the strongest algorithm parameters to ensure the effective use of PQC algorithms. The main reason for this is to avoid the cost and resources required to upgrade cryptographic implementations to a higher security level, as it is possible that upgrading may require modifications to the application/software where it will be needed. However, using stronger security parameters may have an adverse effect on application performance. Therefore, it is important to know the impact of using PQC algorithms in various applications before carefully designing an optimization method or selecting a PQC algorithm for data encryption and decryption in various applications. To determine the impact of using PQC algorithms on applications and various networks, a protocol to encrypt and decrypt data transferred over the network is necessary. Therefore, we propose PQC standalone and PQC-AES hybrid PKE protocols that assume safe key exchange between the client and the server using the PQC algorithm KEM.

3.1. PQC Standalone PKE Protocol

The KEM is often used in conjunction with a PKE scheme for securely exchanging encryption keys, which can then be used to encrypt data using the PKE scheme [28]. Therefore, in this paper, we selected PKE as the choice of cryptographic scheme for using PQC algorithms in various applications.

In the improved PQC standalone PKE protocol as shown in Algorithm 1, both the client (C) and the server (S) generate two pairs of keys using post-quantum cryptographic algorithms. These include a public/private (secret) key pair for KEM ($pk{k}_c$ and $sk{k}_c$ for the client, and $pk{k}_s$ and $sk{k}_s$ for the server), and a public/private (secret) key pair for signature ($pk{s}_c$ and $sk{s}_c$ for the client, and $pk{s}_s$ and $sk{s}_s$ for the server). The client begins by signing its public KEM key ($pk{k}_c$) with its private signature key ($sk{s}_c$) to create a digital certificate ($cert\left[pk{k}_c\right]$). This certificate, along with the client’s public signature key ($pk{s}_c$), is sent to the server to establish authenticity (Algorithm 1, lines 3–7).

Upon receiving the certificate and public signature key from the client, the server verifies the authenticity of the certificate using the client’s public signature key ($pk{s}_c$). If the verification is successful, it confirms that the client’s public KEM key ($pk{k}_c$) has not been tampered with. The server then performs encapsulation on the client’s public KEM key ($pk{k}_c$) to generate a shared secret ($s{s}_c$) and a ciphertext ($c{t}_c$). The server also signs its public KEM key ($pk{k}_s$) with its private signature key ($sk{s}_s$) to create its digital certificate ($cert\left[pk{k}_s\right]$). The server sends this certificate, along with its public signature key ($pk{s}_s$) and the ciphertext ($c{t}_c$), back to the client (Algorithm 1, lines 8–18).

The client receives the server’s certificate, public signature key, and ciphertext. It verifies the authenticity of the server’s certificate using the server’s public signature key ($pk{s}_s$). If this verification succeeds, it confirms that the server’s public KEM key ($pk{k}_s$) is legitimate. The client then decapsulates the received ciphertext ($c{t}_c$) using its private KEM key ($sk{k}_c$) to derive a shared secret ($s{s}_c^{\prime }$). Next, the client performs encapsulation on the server’s public KEM key ($pk{k}_s$) to generate another shared secret ($s{s}_s$) and a ciphertext ($c{t}_s$), which it sends back to the server (Algorithm 1, lines 19–30).

Algorithm 1 PQC standalone PKE Protocol

1: Input: Data to be transmitted ($dat{a}_c$ from Client, $dat{a}_s$ from Server)  2: Output: Securely exchanged data  3: Client (C):  4: Generate PQC KEM key pair: $(pk{k}_c,sk{k}_c)$  5: Generate PQC Signature key pair: $(pk{s}_c,sk{s}_c)$  6: $cert\left[pk{k}_c\right]\leftarrow \mathrm{Sign}(sk{s}_c,pk{k}_c)$  7: Send $(cert\left[pk{k}_c\right],pk{s}_c,pk{k}_c)$ to Server (S)  8: Server (S):  9: Generate PQC KEM key pair: $(pk{k}_s,sk{k}_s)$ 10: Generate PQC Signature key pair: $(pk{s}_s,sk{s}_s)$ 11: ${\mathrm{valid}}_c\leftarrow \mathrm{Verify}(pk{s}_c,cert\left[pk{k}_c\right],pk{k}_c)$ 12: if  ${\mathrm{valid}}_c=\mathrm{TRUE}$  then 13:     $(c{t}_c,s{s}_c)\leftarrow \mathrm{Encapsulate}\left(pk{k}_c\right)$ 14:     $cert\left[pk{k}_s\right]\leftarrow \mathrm{Sign}(sk{s}_s,pk{k}_s)$ 15:     Send $(cert\left[pk{k}_s\right],pk{s}_s,pk{k}_s,c{t}_c)$ to Client (C) 16: else 17:     Abort 18: end if 19: Client (C): 20: ${\mathrm{valid}}_s\leftarrow \mathrm{Verify}(pk{s}_s,cert\left[pk{k}_s\right],pk{k}_s)$ 21: if  ${\mathrm{valid}}_s=\mathrm{TRUE}$  then 22:     $s{s}_c^{\prime }\leftarrow \mathrm{Decapsulate}(sk{k}_c,c{t}_c)$ 23:     $(c{t}_s,s{s}_s)\leftarrow \mathrm{Encapsulate}\left(pk{k}_s\right)$ 24:     Send $c{t}_s$ to Server (S) 25:     $se{q}_c\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 26:     $ma{c}_c\leftarrow \mathrm{HMAC}-\mathrm{SHA}256(s{s}_c^{\prime }\left|\right|s{s}_s,\text{"}\mathrm{confirmation}\text{"}\left|\right|se{q}_c)$ 27:     Send $ma{c}_c$ to Server (S) 28: else 29:     Abort 30: end if 31: Server (S): 32: $s{s}_s^{\prime }\leftarrow \mathrm{Decapsulate}(sk{k}_s,c{t}_s)$ 33: $se{q}_s\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 34: $ma{c}_s\leftarrow \mathrm{HMAC}-\mathrm{SHA}256(s{s}_c\left|\right|s{s}_s^{\prime },\text{"}\mathrm{confirmation}\text{"}\left|\right|se{q}_s)$ 35: if  $ma{c}_c=\mathrm{HMAC}-\mathrm{SHA}256(s{s}_c\left|\right|s{s}_s^{\prime },\text{"}\mathrm{confirmation}\text{"}\left|\right|se{q}_c)$  then 36:     Send $ma{c}_s$ to Client(C) 37: else 38:     Abort 39: end if 40: Client(C): 41: if  $ma{c}_s=\mathrm{HMAC}-\mathrm{SHA}256(s{s}_c^{\prime }\left|\right|s{s}_s,\text{"}\mathrm{confirmation}\text{"}\left|\right|se{q}_s)$  then 42:     Secure communication is established. 43: else 44:     Abort 45: end if 46: Client-to-Server Data Transfer: 47: $c{t}_{dat{a}_c}\leftarrow \mathrm{PQC}-\mathrm{Encrypt}(pk{k}_s,dat{a}_c\left|\right|se{q}_c)$ 48: Send $c{t}_{dat{a}_c}$ to Server (S) 49: Server (S): 50: $dat{a}_c^{\prime }\left|\right|se{q}_c^{\prime }\leftarrow \mathrm{PQC}-\mathrm{Decrypt}(sk{k}_s,c{t}_{dat{a}_c})$ 51: if $se{q}_c^{\prime }$ is valid (not replayed) then 52:     Process $dat{a}_c^{\prime }$ 53: else 54:     Reject // Replay detected 55: end if 56: Server-to-Client Data Transfer: 57: $c{t}_{dat{a}_s}\leftarrow \mathrm{PQC}-\mathrm{Encrypt}(pk{k}_c,dat{a}_s\left|\right|se{q}_s)$ 58: Send $c{t}_{dat{a}_s}$ to Client (C) 59: Client (C): 60: $dat{a}_s^{\prime }\left|\right|se{q}_s^{\prime }\leftarrow \mathrm{PQC}-\mathrm{Decrypt}(sk{k}_c,c{t}_{dat{a}_s})$ 61: if $se{q}_s^{\prime }$ is valid then 62:     Process $dat{a}_s^{\prime }$ 63: else 64:     Reject 65: end if

To ensure that both parties have derived identical shared secrets and to prevent MITM attacks, a robust key confirmation process is employed. After deriving their respective shared secrets, both parties compute Message Authentication Codes (MACs) using “Hash-based Message Authentication Code using the Secure Hash Algorithm 256-bit (HMAC-SHA256)” over a combination of their shared secrets, a predefined “confirmation” string, and unique sequence numbers generated independently by each party. The MACs are exchanged between the client and server. After receiving a MAC, each party verifies that it matches its locally computed MAC. If both MACs match, it confirms that no MITM attack has occurred and that both parties share identical secrets (i.e., as shown for the client and the server in Algorithm 1, lines 31–45).

Once mutual confirmation is achieved, secure communication is established. For data transfer from the client to the server, the client encrypts the data ($dat{a}_c$) along with its sequence number ($se{q}_c$) using the server’s public KEM key ($pk{k}_s$). The resulting ciphertext ($c{t}_{dat{a}_c}$) is sent to the server (Algorithm 1, lines 46–48). The server receives and decrypts the ciphertext using its private KEM key ($sk{k}_s$), retrieves both the data and sequence number, and checks if the sequence number is valid (i.e., not replayed or out of order). If valid, it processes the data; otherwise, it rejects the data to prevent replay attacks (Algorithm 1, lines 49–55).

A similar process is followed for data transfer from the server back to the client. The server encrypts its data ($dat{a}_s$) along with its sequence number ($se{q}_s$) using the client’s public KEM key ($pk{k}_c$) to produce a ciphertext $c{t}_{dat{a}_s}$, which is sent to the client. Upon receiving this ciphertext, the client decrypts it using its private KEM key ($sk{k}_c$), retrieves both the data and the sequence number ($se{q}_s^{\prime }$), and checks if the sequence number is valid. If the sequence number is valid, the client processes the data; otherwise, it rejects the data to prevent replay attacks (Algorithm 1, lines 56–65).

The proposed protocol introduces various enhancements compared to the traditional PQC-based PKE protocols. Firstly, we incorporate explicit certificate validation during initial handshake steps to ensure that exchanged KEM public keys are authentic and untampered. Secondly, we strengthen the replay protection by including unique sequence numbers in data encryption. Finally, by integrating robust HMAC-based key confirmation mechanisms, our proposed protocol significantly reduces vulnerabilities to MITM attacks while ensuring that shared secrets are securely established between legitimate parties.

3.2. PQC-AES Hybrid PKE Protocol

To enhance security and performance, especially against evolving threats, we introduce a PQC-AES hybrid PKE protocol. This approach combines the quantum-resistance of PQC with the efficiency and established security of AES.

In this PQC-AES hybrid PKE protocol as shown in Algorithm 3, we build upon the key exchange process already outlined for the PQC standalone PKE protocol. Following a successful and authenticated key exchange between the client and the server using the steps defined in Algorithm 1 (lines 1–24), both parties then transition to utilizing AES encryption for subsequent data transfer. This approach leverages the best attributes of both cryptographic approaches: the quantum-resistant security offered by PQC algorithms during the initial key establishment and the speed and efficiency of AES for the bulk encryption of application data.

Algorithm 2 PQC-AES hybrid PKE Protocol

1: Input: Data to be transmitted ($dat{a}_c$ from Client, $dat{a}_s$ from Server)  2: Output: Securely exchanged data  3: Client (C) and Server (S):  4: Perform steps 1-24 of Algorithm 1 to establish shared secrets $s{s}_c$ and $s{s}_s$ (and confirm them).  5: Key Derivation (Both C and S):  6: $(K_1,K_2)\leftarrow \mathrm{HKDF}(s{s}_c\left|\right|s{s}_s,\mathrm{salt},\text{"}\mathrm{AES}\mathrm{keys}\text{"})$  7: Client-to-Server Data Transfer:  8: $i{v}_c\leftarrow \mathrm{GenerateInitializationVector}\left(\right)$  9: $se{q}_c\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 10: $c{t}_{dat{a}_c}\leftarrow \mathrm{AES}-\mathrm{Encrypt}(K_1,i{v}_c,dat{a}_c,\mathrm{mode},se{q}_c)$ 11: $ma{c}_{dat{a}_c}\leftarrow \mathrm{HMAC}-\mathrm{SHA}256(K_2,c{t}_{dat{a}_c}\left|\right|se{q}_c)$ 12: Send $(i{v}_c,c{t}_{dat{a}_c},ma{c}_{dat{a}_c})$ to Server (S) 13: Server (S): 14: if  $ma{c}_{dat{a}_c}=\mathrm{HMAC}-\mathrm{SHA}256(K_2,c{t}_{dat{a}_c}\left|\right|se{q}_c)$   then 15:     $dat{a}_c^{\prime }\leftarrow \mathrm{AES}-\mathrm{Decrypt}(K_1,i{v}_c,c{t}_{dat{a}_c},\mathrm{mode},se{q}_c)$ 16:     if $se{q}_c$ is valid (not a replay) AND Decryption successful then 17:         Process $dat{a}_c^{\prime }$ 18:     else 19:         Reject 20:     end if 21: else 22:     Abort 23: end if 24: Server-to-Client Data Transfer: 25: $i{v}_s\leftarrow \mathrm{GenerateInitializationVector}\left(\right)$ 26: $se{q}_s\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 27: $c{t}_{dat{a}_s}\leftarrow \mathrm{AES}-\mathrm{Encrypt}(K_1,i{v}_s,dat{a}_s,\mathrm{mode},se{q}_s)$ 28: $ma{c}_{dat{a}_s}\leftarrow \mathrm{HMAC}-\mathrm{SHA}256(K_2,c{t}_{dat{a}_s}\left|\right|se{q}_s)$ 29: Send $(i{v}_s,c{t}_{dat{a}_s},ma{c}_{dat{a}_s})$ to Client (C) 30: Client (C): 31: if  $ma{c}_{dat{a}_s}=\mathrm{HMAC}-\mathrm{SHA}256(K_2,c{t}_{dat{a}_s}\left|\right|se{q}_s)$  then 32:     $dat{a}_s^{\prime }\leftarrow \mathrm{AES}-\mathrm{Decrypt}(K_1,i{v}_s,c{t}_{dat{a}_s},\mathrm{mode},se{q}_s)$ 33:     if $se{q}_s$ is valid AND Decryption Successful then 34:         Process $dat{a}_s^{\prime }$ 35:     else 36:         Reject 37:     end if 38: else 39:     Abort 40: end if

After the successful establishment and confirmation of shared secrets $s{s}_c$ and $s{s}_s$, both the client and the server independently derive AES encryption and MAC keys. A KDF is applied to the concatenation of these shared secrets, along with a unique salt and information string, using the HMAC-based Extract-and-Expand Hash-based Message Authentication Code Key Derivation Function (HKDF) with SHA-256. This process yields two essential keys: $K_1$, designated for AES encryption, and $K_2$, purposed for HMAC-based message authentication (Algorithm 3, lines 5–6).

For data transfer from the client to the server, the client begins by generating a unique initialization vector, $i{v}_c$, and a sequence number, $se{q}_c$, to provide replay protection. Subsequently, the data, $dat{a}_c$, is encrypted using AES with the derived key $K_1$, the initialization vector $i{v}_c$, and incorporating the sequence number, $se{q}_c$, to form the ciphertext $c{t}_{dat{a}_c}$. Following this, the client computes a MAC over the ciphertext, $c{t}_{dat{a}_c}$, and the sequence number, $se{q}_c$, using HMAC-SHA-256 with the authentication key $K_2$, resulting in $ma{c}_{dat{a}_c}$. The client then sends the initialization vector $i{v}_c$, the ciphertext $c{t}_{dat{a}_c}$, and the MAC $ma{c}_{dat{a}_c}$ to the server (Algorithm 3, lines 7–12).

Upon receiving this data, the server begins by verifying the integrity and authenticity of the data. It computes its own MAC over the received ciphertext, $c{t}_{dat{a}_c}$, and the sequence number, $se{q}_c$, using the same HMAC-SHA-256 algorithm and the authentication key $K_2$, generating a server-side MAC. The server then compares the received client-side MAC, $ma{c}_{dat{a}_c}$, with its locally computed MAC. If these MACs match, the server proceeds with decryption, ensuring that the received data has not been tampered with during transmission and that it originates from the authenticated client. The server decrypts the ciphertext $c{t}_{dat{a}_c}$ using AES with the key $K_1$ and initialization vector $i{v}_c$, and after decryption, it also verifies the sequence number against previously received sequence numbers to detect any replay attacks. If the sequence number is valid and the decryption is successful, the server processes the received data $dat{a}_c^{\prime }$; otherwise, the server rejects the data to prevent potential security breaches (Algorithm 3, lines 13–23).

Similarly, the procedure is used for secure data transfer from the server to the client. The server also generates a unique $i{v}_S$ and uses $K_1$ to encrypt the data while applying HMAC to ensure authentication of the server by the client. The client also performs decryption in a similar way to ensure that the data is from the intended server (Algorithm 3, lines 24–40).

By combining the strengths of both PQC and AES, the protocol provides resilience against both classical and quantum attacks. Even if the PQC algorithm used for key exchange were to be compromised, AES encryption would still provide a layer of security. Furthermore, the hybrid approach can improve overall performance since AES encryption and decryption operations are generally faster and more efficient than PQC standalone encryption. Moreover, since AES-256 is a widely adopted and well-analyzed symmetric encryption algorithm, it has been demonstrated to be secure.

4. Practical Implementation of PQC-Based PKE Protocols

Since there is no function for data encryption and decryption in the current PQC algorithm implementation, we enhance the PQC algorithm implementation by adding encryption and decryption functions. This enables data encryption and decryption in both PQC standalone and the PQC-AES hybrid protocols. Both protocols utilize the PQC Key Encapsulation Mechanism (KEM) for secure key exchange between the client and server.

4.1. Post-Quantum Cryptography (PQC) Uses in Applications

To demonstrate the impact of PQC algorithm characteristics, PQC-based protocols need to be utilized in real-world applications. However, current implementations of PQC algorithms using PKE are limited to encrypting messages of a fixed size. For instance, CRYSTAL-KYBER, which is based on an IND-CPA-secure PKE scheme, and BIKE, which utilizes PKE from Quasi-Cyclic Moderate Density Parity-Check (QC-MDPC) codes, can only encrypt messages of a fixed length of 32 bytes. HQC, also based on IND-CPA-secure properties, can accept messages of a fixed length of 16 bytes before encoding.

Conversely, when a hybrid approach is employed, the KDF generates symmetric keys of a consistent length for all PQC algorithms. While AES cipher mode typically uses key sizes of 128, 192, or 256 bits (16, 24, or 32 bytes), it does not support key sizes larger than 256 bits. Therefore, to use the PQC standalone PKE directly for the encryption and decryption of data that exceeds these fixed sizes without the hybrid implementation, it is necessary to segment the message and reassemble the decrypted segments to reconstruct the original data. The process for this splitting is shown in Figure 1. In this instance, first, the read-in data is split into blocks of an acceptable size as shown in Figure 1a. Each block is then encrypted sequentially using the receiver’s public key. The encrypted blocks are sent to the intended party. As shown in Figure 1b, on the receiver side, after the data is received, the encrypted data is processed as a stream. Each block is decrypted using the receiver’s private key and then aggregated to reconstruct the original data.

4.1.1. Real-World Scenario Use Cases for Proposed PQC-Based Protocols

Consider a real-world scenario of using our PQC-based protocols in real applications as shown in Figure 2: two devices representing the client and server, with the necessary software installed, including PQC and AES libraries. For the application scenario, we consider four use case scenarios: (i) File transfer, (ii) Chat, (iii) Video streaming, and (iv) Live streaming. Assuming that the client is Alice while the server is Bob, for Alice to send confidential data to Bob using any of the use case scenarios and a PQC-based system to ensure the data remains secure against potential future quantum computer attacks, the following process is used:

• Bob generates a pair of keys using a PQC algorithm: a public key ($pk{k}_b$) and a private key ($sk{k}_b$).
• Bob shares his public key ($pk{k}_b$) with Alice while keeping the private key ($sk{k}_b$) secret.
• Alice prepares the data for transmission and splits it into multiple blocks if it exceeds the maximum byte size allowed by the PQC algorithm as shown in Figure 1a.
• Alice encrypts each block sequentially using Bob’s public key ($pk{k}_b$). The encrypted blocks are aggregated into a single encrypted file.
• Alice sends the encrypted file through a secure network.
• The access point routes the encrypted file to Bob without being able to decrypt it.
• Bob receives the encrypted file and uses his private key ($sk{k}_b$) to decrypt each block.
• He reassembles the decrypted blocks to reconstruct the original document as shown in Figure 1b.

Using PQC-based protocols, Alice and Bob can ensure that their communication remains secure against threats posed by quantum computing. The proposed PQC-based protocols workflow described above can be applied to various real-world scenarios. For file transfers, the protocol ensures the secure transmission and reconstruction of files divided into encrypted blocks. In chat applications, messages can be encrypted as individual small blocks to ensure confidentiality in real-time conversations. For video streaming, secure pre-encryption of video segments guarantees that the data remains protected during transmission. Lastly, live streaming scenarios benefit from the protocol by encrypting and transmitting content in near real-time, with a focus on maintaining low latency to preserve the streaming quality. These use cases demonstrate the versatility of the PQC-based protocols in addressing diverse application requirements while securing data against quantum threats.

5. Security Analysis of the Proposed Protocols

This section presents a detailed security analysis of the proposed PQC standalone and PQC-AES hybrid PKE protocols. We evaluate the resilience of these protocols against various attack vectors, considering both classical and post-quantum threats.

5.1. Security Analysis of the PQC Standalone PKE Protocol

The PQC standalone PKE protocol relies entirely on the security of the chosen PQC algorithms for both key exchange (KEM) and data encryption. We analyze its strengths and weaknesses, detailing the assumptions underlying its security.

5.1.1. Key Exchange (PQC KEM)

The key exchange phase uses a post-quantum secure KEM, such as CRYSTALS-KYBER, standardized by NIST.

Security Assumption

The security of the key exchange depends on the computational infeasibility of solving the underlying hard mathematical problem (e.g., Module-LWE problem) for quantum adversaries. This is based on cryptanalysis during the NIST PQC standardization process.

Formal Statement

The confidentiality of the shared secret rests on the IND-CCA2 security of the chosen KEM in the quantum random oracle model. If an adversary solves the Module-LWE problem or breaks the IND-CCA2 security of the KEM, the confidentiality of the shared secret is compromised.

5.1.2. Key Confirmation

Key confirmation prevents MITM attacks, ensuring both parties derive the same shared secret.

Mechanism

After decapsulation, parties derive a shared secret using a KDF. To confirm the shared secret, each party computes and exchanges HMACs (using HMAC-SHA-256) over a pre-defined data structure including nonces or session identifiers. The MACs are verified to ensure that no MITM attacker has replaced either public key.

Security Assumption

The security relies on:

• The collision resistance of SHA-256.
• The pseudorandomness and unforgeability properties of HMAC.

Formal Statement

If SHA-256 is collision-resistant and HMAC is unforgeable under chosen-message attacks, an adversary cannot impersonate either party or inject their own public key into the key exchange process.

5.1.3. Data Encryption and Decryption (PQC)

The PQC standalone PKE protocol directly encrypts data using a public-key encryption scheme derived from the chosen KEM.

Security Assumption

The confidentiality of encrypted data depends on the post-quantum security of the public-key encryption scheme derived from the KEM, including resistance to chosen-ciphertext attacks (IND-CCA2 security).

Formal Statement

The confidentiality of transmitted data is guaranteed if:

• The underlying KEM is IND-CCA2 secure.
• The public-key encryption scheme derived from it inherits this IND-CCA2 security property.

If an adversary breaks these assumptions, they can recover plaintext data from ciphertexts.

5.1.4. Replay Protection

Replay attacks involve an attacker capturing and retransmitting a valid encrypted message to disrupt communication or impersonate a legitimate sender.

Mechanism

To mitigate replay attacks:

• Sequence numbers are incorporated into each encrypted message.
• The receiver tracks these sequence numbers to ensure they are unique and monotonically increasing.
• Messages with duplicate or out-of-order sequence numbers are rejected.

Security Assumption

This approach assumes sequence numbers are generated uniquely and managed correctly by both parties during communication.

Formal Statement

If sequence numbers are generated uniquely and monotonically increasing, replayed messages will be detected and rejected, preventing replay attacks.

5.1.5. Overall Security

The overall security of this protocol is determined by its reliance on post-quantum cryptographic primitives for both key exchange and data encryption.

Advantages

• Post-Quantum Security: Provides resistance against quantum adversaries due to reliance on PQC algorithms.
• Simplified Design: Avoids additional complexity introduced by hybrid schemes.

Limitations

• Single Point of Failure: The protocol’s security entirely depends on the strength of the PQC algorithms used (both KEM and public-key encryption). If these algorithms are compromised, both key exchange and data confidentiality are at risk.
• Performance Overhead: PQC standalone encryption/decryption operations are computationally intensive compared to symmetric cryptography like AES-256.

Formal Statement

The overall security of this protocol is bounded by the security assumptions of its underlying PQC algorithms: If an adversary breaks the post-quantum hardness assumptions (e.g., Module-LWE problem), they can compromise both key exchange and data confidentiality. Unlike hybrid approaches, breaking one component compromises overall security.

5.2. Security Analysis of the PQC-AES Hybrid PKE Protocol

The hybrid protocol leverages the strengths of both post-quantum key exchange (using a KEM like CRYSTALS-KYBER) and a well-established symmetric cipher (AES-256). This defense-in-depth approach enhances overall security.

5.2.1. Security of the Key Exchange (PQC KEM)

The key exchange phase is identical to that described for the PQC standalone PKE protocol (Section 5.1.1). The same security assumptions and formal statements are applied.

5.2.2. Key Confirmation

Key confirmation is performed as described for the PQC standalone PKE protocol (Section 5.1.2) to prevent MITM attacks.

5.2.3. Security of Data Encryption (AES-256)

Data encryption uses AES-256, which is well-established and understood.

Security Assumption

AES-256 is considered secure against classical attacks. The hybrid approach aims for defense-in-depth; even if the PQC KEM is broken, the attacker still faces the challenge of breaking AES-256.

Formal Statement

The data encryption phase utilizes AES-256 with a key size of 256 bits. AES-256 is widely considered secure against classical computational attacks, providing 256 bits of security. We assume that the AES-256 implementation is free from side-channel vulnerabilities.

5.2.4. Overall Security Argument

The hybrid protocol combines the strengths of both PQC and AES-256.

Formal Statement

The overall security of the hybrid protocol is bounded by the security of its weakest component. By combining a post-quantum KEM with AES-256, we achieve a defense-in-depth strategy. An attacker would need to break both the PQC KEM and AES-256 to compromise the confidentiality of the data. This provides a higher level of assurance against both classical and potential future quantum attacks. Since PQC encryption/decryption algorithms are often significantly slower than AES-256, the hybrid approach leverages the best aspects of both: post-quantum key exchange and fast symmetric encryption.

5.3. Robustness Against Common Attack Vectors

To demonstrate the robustness of our proposed protocols, we consider their resilience against several standard cybersecurity attack vectors.

5.3.1. Eavesdropping

Eavesdropping is the primary threat addressed by encryption. Both protocols provide protection against eavesdropping, although with different security assumptions.

PQC Standalone PKE

The PQC standalone PKE protocol protects against eavesdropping by encrypting data using the receiver’s public key, which only the receiver, possessing the corresponding private key, can decrypt. The security relies on the post-quantum security of the underlying PQC algorithm.

PQC-AES Hybrid PKE

The hybrid protocol provides two layers of protection against eavesdropping. First, the AES-256 key is exchanged securely using the PQC KEM. Second, the data itself is encrypted with AES-256. An eavesdropper would need to break both the PQC KEM and AES-256 to recover the plaintext.

5.3.2. Man-in-the-Middle (MITM) Attacks

Both protocols include a key confirmation step using HMAC-SHA-256 to prevent MITM attacks, ensuring that both parties have derived the same shared secret and that an attacker has not successfully impersonated either party during the key exchange.

5.3.3. Replay Attacks

Replay attacks are mitigated differently depending on the AES mode used.

PQC-AES Hybrid PKE (with AES-GCM)

When using AES-GCM, the GCM mode inherently provides protection against replay attacks. The authentication tag incorporates the Initialization Vector (IV) and ciphertext, so replaying a message with a different IV (which should be unique for each message) will result in an authentication failure.

PQC-AES Hybrid PKE (with AES-CBC or Other Modes)

To mitigate replay attacks when not using GCM, we incorporate sequence numbers or timestamps within the HMAC calculation. The receiver tracks these sequence numbers/timestamps and rejects any messages that are out of order or outside an acceptable time window.

5.3.4. Chosen-Ciphertext Attacks (CCA)

This is relevant to the PQC KEM used in both protocols. NIST’s selections are generally CCA-secure. The chosen PQC KEM (e.g., CRYSTAL-KYBER) is designed to be IND-CCA2 secure, meaning it is resistant to chosen-ciphertext attacks. This ensures that an attacker cannot gain information about the plaintext even if they can choose ciphertexts to be decrypted.

5.3.5. Side-Channel Attacks

Side-channel attacks exploit physical information leaked during computation (power consumption, timing, etc.). While our current evaluation does not directly address side-channel attacks, we acknowledge their importance, particularly for implementations on embedded devices. Mitigation strategies against side-channel attacks, such as constant-time implementations and masking techniques, are crucial for real-world deployments and are a subject of ongoing research in the PQC community. We recommend using side-channel resistant implementations of the chosen PQC algorithms.

5.3.6. Known-Plaintext Attacks

If an attacker somehow knows some of the plaintext and tries to use this to break the encryption, AES-256 and PQC KEMs are designed to be resistant to known-plaintext attacks.

6. Evaluation

In this section, we conduct a performance evaluation of the proposed protocols through experiments with PQC algorithms. First, we enhance the PQC algorithm implementation by adding encryption and decryption functions, enabling data encryption and decryption using both PQC standalone algorithms and the PQC-AES hybrid protocols. Second, we implement our proposed protocols (i.e., PQC standalone and PQC-AES hybrid PKE protocols). Furthermore, we measure the time required to generate PQC KEM keys, PQC signature keys, perform encapsulation, decapsulation, file encryption, and decryption, among other operations. We use our extended version (i.e., our version can now handle data encryption and decryption) of the PQC algorithm API implementation provided by Open Quantum Safe (Liboqs) [29], which is a collection of all the PQC algorithms provided directly by NIST as one standard library.

6.1. Experiment Settings

In our implementation, we describe four use cases in Section 4.1.1. However, we use the file transfer application scenario to evaluate our protocols. The experiment is conducted on two separate machines: one with an Intel® Core™ i7-10510U CPU running at a base frequency of 1.80 GHz × 8 and 15.5 GiB of memory, running Ubuntu 20.04 LTS desktop, which serves as the server; and another with an Intel® Core™ i7-10510U CPU running at a base frequency of 1.80 GHz × 8 and 15.5 GiB of memory, running Ubuntu 20.04 LTS desktop, which is used as the client system. Table 1 presents the PC settings and the configuration of the systems used.

Furthermore, the standardized algorithms by NIST were considered, including KEM algorithms CRYSTAL-KYBER – ML-KEM-512, ML-KEM-768, and ML-KEM-1024 (referred to as K512, K768, and K1024 hereafter), as well as alternative KEM algorithms BIKE (Round 4) – BIKE-L1, BIKE-L3, and BIKE-L5, and HQC - HQC-128, HQC-192, and HQC-256. For the signature algorithms, FALCON (Round 3) - FN-DSA-512 and FN-DSA-1024 (referred to as F512 and F1024 hereafter) were also evaluated. To evaluate the proposed protocols and the impact of PQC algorithms on applications, we used client-server file transfer communication between two different systems using Python socket programming with Transport Control Protocol/User Datagram protocol (TCP/UDP protocols). The client encrypts a file and sends it to the server. TCP traffic generated from the socket programming was sent from the PC2 machine, acting as the client, and received by the PC1 machine, serving as the server. The traffic data was transmitted using IEEE 802.11n wireless LAN in the 2.4 GHz band. The traffic between the client and server was captured using Wireshark [30] from the server system. The experiment setup is shown in Figure 3.

Additionally, we implemented conventional cryptography (i.e., AES-256) to compare and highlight the impact of the PQC algorithm on file transfer. The goal of our evaluation is to facilitate the practical use of the PQC algorithm in real applications and to determine the impact of using the PQC algorithm in the application use case and network. We consider two main evaluation scenarios: (i) the PQC algorithm performance—to determine how fast it takes to generate PQC algorithm keys, complete the KEM process (encapsulation and decapsulation) (i.e., non-pre-distributed key agreement process), apply digital signatures (signing and verification), and perform encryption and decryption; and (ii) the impact of the PQC algorithm on file transfer applications—to determine which PQC algorithm will be more suitable for real-time applications without extensive overhead. For this, we measure the processing time for encrypted data transfer, throughput, and round-trip time (RTT).

6.2. Results and Discussion

6.2.1. PQC Algorithm Key Generation Performance Results

In order to evaluate the performance implications of integrating PQC algorithms, we measured the average key generation time (Avg. Key Gen.), key encapsulation time (Avg. Encap.), and key decapsulation time (Avg. Decap.) for PQC KEM algorithms, as well as the signature key generation time (Sig. Gen. Time), signing time, and verification time for PQC signature algorithms across client and server systems. Table 2 and Table 3 show the average time it takes to generate PQC algorithm keys (PQC KEM and signature keys), encapsulate and decapsulate the KEM key, and sign and verify PQC signatures on the client and server systems.

For the KEM algorithms, Table 2 shows that for BIKE KEM at all security levels, BIKE-L1 achieved the fastest average key generation time on the client system at $0.43\mathrm{ms}$ while BIKE-L5 required $1.95\mathrm{ms}$, reflecting a progressive increase in computational demand with higher security levels. Conversely, server-side key generation times for BIKE-L1, BIKE-L3, and BIKE-L5 were $1.23\mathrm{ms}$, $2.63\mathrm{ms}$, and $5.05\mathrm{ms}$, respectively, demonstrating a more pronounced rise compared to client-side metrics. Encapsulation and decapsulation times also increased with security levels, particularly for BIKE and HQC algorithms.

Furthermore, as shown in the table, BIKE-L5 decapsulation required $14.42\mathrm{ms}$ on the client system and $15.94\mathrm{ms}$ on the server system, while HQC-256 decapsulation reached $49.77\mathrm{ms}$ (client) and $53.40\mathrm{ms}$ (server). Additionally, CRYSTAL-KYBER algorithms diverged from this trend: K768 achieved the fastest client-side key generation at $0.19\mathrm{ms}$, surpassing both K512 ($0.92\mathrm{ms}$) and K1024 ($1.00\mathrm{ms}$). On the Server-side, CRYSTAL-KYBER results showed K1024 as the most efficient for key generation ($0.49\mathrm{ms}$), encapsulation ($0.09\mathrm{ms}$), and decapsulation ($0.11\mathrm{ms}$), with K512 and K768 achieving marginally higher times.

Furthermore, Table 3 shows the performance of PQC signature algorithms. According to the results, F512 required $7.01\mathrm{ms}$ for key generation on the client system, significantly faster than F1024’s $19.00\mathrm{ms}$. A similar trend was observed on the server system, with F512 and F1024 requiring $9.68\mathrm{ms}$ and $24.51\mathrm{ms}$, respectively. Signing times for F512 were $0.33\mathrm{ms}$ (client) and $0.37\mathrm{ms}$ (server), whereas F1024 took $1.43\mathrm{ms}$ (client) and $0.67\mathrm{ms}$ (server), illustrating a reversal in efficiency favoring the server for higher security levels. Verification times remained consistent for F512 across systems ($0.28\mathrm{ms}$ client vs. $0.82\mathrm{ms}$ server) but diverged slightly for F1024 ($0.31\mathrm{ms}$ client against $0.79\mathrm{ms}$ server). These results underscore the trade-offs between security levels and computational overhead, particularly for server-side operations.

6.2.2. PQC Algorithm Encryption and Decryption Performance Results

In the hybrid protocol, AES-256 was used for data encryption/decryption. It is expected that the time for these operations should remain consistently low, meaning it will not vary significantly with the chosen PQC KEM, as the underlying symmetric cipher remains the same. To focus on the computational differences introduced by the PQC algorithms themselves, the subsequent figures present the encryption and decryption times specifically for the PQC-only protocol. Therefore, we evaluated the data encryption and decryption times for the PQC KEM algorithms. The encryption process was executed on the client system, while decryption was performed on the server system.

Figure 4 shows the total encryption times for BIKE, HQC, and CRYSTAL-KYBER algorithms when processing files of sizes $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$. For BIKE-L1, as shown in Figure 4a, encryption times remained stable across file sizes, with $0.20\mathrm{s}$ for $1\mathrm{KB}$, $0.198\mathrm{s}$ for $10\mathrm{KB}$, and $0.148\mathrm{s}$ for $100\mathrm{KB}$. However, BIKE-L3 exhibited higher encryption times for larger files, increasing from $0.47\mathrm{s}$ ($1\mathrm{KB}$) to $0.63\mathrm{s}$ ($10\mathrm{KB}$) before declining to $0.46\mathrm{s}$ ($100\mathrm{KB}$), while BIKE-L5 ranged from $0.58\mathrm{s}$ ($1\mathrm{KB}$) to $0.93\mathrm{s}$ ($10\mathrm{KB}$) and $0.82\mathrm{s}$ ($100\mathrm{KB}$).

Conversely, HQC encryption times scaled significantly with both security levels and file sizes as shown in Figure 4b: HQC-128 averaged $6.13\mathrm{s}$ ($1\mathrm{KB}$), $5.96\mathrm{s}$ ($10\mathrm{KB}$), and $5.63\mathrm{s}$ ($100\mathrm{KB}$), whereas HQC-256 required $32.82\mathrm{s}$ ($1\mathrm{KB}$), $37.04\mathrm{s}$ ($10\mathrm{KB}$), and $37.54\mathrm{s}$ ($100\mathrm{KB}$). CRYSTAL-KYBER algorithms, as shown in Figure 4c, demonstrated modest variations, with K768 achieving the fastest $1\mathrm{KB}$ encryption at $0.10\mathrm{s}$, outperforming K512 ($0.15\mathrm{s}$) and K1024 ($0.13\mathrm{s}$). For $100\mathrm{KB}$ files, CRYSTAL-KYBER encryption times remained low compared to other algorithms, ranging from $0.10\mathrm{s}$ (K768) to $0.17\mathrm{s}$ (K1024).

Figure 5 shows the decryption times on the server system. BIKE-L1 decryption times increased sharply with file size, rising from $0.04\mathrm{s}$ ($1\mathrm{KB}$) to $2.89\mathrm{s}$ ($100\mathrm{KB}$), while BIKE-L5 shows the same trend from $0.29\mathrm{s}$ ($1\mathrm{KB}$) to $27.60\mathrm{s}$ ($100\mathrm{KB}$) as shown in Figure 5a. Figure 5b shows that HQC followed a similar pattern: HQC-128 decryption required $13.52\mathrm{s}$ for $100\mathrm{KB}$ files, whereas HQC-256 reached $71.31\mathrm{s}$ for the same file size.

In contrast, Figure 5c shows that the results of CRYSTAL-KYBER algorithms maintained consistently low decryption times, with K512 requiring $0.17\mathrm{s}$ and K1024 requiring $0.24\mathrm{s}$ for $100\mathrm{KB}$ files. Notably, K768’s decryption time decreased slightly from $0.17\mathrm{s}$ ($10\mathrm{KB}$) to $0.17\mathrm{s}$ ($100\mathrm{KB}$), deviating from the upward trend observed in BIKE and HQC. Smaller file sizes ($1-10\mathrm{KB}$) exhibited minimal decryption times for all algorithms, with K512 and K768 achieving $0.03\mathrm{s}$ performance. These results reaffirm the computational efficiency of lattice-based algorithms like CRYSTAL-KYBER for both encryption and decryption, particularly when handling larger file sizes, compared to code-based (BIKE) and hash-based (HQC) alternatives.

The results confirm the computational overhead of PQC algorithms compared to conventional AES-256, as shown in Figure 4 and Figure 5. In comparison, for encryption, AES-256 shows consistently lower times across all file sizes, with $0.024\mathrm{s}$ for $1\mathrm{KB}$, $0.019\mathrm{s}$ for $10\mathrm{KB}$, and $0.024\mathrm{s}$ for $100\mathrm{KB}$, significantly outperforming the PQC algorithms. In the PQC standalone algorithm, the encryption time increased as the security level increases, with the exception of CRYSTAL-KYBER. The CRYSTAL-KYBER algorithm at all security levels has the lowest encryption time for all file sizes, as shown in Figure 4c. The fastest PQC algorithm in terms of encryption time is K768, which requires $0.10\mathrm{s}$ for $1\mathrm{KB}$, exceeding AES-256’s time by approximately 4.1 times.

Similarly, Figure 5 shows that the decryption times for AES-256 remained stable across the file sizes, with $0.053\mathrm{s}$ ($1\mathrm{KB}$), $0.024\mathrm{s}$ ($10\mathrm{KB}$), and $0.023\mathrm{s}$ ($100\mathrm{KB}$), which is in contrast with BIKE-L5’s $27.60\mathrm{s}$ and HQC-256’s $71.31\mathrm{s}$ for $100\mathrm{KB}$ files, as shown in Figure 5a,b. CRYSTAL-KYBER algorithms demonstrated closer alignment with AES-256, as shown in Figure 5c, with K768 decryption times of $0.17\mathrm{s}$ for $100\mathrm{KB}$ files, yet still exceeding AES-256 by 7.3 times. These findings underscore the persistent efficiency gap between conventional and post-quantum algorithms, emphasizing the trade-offs between quantum-resistant security and computational performance.

6.2.3. PQC Algorithm Impact on File Transfer Performance Results

To demonstrate the impact of PQC algorithms on various applications, we evaluated the proposed protocols using a file transfer application. We measured the overall processing time, throughput, and RTT as metrics. Since PQC algorithms can be more computationally intensive than traditional algorithms, the requirements to secure the data may adversely affect the performance of the applications and the network’s data handling capabilities. Therefore, file transfer was selected as the use case to test the application of the proposed protocols.

In our evaluation, we paired F512 and F1024 with each PQC KEM algorithm (the KEM algorithm paired with F512 and F1024 signature algorithms is denoted as: BIKE-L1-F512, BIKE-L3-F512, BIKE-L5-F512, BIKE-L1-F1024, BIKE-L3-F1024, BIKE-L5-F1024, HQC-128-F512, HQC-192-F512, HQC-256-F512, HQC-128-F1024, HQC-192-F1024, HQC-256-F1024, K512-F512, K768-F512, K1024-F512, K512-F1024, K768-F1024, and K1024-F1024) to perform the key exchange process for each algorithm in our proposed protocols. We measured the processing time, throughput, and RTT efficiency of the proposed protocols using various paired KEM and signature PQC algorithms during data transfer across a wireless LAN environment and compared the measured metrics achieved by different PQC algorithms for data sizes of $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$. The results of the data transfer between the client system and the server (this includes a key agreement (i.e., the PQC public keys of both parties are exchanged securely) and the actual file transfer) are shown in Figure 6, Figure 7 and Figure 8.

Processing time:

The overall processing times on the server system, segmented into $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$ file sizes, highlight the performance disparities between PQC-AES hybrid implementations, PQC standalone methods, and conventional AES-256. For $1\mathrm{KB}$ files shown in Figure 6a, the hybrid BIKE-L1-F512 protocol recorded an average processing time of $1.56\mathrm{s}$, marginally faster than its PQC standalone counterpart ($1.64\mathrm{s}$), while AES-256 achieved $2.06\mathrm{s}$, demonstrating comparable efficiency to hybrid approaches. Conversely, according to the results in Figure 6b, it was confirmed that HQC-256-F512 in PQC standalone mode required $2.79\mathrm{s}$, exceeding AES-256 by 35%, whereas its hybrid variant reduced this to $1.56\mathrm{s}$. For CRYSTAL-KYBER, K512-F512 exhibited inverse behavior, with hybrid processing at $1.73\mathrm{s}$ outperformed by PQC standalone ($1.48\mathrm{s}$), suggesting algorithm-specific optimization variances.

Figure 6c shows the processing time for $10\mathrm{KB}$ files. Hybrid methods generally retained advantages: BIKE-L5-F512 hybrid processing required $1.48\mathrm{s}$, significantly lower than its PQC standalone time of $6.08\mathrm{s}$, while AES-256 maintained $1.82\mathrm{s}$. HQC-256-F512’s PQC standalone time, shown in Figure 6d, surged to $16.78\mathrm{s}$, 9.2 times slower than AES-256, whereas its hybrid implementation reduced this to $2.11\mathrm{s}$. In comparison, K1024-F512 hybrid processing ($2.73\mathrm{s}$) was slightly lower than the PQC standalone mode variant ($1.55\mathrm{s}$), though both remained within $1.5-2.7\mathrm{s}$, aligning closer to AES-256’s efficiency.

Similarly, the $100\mathrm{KB}$ file processing time shows the scalability challenges of PQC standalone methods. As shown in Figure 6f, BIKE-L5-F512 PQC standalone processing reached $32.02\mathrm{s}$, 13.5 times slower than AES-256’s $2.37\mathrm{s}$, while its hybrid variant, as shown in Figure 6e, achieved $1.62\mathrm{s}$, outperforming AES-256 by 34%. HQC-256-F512’s PQC standalone time escalated to $119.18\mathrm{s}$, 50.3 times higher than AES-256, whereas hybrid modes like HQC-128-F512 ($1.77\mathrm{s}$) and K768-F512 ($2.20\mathrm{s}$) demonstrated competitive or superior performance relative to AES-256. These results emphasize hybrid protocols’ potential to mitigate PQC overheads, particularly for larger data volumes, while reaffirming AES-256’s fixed efficiency for mid-sized operations.

Throughput:

Furthermore, the server-side throughput results shown in Figure 7, categorized by $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$ file sizes, illustrate the interplay between ciphertext dimensions and quantum-resistant algorithm performance relative to AES-256. Figure 7a shows the $1\mathrm{KB}$ data; it was confirmed that AES-256 achieved a throughput of $0.170\mathrm{Mbps}$, while PQC standalone methods with larger ciphertexts, such as HQC-128-F1024, reached $7.841\mathrm{Mbps}$, a 46.1-fold increase. This variation arises not from superior efficiency but from expanded ciphertext sizes increasing throughput metrics. It was observed in the hybrid implementations that the CRYSTAL-KYBER algorithm paired with the FALCON digital signature achieved throughput results comparable to AES-256. The results show total throughput values of K512-F512 ($0.476\mathrm{Mbps}$), K768-F512 ($0.517\mathrm{Mbps}$), K512-F1024 ($0.554\mathrm{Mbps}$), and K768-F1024 ($0.647\mathrm{Mbps}$).

Additionally, Figure 7b shows that at $10\mathrm{KB}$, AES-256 throughput dropped to $0.0035\mathrm{Mbps}$, whereas PQC standalone total throughput increased. K1024-F512 achieved a total throughput of $19.516\mathrm{Mbps}$, reflecting the compounding effect of ciphertext expansion on throughput. BIKE-L1-F512 followed this trend at $10.578\mathrm{Mbps}$, though its larger ciphertexts inherently amplify throughput values despite slower absolute processing speeds. For the hybrid modes, HQC-256-F512 showed a total throughput of $2.005\mathrm{Mbps}$, 565 times higher than that of AES-256. This confirms the higher computational latency of the PQC standalone algorithm compared to the conventional cryptographic algorithm, underscoring the metric’s limitations in isolation.

For $100\mathrm{KB}$ files shown in Figure 7c, the AES-256 total throughput further increased to $9.725\mathrm{Mbps}$, while PQC standalone K768-F1024 also increased to $33.320\mathrm{Mbps}$, 3.4 times higher than AES-256. This disparity arises from the file size and ciphertext increase, as PQC algorithms produce larger ciphertexts than AES-256. BIKE-L1-F1024 achieved $11.968\mathrm{Mbps}$ in PQC standalone mode, while its hybrid variant ($7.459\mathrm{Mbps}$) lagged, emphasizing the trade-offs between post-quantum security and protocol complexity. Notably, BIKE-L5-F512’s PQC standalone mode achieved a throughput ($4.476\mathrm{Mbps}$) lower than AES-256 by a factor of 1.9, despite its larger ciphertexts, highlighting vulnerabilities in the scalability of code-based algorithms. These results underscore that while PQC algorithms may exhibit higher throughput due to ciphertext expansion, this metric alone does not capture their operational efficiency or suitability for latency-sensitive applications. The results demonstrate that although larger ciphertext sizes in PQC algorithms result in higher throughput compared to AES-256, this does not directly imply greater efficiency.

Round-trip-time:

The RTT measurements for $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$ file sizes in Figure 8 reflect the operational latency introduced by ciphertext expansion and protocol overhead in PQC algorithms, despite their higher throughput metrics. Figure 8a shows the results for $1\mathrm{KB}$ files: AES-256 recorded an RTT of $0.0115\mathrm{s}$, while multiple PQC standalone protocol algorithms achieved slightly lower latencies in comparison. HQC-128-F1024 achieved $0.0022\mathrm{s}$, 5.2 times faster than AES-256, and K512-F512 achieved $0.0024\mathrm{s}$. In the hybrid implementations, however, the algorithm pairs incurred slightly lower latency than AES-256, while K1024-F512 ($0.0115\mathrm{s}$) achieved similar latency, underscoring the handshake overhead inherent to hybrid protocols.

At $10\mathrm{KB}$, as shown in Figure 8b, AES-256’s RTT decreased to $0.0063\mathrm{s}$, while PQC standalone protocol algorithms maintained similar trends as with the $1\mathrm{KB}$ file size, except for a few algorithm pairs that showed slightly higher latencies. The PQC standalone protocols achieved higher RTTs for BIKE-L5-F512 ($0.017\mathrm{s}$), BIKE-L3-F1024 ($0.014\mathrm{s}$), BIKE-L5-F1024, HQC-256-F512 ($0.016\mathrm{s}$), HQC-192-F1024 ($0.013\mathrm{s}$), and HQC-256-F1024 ($0.012\mathrm{s}$) compared to the same algorithm pairs in the hybrid mode and AES-256. Conversely, it was observed that the remaining algorithm pairs in hybrid mode showed the opposite trend, with slightly higher RTTs (HQC-128-F512 ($0.007\mathrm{s}$), HQC-192-F512 ($0.008\mathrm{s}$), HQC-128-F1024 ($0.007\mathrm{s}$), and K512-F1024 ($0.011\mathrm{s}$)) than those of the PQC standalone mode. This showcases the suitability of migrating applications to PQC algorithms while highlighting the processing bottlenecks associated with certain PQC algorithms.

Lastly, Figure 8c shows that for $100\mathrm{KB}$ files, AES-256 achieved $0.0048\mathrm{s}$, outperforming many PQC standalone methods: BIKE-L5-F512 required $0.0117\mathrm{s}$, 2.4 times slower, and HQC-256-F512 reached $0.015\mathrm{s}$, 3.1 times slower. Exceptions included HQC-128-F512 ($0.0028\mathrm{s}$) and K768-F1024 ($0.0039\mathrm{s}$), which were 1.7 times and 1.2 times faster than AES-256, respectively. Notably, in hybrid mode, K768-F1024 incurred a latency of $0.0275\mathrm{s}$ for $100\mathrm{KB}$ files, 5.7 times slower than AES-256, highlighting the compounded overhead of hybrid key exchanges and ciphertext processing. These results emphasize that while PQC algorithms may exhibit competitive throughput due to larger ciphertexts, their RTT performance is more directly tied to computational complexity and protocol design, exposing trade-offs between post-quantum security and real-time application suitability.

7. Conclusions

This paper proposed PQC standalone and PQC-AES hybrid protocols as part of the advancement towards the post-quantum era to evaluate their practical implementation within real-world secure communication systems. Specifically, the secure exchange of keys is achieved using PQC algorithms (KEM) in either PQC standalone or hybrid protocol modes. In the PQC standalone mode, the KEM (public) keys exchanged between the client and server are used for encryption, while each party uses their secret keys for decryption, following the principles of asymmetric key cryptography. Conversely, the PQC-AES hybrid mode uses two-way shared secret keys derived from PQC KEM algorithms to generate encryption and decryption keys using a key derivation function (KDF), thereby reducing the computational overhead for larger data sizes. The proposed PQC-based protocols mitigate replay, MITM attacks, and other conventional cybersecurity attack vectors.

The performance evaluation of the proposed protocols was conducted through experiments using a file transfer communication scenario in a client-server setup. The results confirm that the hybrid protocols, which leverage the strengths of PQC algorithms, achieved performance relatively closer to AES-256 for larger payloads. For $10\mathrm{KB}$ files, the K512-F1024 hybrid mode achieved $0.8\mathrm{Mbps}$, compared to AES-256’s $0.0035\mathrm{Mbps}$, without the significant latency bottlenecks observed in standalone PQC modes, such as $19.5\mathrm{Mbps}$ for K1024-F512 and K1024-F1024. Additionally, it was confirmed that the hybrid protocol achieved better performance in terms of processing time compared to the PQC standalone protocol and AES-256. The processing times for hybrid protocols were consistently lower across all algorithms except for HQC192-F512 and K1024-F512, which recorded $2.6\mathrm{s}$ and $5.4\mathrm{s}$ for $1\mathrm{KB}$ files, compared to AES-256’s $2.06\mathrm{s}$. For $100\mathrm{KB}$ files, HQC-192-F1024 and HQC-256-F1024 experienced higher processing times of $2.4\mathrm{s}$ and $3.1\mathrm{s}$, compared to AES-256’s $2.37\mathrm{s}$.

Our findings confirmed that standalone PQC algorithms remain effective for lightweight applications with smaller data sizes, while PQC-AES hybrid protocols provide a scalable and efficient solution for larger payloads, ensuring quantum-resistant security with manageable latency and computational complexity. Using PQC algorithms directly in most applications will adversely impact performance, while the hybrid mode approach is more suitable for most applications. This reaffirms the necessity of designing and migrating applications to quantum-safe cryptographic protocols tailored to specific application requirements.

In the advancement towards quantum-resistant security, a critical focus should be directed towards addressing stringent QoS requirements in emerging technologies like 5G/6G and IoT. These sectors necessitate a balance between robust security and real-time performance, requiring a hybrid approach that leverages the strengths of both PQC and traditional cryptographic algorithms. Our evaluation results provide valuable insights into the practicality and performance of PQC standalone protocols and can be extended to facilitate secure data transmission in 5G and IoT environments, ensuring a balance between quantum-resistant security and operational efficiency that upholds the stringent QoS requirements inherent in modern wireless networks.

As part of our future work, we will evaluate the complexity and overhead of the proposed protocols. We will extend the implementation of the PQC standalone PKE to include other PQC algorithms such as McEliece, Dilithium, and $SPHINC{S}^{+}$, among others. In addition, we will explore methods to utilize the PQC standalone PKE in various applications like video streaming and messaging. Furthermore, we will perform an extensive evaluation using various network environments, including 5G/6G.