CD-KLM: A Secure Storage Architecture for Agricultural Big Data

1. Introduction

With the rapid development of smart agriculture, massive volumes of agricultural big data are being continuously generated. Agricultural big data platforms are gradually becoming important infrastructure for supporting food security and the development of smart agriculture [1,2,3]. Some agricultural data involve satellite imagery, remote sensing data, and core breeding technologies that are closely related to food security [4,5,6]. Once such data are leaked or compromised, they may have a significant impact on food security. As shown in Figure 1, to support the storage and processing demands of massive agricultural big data, current agricultural big data platforms commonly adopt Hadoop-based distributed storage architectures [7,8], thereby enabling the efficient management and long-term storage of information such as remote sensing imagery, meteorological data, agricultural machinery operation trajectories, and environmental parameters. As noted in [9], Hadoop is a robust, fault-tolerant, and open-source distributed computing engine that can be easily scaled to thousands of machines [10]. Compared with general enterprise data, agricultural big data usually exhibit characteristics such as long-term archiving, multi-source heterogeneity, edge-side dispersion, and weak maintenance support, which make traditional secure storage models that continuously depend on online KMS more likely to expose insufficient continuous decryptability and limited recovery capability in agricultural scenarios. Therefore, how to construct a secure storage architecture for agricultural big data has become an unavoidable practical requirement in current agricultural information systems.

For secure data storage, Hadoop provides a solution based on transparent encryption and key management services [11,12]. This solution employs the AES-CTR algorithm [13] for data encryption and achieves centralized key management through the use of key encryption keys (KEKs) and data encryption keys (DEKs), thereby enabling rapid deployment and data encryption/decryption without modifying upper-layer application logic. However, from the perspective of system architecture, traditional centralized key management models still exhibit certain limitations in agricultural big data platforms. First, agricultural data often originate from multiple clients, such as grassroots stations, research and experimental platforms, and central data platforms, whereas under centralized KMS architectures, client participation in the key lifecycle is relatively limited, and the lifecycle of data keys mainly revolves around the online KMS service. Second, agricultural data often have long-term archival characteristics spanning seasons, years, or even breeding cycles, and many historical datasets are irreproducible; however, such architectures generally assume that critical keys remain continuously available, so once keys are damaged or lost, historical agricultural data may still exist but become undecryptable. Third, centralized KMSs usually adopt an envelope encryption mechanism and persistently store critical key materials in the key management system over the long term. Once the KMS is compromised, sensitive agricultural data, such as breeding materials, remote sensing analysis results, and historical field archives, may be decrypted, thereby amplifying the security risks associated with centralized key custody.

To address the limited participation of agricultural clients in the key lifecycle, existing studies have explored improvements from multiple perspectives. Some studies have pointed out that client-side encryption and decryption offer certain advantages over server-side approaches [14,15], while others adopt client-driven approaches for encrypting and decrypting big data [16,17], thereby providing a practical foundation for the application of client-side encryption and decryption in large-scale data scenarios. Although KMS performs encryption and decryption on the client side, its key lifecycle still depends on centralized services. To address this issue, some studies attempt to mitigate the risk of centralized service failures by enhancing the reliability and distributed coordination capability of key management services [18,19]. However, the main focus of such methods is usually on enhancing the reliability and distributed coordination capability of the key management mechanism itself, rather than further decoupling the decryption capability of historical files from the online key-service path. For agricultural big data platforms, historical data are typically archived over the long term. Therefore, such methods still exhibit certain limitations in enabling the client to independently control continuous decryptability. Some studies adopt distributed key management mechanisms and use multiple nodes to store key materials [20,21,22], thereby improving key availability and fault tolerance; however, in such systems, clients still usually participate in data read and write operations mainly as key users, while recovery capability has not been uniformly converged to the root-key level. Therefore, for agricultural big data platforms that require continuous access to historical data, such methods still exhibit certain limitations in ensuring the long-term, independently controllable decryption capability. In addition, some studies attempt to introduce blockchain to address the problems of centralized services [23,24,25,26]. Although these approaches provide certain advantages in tamper resistance and access control, they still expose limitations such as throughput bottlenecks and storage expansion when facing massive agricultural big data.

To address the problem that historical encrypted agricultural data may become undecryptable after key corruption or loss, common approaches usually rely on backing up key management states or using external recovery mechanisms to cope with key loss [27,28,29]. However, such methods usually fail to uniformly converge the recovery object to the root-key level, and when facing massive agricultural big data, they still need to maintain additional recovery information around file-level key states, backup materials, or external recovery conditions. Some studies combine Shamir threshold schemes with file-level encryption keys [30,31]. Although this can ensure key recoverability, such methods usually require maintaining corresponding sets of threshold shares for a large number of files, which leads to considerable additional storage and management overhead as the number of files increases. Some studies focus on distributed protection or recovery of upper-level critical key materials [32,33]. Although such methods can enhance the security and recoverability of upper-level critical keys, their recovery capability mainly operates at the private-key or master-key level and has not been further unified to the root-key level for supporting the continuous decryptability of historical files. Other studies attempt to combine secret sharing with key protection mechanisms for critical keys in cloud storage [34,35,36] in order to enhance key recovery capability. However, such methods have relatively low general applicability to Hadoop-based agricultural big data platforms and fail to uniformly converge the secret-sharing mechanism to the root-key level.

Regarding the risk that the limitations of envelope encryption may enable attackers to decrypt agricultural data, some studies employ mechanisms such as TEE and HSM [37,38,39,40] to isolate and protect critical keys, thereby reducing, to some extent, the risk of malicious data decryption after key leakage. However, such methods usually rely heavily on dedicated hardware environments, incur relatively high deployment and maintenance costs, and introduce a certain performance overhead; therefore, they still have certain limitations for high-throughput agricultural big data platforms. Some studies adopt dynamic key derivation and generate working keys on demand [41,42,43], thereby reducing the need to persistently store a large number of data encryption keys or their protecting keys in key management services. Nevertheless, such methods still depend on the critical materials required during the derivation process. Once these materials are damaged or lost, agricultural big data systems may still face the problem that the derived keys can no longer be reconstructed, and thus their recovery capability remains limited.

To address the above issues, this paper proposes a secure storage architecture for agricultural big data, named CD-KLM. The main contributions of this work are as follows:

(1)

We propose a client-driven, server-lightweight-collaborative secure storage architecture (CD-KLM) for agricultural big data platforms. In this architecture, data encryption and decryption capabilities are independently controlled by the client, while the server only issues collaborative salt values to authenticated clients during new file creation, thereby increasing client participation in the key lifecycle and improving data accessibility and system robustness in long-running environments.

(2)

We design a dynamic key derivation mechanism based on file-related contextual information, in which working keys are dynamically generated using the client root key, file identifier, key version, and the collaborative salt value issued by the server. This design avoids the long-term persistent storage of large numbers of file-level encryption keys for massive agricultural big data, thereby reducing the risk of unauthorized data decryption caused by prolonged exposure of critical key material.

(3)

We construct a root-level threshold key recovery mechanism that enables the system to restore its decryption capability even when critical keys are damaged or lost. Compared with file-level threshold methods, this mechanism unifies recovery operations at the root-key level, thereby achieving continuous decryptability at the system level while reducing the scale of key custody and the complexity of system management.

(4)

We implement a CD-KLM prototype in a real Hadoop distributed environment and evaluate its performance and reliability through security analysis and system experiments. The experimental results show that, while ensuring key recoverability and continuous data decryptability, the proposed architecture imposes only a low impact on system throughput and demonstrates good engineering deployability.

The remainder of this paper is organized as follows. Section 2 presents the system model and threat model, and clarifies the trust assumptions and security objectives of the system. Section 3 describes the implementation details and key workflows of the proposed scheme. Section 4 provides the security analysis of the implemented CD-KLM architecture. Section 5 presents the experimental evaluation and analysis of system performance and reliability. Section 6 discusses the security, applicability, and limitations of the proposed scheme. Finally, Section 7 concludes the paper.

2. System Model and Threat Model

This paper aims to propose a secure storage architecture for agricultural big data platforms, namely CD-KLM. To accurately describe the operating environment and security boundaries of this architecture, this section first introduces the Hadoop-based secure storage system model for agricultural big data, and then presents the threat model that may affect the system’s security and recoverability.

2.1. System Model

The proposed CD-KLM architecture is deployed in a Hadoop-based distributed storage environment and is compatible with existing agricultural big data platforms. In this architecture, data encryption and decryption are client-driven, while the server side provides only lightweight collaborative functions. The overall system consists of several core entities with functionally independent roles, which cooperate through clearly defined data interactions to accomplish secure data storage and recovery.As shown in Figure 2, the system mainly includes the following four types of entities:

Agricultural Client (Client):The agricultural client is the core security execution unit of the system. It is responsible for generating and holding the root key, and for performing working-key derivation, data encryption, and data decryption during data write and read operations. It also triggers lifecycle management operations such as key recovery and key rotation. This entity can be deployed on agricultural research platforms, grassroots data-governance nodes, or edge-side data collection and aggregation terminals, so as to directly carry the decryption control authority and lifecycle management capability for agricultural data.

Independent Salt Provisioning Service (ISP): The ISP is a lightweight collaborative service used during the file-creation phase. It only issues collaborative salt values to authenticated clients when new files are created, for participation in the working-key derivation process. It should be emphasized that the ISP neither holds the root key nor participates in decryption during the historical-file reading phase. Once the collaborative salt has been written into the file metadata, subsequent historical-file access relies solely on the RootKey held or recovered by the client and the public context to re-derive the working key, without requiring any further access to the ISP. Therefore, the ISP actually serves as an authorized issuer of collaborative context after creation-time authentication, rather than as a continuously online decryption service.

Distributed Storage Nodes (DSN):The distributed storage system is implemented based on the Hadoop Distributed File System (HDFS), in which the NameNode and DataNode are responsible for metadata management and data storage, respectively. File contextual information related to key derivation is recorded in the NameNode extended attributes (fid, epoch, salt), while encrypted data blocks are stored in ciphertext form in the DataNode. Here, epoch is used to distinguish different key versions and prevent old keys from being reused in a new cycle. This distributed storage layer can be used to host long-term agricultural data objects, such as remote sensing imagery, historical field archives, environmental monitoring records, and agricultural machinery operation data.

Secret Share Storage Nodes (SSN):These nodes are used to store root-key shares generated by the threshold secret sharing algorithm. Each node holds only partial key information, and a single node cannot recover the complete key. Only when the agricultural client obtains a threshold-satisfying number of key shares from multiple storage nodes can the root key be reconstructed.

2.2. Threat Model

Although Hadoop-based distributed storage systems provide certain fault-tolerance and reliability mechanisms, in long-running agricultural big data environments the system may still face various operational risks and security threats, such as service failures, critical key corruption, and ciphertext data leakage. To systematically analyze the security of the proposed CD-KLM architecture under complex operating conditions, this paper constructs the following threat model by formally describing the attacks that the system may face based on three common assumptions widely used in distributed systems and cryptography, namely the Crash Failure Model [44], the Honest-but-Curious Model [45], and the Threshold Adversary Model [46].

2.2.1. Threats

As shown in Figure 3, this paper considers that an attacker may access system components or obtain partial system information through multiple paths. Specifically, an attacker may eavesdrop on the data transmitted between the agricultural client and the distributed storage nodes in an attempt to obtain metadata and ciphertext information during communication.

In addition, the independent salt provider in the system may experience failures such as the crash of some service instances, thereby affecting the creation of new files and the derivation of new working keys.

Considering that agricultural data links usually span grassroots stations, edge terminals, and central platforms, and that some agricultural clients operate for long periods in weakly maintained environments, their devices are more susceptible to root-key corruption or loss due to attacks, misoperations, or failures, thereby affecting the overall decryption capability of the system. However, this paper does not consider the extreme scenario in which the agricultural client is fully compromised.

Meanwhile, an attacker may gain unauthorized access to distributed storage nodes and obtain ciphertext data stored on the DataNodes or read file-related contextual information recorded in the NameNode.

For the key recovery process, this paper further considers that an attacker may compromise a subset of secret share storage nodes and obtain a certain number of key shares, thereby attempting to recover the critical keys in the system.

2.2.2. Assumptions

The distributed storage nodes in the system are assumed to perform data storage and management operations according to the prescribed protocols; however, the data stored internally are not regarded as fully trustworthy. An attacker may obtain ciphertext data or related contextual information by controlling or compromising some of these nodes. Therefore, this paper assumes that the system nodes follow the Honest-but-Curious Model.

Considering that agricultural big data platforms typically operate in long-running distributed environments, some system components may experience operational failures or service interruptions, such as storage node crashes, unavailability of Independent Salt Provisioning Service (ISP) instances, or abnormal client device failures. In this paper, we assume that system nodes may crash or become unavailable, but they do not actively return forged data, and an attacker cannot bypass the creation-time authentication process to directly obtain legitimate collaborative salt values from the ISP. That is, the system follows the crash failure model.

During the key recovery process, an attacker may attempt to compromise some secret share storage nodes in order to obtain key shares. This paper assumes that, within any key epoch, the number of valid key shares successfully obtained by the attacker always remains below the predefined recovery threshold, i.e., m < t. Therefore, the attacker cannot independently recover the complete root key.

This paper assumes that the agricultural client is the only entity in the system that holds the complete root key (RootKey) and is able to correctly perform key derivation as well as data encryption and decryption operations. As long as the client is not fully compromised, its internal key material is assumed not to be actively leaked.

2.2.3. Objectives

Under the above threat assumptions, the design objective of the proposed system is to ensure that, even if some components fail or some information is leaked, the system can still maintain data confidentiality, continuous decryptability, and system robustness, and can reconstruct its decryption capability through the threshold recovery mechanism when critical keys are damaged or lost.

3. CD-KLM System Design and Key Workflows

This section focuses on the core design of CD-KLM, with emphasis on the underlying cryptographic building blocks, the stateless key derivation mechanism, and the secure read/write workflows in a Hadoop-based distributed environment. On this basis, it further explains how the system operates during lifecycle management stages such as root-key generation, key rotation, and threshold-based recovery, thereby providing a complete view of how CD-KLM achieves secure data storage, continuous decryptability, and root-level recoverability while reducing its dependence on the continuous availability of centralized key management services.

3.1. Cryptographic Primitives and Fundamental Construction

The CD-KLM architecture is built upon symmetric authenticated encryption[47], cryptographic hash functions[48], and threshold secret sharing[49], and further constructs a stateless key derivation mechanism, secure read/write workflows, and a root-key-level lifecycle management mechanism, thereby forming a complete secure storage solution.

3.1.1. Stateless Derivation and Authenticated Encryption

Considering that data objects in agricultural big data platforms usually exhibit characteristics of long-term accumulation and continuously growing scale, the long-term persistent maintenance of a large number of file-level working keys would further amplify the complexity of key custody and the risk of prolonged exposure. To avoid the long-term persistent storage of file-level working keys while simultaneously ensuring data confidentiality and integrity, CD-KLM introduces a context-based stateless key derivation mechanism and an authenticated encryption mechanism during data read and write operations. The specific process is as follows:

Advanced Encryption Standard (AES) is a symmetric block cipher, and its encryption function can be formalized as:

$$E_K:{\{0,1\}}^{128}\to {\{0,1\}}^{128}.$$

(1)

where K is the symmetric key.

$$K\in {\{0,1\}}^k.$$

(2)

Galois/Counter Mode (GCM) is an authenticated-encryption (AEAD) mode that provides confidentiality via Counter (CTR) mode and integrity/authentication via Galois-field-based authentication. Under AES-GCM, encryption is defined as:

$$(C,T)={\mathrm{Enc}}_K(P,IV,AAD).$$

(3)

Decryption is defined as:

$$P={\mathrm{Dec}}_K(C,IV,AAD,T).$$

(4)

Here, P is the plaintext, C is the ciphertext, $IV$ is the initialization vector, $AAD$ is the additional authenticated data, and T is the authentication tag.

The authentication component of AES-GCM is based on $\mathrm{GHASH}$, a polynomial hash over the finite field $\mathrm{GF}\left(2^{128}\right)$. Its security relies on the pseudorandomness of the AES block cipher and the collision resistance of $\mathrm{GHASH}$. AES-GCM supports streaming and parallel computation and is widely used in high-throughput, low-latency distributed storage and network communication systems.

HKDF (HMAC-based Key Derivation Function) is a MAC-based key derivation function that derives multiple independent subkeys from input key material. HKDF consists of two stages: Extract and Expand. The Extract stage is defined as:

$$\mathrm{PRK}={\mathrm{HMAC}}_{\mathrm{salt}}\left(\mathrm{IKM}\right).$$

(5)

Here, $\mathrm{IKM}$ is the input key material and $\mathrm{salt}$ is an optional random salt. In our model, the input key material and salt are instantiated as:

$$\begin{array}{c}\mathrm{IKM}=\mathtt{RootKey},\end{array}$$

(6)

$$\begin{array}{c}\mathrm{salt}=\mathtt{Salt},\end{array}$$

(7)

$$\begin{array}{c}\mathtt{Salt}\stackrel{\$}{\leftarrow }{\{0,1\}}^{256}.\end{array}$$

(8)

The 256-bit Salt is generated using a cryptographically secure pseudorandom number generator.

The per-file salt is used as the HKDF salt in the Extract step. In CD-KLM, the Salt value is issued by the ISP to authenticated clients during new-file creation and is then bound to the derivation context of a specific file and key version. Therefore, the role of Salt is not merely to introduce per-file randomness, but also to serve as a server-assisted contextual input in working-key derivation. Once Salt is recorded in the file metadata, subsequent historical-file access no longer requires any further interaction with the ISP. The Expand step is defined as:

$$K={\mathrm{HMAC}}_{\mathrm{PRK}}\left(\mathrm{info}\Vert 1\right).$$

(9)

In our model, the expansion context $\mathrm{info}$ is explicitly constructed as:

$$\mathsf{info}=UTF8\left(\mathtt{fid}\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(\mathtt{epoch}\right).$$

(10)

Here, ${BE}_8\left(\mathtt{epoch}\right)$ denotes the 8-byte big-endian encoding of the key-version identifier. fid is the file-unique identifier and epoch is the key-version number, which together enable file-level key isolation and controlled key rotation. The resulting file-level working key is:

$$WK=\mathrm{HKDF}-\mathrm{Expand}\left(\mathrm{HKDF}-\mathrm{Extract}(\mathtt{Salt},\mathtt{RootKey}),\mathsf{info},32\right).$$

(11)

Therefore, $WK\in {\{0,1\}}^{256}$.As a file-level key context, fid and epoch, together with Salt, are persistently stored as HDFS inode extended attributes.

This design makes working keys for different files and different versions cryptographically independent: compromise of any derived key does not affect the security of other files or historical versions. The working-key generation and client-side encryption procedure are summarized in Algorithm 1.

Algorithm 1:Client-Driven Encryption and Key Derivation

Input:RootKey, Salt, file F, file identifier fid, key epoch epochOutput: Encrypted file C 1: The client obtains file metadata and initializes encryption context; 2: Set input key material $\mathrm{IKM}\leftarrow \mathtt{RootKey}$; 3: Set salt $\mathrm{salt}\leftarrow \mathtt{Salt}$; 4: Compute pseudorandom key 4:    $\mathrm{PRK}\leftarrow \mathrm{HKDF}-\mathrm{Extract}(\mathrm{salt}=\mathtt{Salt},\mathrm{IKM}=\mathtt{RootKey})$; 5: Construct expansion context $\mathrm{info}\leftarrow UTF8\left(\mathtt{fid}\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(\mathtt{epoch}\right)$; 6: Derive file-level working key 6:    $WK\leftarrow \mathrm{HKDF}-\mathrm{Expand}(\mathrm{PRK},\mathrm{info},L=32)$; 7: Read file data in buffered byte stream; 8: Split file into fixed-size chunks $\left\{P_i\right\}$; 9: For each chunk $P_i$: 9:     Sample a fresh 96-bit random IV ${\mathrm{IV}}_i\stackrel{\$}{\leftarrow }{\{0,1\}}^{96}$; 9:    $i\leftarrow ⌊\mathrm{logical}\_\mathrm{offset}/\mathrm{chunk}\_\mathrm{size}⌋$; 9:     Construct ${\mathrm{AAD}}_i\leftarrow UTF8\left(\mathtt{fid}\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(\mathtt{epoch}\right)\Vert {BE}_4\left(i\right)$; 9:     Perform AES-GCM encryption using $WK$, ${\mathrm{IV}}_i$, and ${\mathrm{AAD}}_i$; 10: Construct encrypted chunk frame: 10:     ${\ell }_i\leftarrow |{\mathrm{CT}}_i\Vert {\mathrm{TAG}}_i|$; 10:     ${\mathrm{Frame}}_i\leftarrow 〈{BE}_4\left(i\right)\Vert {BE}_4\left({\ell }_i\right)\Vert {\mathrm{IV}}_i\Vert {\mathrm{CT}}_i\Vert {\mathrm{TAG}}_i〉$; 11: Output encrypted file $C\leftarrow \left\{{\mathrm{Frame}}_i\right\}$;

3.1.2. Root-Key-Level Threshold Secret Sharing

On the basis of enabling on-demand derivation of file-level working keys and providing authenticated encryption protection for data, CD-KLM further introduces a threshold secret sharing mechanism at the root-key level to ensure that critical keys remain recoverable in scenarios of key corruption or loss. The specific computation process is as follows:

The 256-bit client root key is represented as a sequence of 32 byte-sized field elements:

$$\mathrm{RootKey}=(s_1,s_2,\dots ,s_{32})\in {\mathbb{F}}_{2^8}^{32}.$$

(12)

In the implementation, arithmetic is performed over ${\mathbb{F}}_{2^8}$ using the irreducible polynomial

$$m\left(z\right)=z^8+z^4+z^3+z+1,$$

(13)

corresponding to the AES polynomial 0x11B.

For each byte position $b\in \{1,\dots ,32\}$, the client independently samples $t-1$ coefficients

$$a_{1,b},a_{2,b},\dots ,a_{t-1,b}\stackrel{\$}{\leftarrow }{\mathbb{F}}_{2^8},$$

(14)

and constructs a degree-$(t-1)$ polynomial

$$f_b\left(x\right)=s_b+a_{1,b}x+a_{2,b}{x}^2+\dots +a_{t-1,b}{x}^{t-1}.$$

(15)

For n distinct nonzero evaluation points $x_i=i$, $i=1,\dots ,n$, where $n\le 255$, the i-th share is constructed as

$${\mathsf{share}}_i=\left(x_i,f_1\left(x_i\right),f_2\left(x_i\right),\dots ,f_{32}\left(x_i\right)\right).$$

(16)

Given any set $\mathcal{S}$ containing at least t valid shares, each byte of the root key is reconstructed independently using Lagrange interpolation:

$$s_b=f_b\left(0\right)=\sum _{i\in \mathcal{S}}{f}_b\left(x_i\right)\prod _{\begin{array}{c}j\in \mathcal{S}\\ j\ne i\end{array}}{\displaystyle \frac{x_j}{x_i+x_j}},b=1,\dots ,32,$$

(17)

where all arithmetic is performed over ${\mathbb{F}}_{2^8}$. Since the field has characteristic two, addition and subtraction are both implemented as bitwise XOR.

The complete root key is then reconstructed as

$$\mathrm{RootKey}=(s_1,s_2,\dots ,s_{32}).$$

(18)

Any set of fewer than t valid shares reveals no information about the RootKey in the information-theoretic sense.

The Shamir threshold mechanism in CD-KLM is confined to the storage and recovery of the client root key RootKey, and is excluded from the online data encryption and decryption path, thereby avoiding additional computational overhead on high-frequency I/O operations.

The central design principle is that decryptability is not embodied in any individual key share. Instead, it is reconstructed by the client only when needed through threshold recovery of RootKey and subsequent deterministic key derivation, as summarized in Algorithm 2.

Algorithm 2:Client-Driven Key Lifecycle Management

Input:RootKey, threshold t, total shares n, key epoch epoch Output: Recoverable RootKey for epoch epoch 1: Client loads existing root key RootKey; 2: Represent $\mathtt{RootKey}=(s_1,\dots ,s_{32})\in {\mathbb{F}}_{2^8}^{32}$; 3: For each byte position b, construct an independent $(t-1)$-degree polynomial $f_b\left(x\right)$ over ${\mathbb{F}}_{2^8}$ with $f_b\left(0\right)=s_b$; 4: Generate n Shamir shares: 4:      ${\mathit{share}}_i=(x_i,f_1\left(x_i\right),\dots ,f_{32}\left(x_i\right)),i=1,\dots ,n$; 5: Distribute each ${\mathit{share}}_i$ to an independent share-storage node SSN${}_i$; 6: Bind shares to key epoch epoch and store epoch metadata; 6: Recovery / Rotation Phase: 7: Upon a recovery/rotation request, collect at least t valid shares for epoch epoch; 8: Reconstruct $\mathtt{RootKey}\left(\mathtt{epoch}\right)$ via Lagrange interpolation; 9: if key rotation is required then 10:     Generate a new RootKey; 11:     Increment epoch and repeat Steps 2–6; 12: end if

3.2. Stateless Key Derivation and Secure Read/Write Workflows

To ensure the engineering deployability of CD-KLM in smart agriculture datacenter scenarios, the data plane is seamlessly integrated into the DFSClient write path of HDFS. As shown in Figure 4, CD-KLM performs context-based stateless encryption encapsulation on the output data stream before data packetization and pipeline sending, achieving end-to-end protection for agricultural data without changing the storage semantics of the NameNode and DataNode.

This design embeds the encryption logic into the stream encapsulation layer within the DFSClient’s hot path, enabling the client to derive working keys on the fly based on the RootKey and public context (Salt, fid, epoch). In this process, HDFS’s native replica placement, pipeline transmission, and checksum mechanisms are fully preserved, ensuring system performance stability in high-throughput scenarios. The specific instrumentation locations are shown in Table 1.

When the agricultural big data cryptographic client initiates a write request, it executes the following stateless encryption workflow within the data path:

(1)

Context initialization:Authenticated clients obtain the random salt value, Salt, from the Independent Salt Provisioning Service (ISP), and combine it with the file identifier fid and the current version identifier epoch as the key-derivation context. These context values are all public parameters and do not contain any sensitive key material.

(2)

Working-key derivation:Based on the locally held RootKey, the client performs context-bound key derivation in memory using HKDF to generate the working key WK for the corresponding file and version.

(3)

Chunk-level authenticated encryption:Within the data transmission workflow of DFSOutputStream, the client divides the byte stream into fixed-size chunks before packetization, and performs AES-GCM authenticated encryption on each chunk independently, thereby providing joint protection for confidentiality and integrity.

During the file open phase, the client retrieves ciphertext data blocks from the DataNode pipeline while simultaneously reading lightweight public context parameters from the NameNode’s extended attributes, including Salt, fid, and epoch. After native HDFS checksum verification and data reassembly are completed within DFSInputStream, the client derives the corresponding working key (WK) in memory based on the local RootKey and then decrypts the data blocks. This design enables previously stored historical agricultural data to be accessed without continuous reliance on online key services, making it better suited to agricultural environments characterized by distributed deployment, service fluctuations, and long-term operation.

3.3. Key Lifecycle Management Mechanism

On the basis of completing the design of stateless key derivation and secure read/write workflows, CD-KLM further requires unified management of root-key generation, rotation, and recovery during long-term operation, and therefore establishes a corresponding key lifecycle management mechanism. The specific process is as follows:

3.3.1. Root Key Generation and Rotation

During the system initialization phase, the client first generates a high-entropy random root key, RootKey, in a secure local environment and uses it as the core secret input for subsequent working-key derivation. At the same time, the system initializes the key version identifier as epoch = 1, which is used to mark the current root-key period and to provide the basis for subsequent key rotation and version isolation. After initialization, the client locally stores the complete RootKey, while the server does not store any complete root-key information. Thereafter, when the client creates a new file, the system dynamically derives the corresponding working key by combining RootKey, the file identifier fid, the version identifier epoch, and the random salt salt, thereby supporting subsequent secure data write and read operations.

Considering that agricultural big data platforms usually operate in long-running distributed environments, the system needs to support periodic updates of the root key in order to reduce the risks associated with the long-term use of a single root key. To this end, CD-KLM introduces an epoch-based root-key rotation mechanism. When key rotation occurs, the client first generates a new root key, RootKey’, locally, increments the version identifier to a new epoch value, and then derives working keys for newly created files based on the new RootKey’. Meanwhile, historical files retain the epoch identifier corresponding to the time of their creation. When reading historical files, the client can select the corresponding root-key period according to the version information recorded in the file metadata and perform working-key derivation accordingly, thereby ensuring strict isolation among different key periods. In this way, the system not only supports the continuous evolution of the root key, but also guarantees that historical data remain continuously decryptable after key rotation. This mechanism is particularly suitable for historical agricultural data that must be preserved continuously across production cycles and years, because their accessibility requirements cannot be interrupted by key rotation.

3.3.2. Root-Key-Level Recovery Protocol

As shown in Figure 5,to ensure that the system remains capable of recovering its overall decryption capability when critical keys are damaged or lost, CD-KLM introduces a threshold secret sharing mechanism at the root-key level. After root-key generation or rotation is completed, the client applies the Shamir threshold secret sharing algorithm to divide the current root key into n key shares, which are then securely distributed to multiple independent Secret Share Storage Nodes (SSNs) according to the predefined recovery threshold t. Each SSN stores only one key share, and a single node cannot recover the complete root key. Only when the client can subsequently collect no fewer than t valid shares can the complete root key be reconstructed. In this way, the system unifies key recovery operations at the root-key level, without requiring the maintenance of separate threshold recovery states for a large number of files.

During the key recovery phase, when the client detects that the local RootKey has been damaged, lost, or become unavailable, it can issue recovery requests to multiple SSNs and collect no fewer than t valid key shares. The client then executes the threshold recovery algorithm locally to reconstruct the complete root key of the corresponding version and regain the system’s overall decryption capability. Since file-level working keys are not persistently stored over the long term but are instead dynamically derived on demand from the root key together with file-related context, once root-key recovery succeeds, the client can re-derive the working keys of historical files based on the fid, epoch, and salt recorded in the file metadata, thereby restoring access to the corresponding ciphertext data. For agricultural big data platforms, this means that long-accumulated data objects, such as historical field archives, temporal remote sensing data, and breeding records, will not lose their overall accessibility due to a single local key incident.

4. Security Analysis

To analyze the security of the proposed CD-KLM architecture in a more rigorous manner, this section formalizes the system security goals, adversarial capabilities, and key security properties based on the threat model defined in Section 2. The main security objectives considered in this work include data confidentiality, data integrity and authenticity, root-key recoverability, unrecoverability below the threshold, and the independence of historical-file decryptability from the online state of the ISP.

According to the system design, the client holds the root key $\mathit{RootKey}$, and the public file context is denoted by

$$ctx=(fid,epoch,salt),$$

(19)

where $fid$ is the unique file identifier, $epoch$ is the key-version identifier, and $salt$ is the collaborative salt value issued by the ISP during file creation and persisted in the file metadata. The corresponding file-level working key is defined as

$$WK=\mathrm{HKDF}-\mathrm{Expand}\left(\mathrm{HKDF}-\mathrm{Extract}(salt,\mathit{RootKey}),\mathsf{info},32\right),$$

(20)

$$\mathsf{info}=UTF8\left(fid\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(epoch\right).$$

(21)

For the i-th plaintext chunk $P_i$ of a file, the system applies AES-GCM encryption and obtains

$$AA{D}_i=UTF8\left(fid\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(epoch\right)\Vert {BE}_4\left(i\right).$$

(22)

Under the adversarial model considered in this paper, the attacker is allowed to read ciphertext data stored on the DataNodes, obtain the public context $(fid,epoch,salt)$ stored in the NameNode, cause partial or complete ISP failures, and compromise fewer than t secret-share storage nodes to obtain partial shares. However, the attacker cannot directly obtain the complete $\mathit{RootKey}$, and the extreme case in which the client is fully compromised is outside the scope of this work.

4.1. Confidentiality of Stored Data

We first analyze data confidentiality. Consider the following confidentiality game. An adversary $\mathcal{A}$ chooses two equal-length plaintexts $M_0$ and $M_1$. The challenger randomly selects a bit $b\in \{0,1\}$, derives

$$WK=\mathrm{HKDF}-\mathrm{Expand}\left(\mathrm{HKDF}-\mathrm{Extract}(salt,\mathit{RootKey}),\mathsf{info},32\right),$$

(23)

$$\mathsf{info}=UTF8\left(fid\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(epoch\right).$$

(24)

and encrypts the corresponding plaintext $M_b$ to produce the challenge ciphertext. The adversary then outputs a guess $b^{\prime }$. The confidentiality advantage of $\mathcal{A}$ is defined as

$$Ad{v}_{\mathcal{A}}^{conf}=\left|Pr[b^{\prime }=b]-{\displaystyle \frac{1}{2}}\right|.$$

(25)

Theorem 1. Assume that HKDF behaves as a pseudorandom key-derivation function under an unknown $\mathit{RootKey}$, and that AES-GCM is secure under a random working key. Then, for any probabilistic polynomial-time adversary $\mathcal{A}$ that does not obtain $\mathit{RootKey}$, the advantage $Ad{v}_{\mathcal{A}}^{conf}$ is negligible.

Proof sketch. Although the adversary can obtain the public context $(fid,epoch,salt)$ and the corresponding ciphertext, none of these values reveals the secret $\mathit{RootKey}$. Since the attacker cannot reconstruct a valid $WK$ without $\mathit{RootKey}$, the attacker cannot perform valid decryption on the challenge ciphertext. Therefore, under the assumed pseudorandomness of HKDF and the confidentiality of AES-GCM, the plaintext remains computationally indistinguishable to the attacker. Hence, CD-KLM preserves data confidentiality under the given threat model.

4.2. Integrity and Authenticity of Encrypted Data

We next analyze integrity and authenticity. For each encrypted chunk, the output takes the form

$$C_i=(I{V}_i,C{T}_i,TA{G}_i),$$

(26)

and the corresponding verification process is

$$P_i=De{c}_{WK}(C{T}_i,I{V}_i,AA{D}_i,TA{G}_i).$$

(27)

To forge a ciphertext chunk that can pass system verification, an adversary must generate a valid authentication tag $TA{G}_i^{\prime }$ for some modified tuple $(I{V}_i^{\prime },C{T}_i^{\prime },AA{D}_i^{\prime })$ without knowing the correct $WK$. Let the forgery advantage of the adversary be defined as the probability of producing a fresh ciphertext tuple that is accepted by the decryption algorithm.

Theorem 2. If AES-GCM satisfies the standard unforgeability property of authenticated encryption with associated data (AEAD), then the forgery advantage of any probabilistic polynomial-time adversary is negligible.

Proof sketch. AES-GCM simultaneously provides confidentiality and integrity. In CD-KLM, the associated authenticated data are defined as

$$AA{D}_i=UTF8\left(fid\right)\Vert \mathtt{0}\mathtt{x}\mathtt{00}\Vert {BE}_8\left(epoch\right)\Vert {BE}_4\left(i\right).$$

(28)

which bind the file identifier, logical chunk index, and key version to the authentication domain. Therefore, not only the ciphertext content itself, but also the file identity, chunk position, and key epoch are cryptographically bound to the authentication tag. Any attempt to tamper with ciphertext content, alter chunk indices, replay chunks across files, or replay chunks across different key epochs will cause authentication verification to fail. Hence, CD-KLM provides detectable integrity and authenticity protection for encrypted data.

4.3. Root-Key Recoverability and Threshold Resistance

We now analyze root-key recoverability and unrecoverability below the threshold. The 256-bit client-held root key is represented as

$$\mathrm{RootKey}=(s_1,\dots ,s_{32})\in {\mathbb{F}}_{2^8}^{32}.$$

(29)

For each byte position b, CD-KLM independently constructs a degree-$(t-1)$ polynomial

$$f_b\left(x\right)=s_b+a_{1,b}x+a_{2,b}{x}^2+\dots +a_{t-1,b}{x}^{t-1},$$

(30)

where $a_{1,b},\dots ,a_{t-1,b}$ are independently and uniformly sampled from ${\mathbb{F}}_{2^8}$.

For n distinct nonzero evaluation points $x_i$, the i-th share is defined as

$${\mathsf{share}}_i=\left(x_i,f_1\left(x_i\right),f_2\left(x_i\right),\dots ,f_{32}\left(x_i\right)\right).$$

(31)

When the client obtains at least t valid shares, each byte of the RootKey can be reconstructed by Lagrange interpolation over ${\mathbb{F}}_{2^8}$. Accordingly, the system satisfies the following recovery correctness property:

$$\left|\mathcal{S}\right|\ge t\Rightarrow Recover\left(\mathcal{S}\right)={\mathrm{RootKey}}_{\mathrm{epoch}},$$

(32)

where $\mathcal{S}$ denotes the set of collected valid shares.

On the other hand, Shamir secret sharing provides information-theoretic threshold security. Any set containing fewer than t valid shares reveals no information about the RootKey. Therefore,

$$\left|\mathcal{S}\right|<t\Rightarrow I(\mathrm{RootKey};\mathcal{S})=0,$$

(33)

where $I(·;·)$ denotes mutual information.

Theorem 3. CD-KLM satisfies both threshold recoverability and threshold resistance: if at least t valid shares are available, the correct root key can be reconstructed; if fewer than t shares are obtained, the attacker gains no useful information about the root key.

Proof sketch. The first part follows from the correctness of byte-wise Shamir interpolation over ${\mathbb{F}}_{2^8}$, while the second part follows from the information-theoretic security of Shamir secret sharing. Since file-level working keys are not persistently stored but are instead re-derived on demand from $\mathit{RootKey}$ and file context, successful recovery of the root key implies recoverability of the decryption capability for all historical files associated with that key epoch.

4.4. Independence of Historical-File Decryptability from ISP Availability

Finally, we analyze the independence of historical-file decryptability from the online state of the ISP. Let the ISP state be denoted by

$$ISPState\in \{0,1\},$$

(34)

where $ISPState=1$ means that the ISP is online and available, and $ISPState=0$ means that the ISP is unavailable.

For any historical file F, suppose that its public context

$$ct{x}_F=(fid,epoch,salt)$$

(35)

has already been persisted in the NameNode metadata, and that the client still holds or can recover the corresponding $\mathit{RootKey}$. Define the historical-file decryptability event as $DecHist\left(F\right)=1$ if and only if the client can successfully re-derive the correct $WK$ from the persisted context and decrypt the file.

Then the following property holds:

$$\begin{array}{cc}\hfill Pr\left[DecHist\right(F)=1\mid & ISPState=1]=\hfill \\ \hfill Pr\left[DecHist\right(F)=1\mid & ISPState=0].\hfill \end{array}$$

(36)

Theorem 4. For any previously created file, once the context $(fid,epoch,salt)$ has been persisted and the corresponding $\mathit{RootKey}$ is available, the ability to decrypt that historical file is independent of the online state of the ISP.

Proof sketch. The ISP only participates in collaborative salt issuance during new-file creation. Once the salt has been written into the file metadata, subsequent historical-file access requires only the client-held or recovered $\mathit{RootKey}$ together with the persisted public context to re-derive the corresponding working key. Therefore, ISP failures affect only new-file creation, but do not propagate to the read and decryption path of historical files.

4.5. Summary of Security Properties

Under the threat model defined in this paper, CD-KLM simultaneously satisfies the following key security properties: (1) an attacker cannot recover plaintext from ciphertext and public context without the client-held $\mathit{RootKey}$; (2) an attacker cannot forge a valid ciphertext chunk without the correct working key; (3) the client can recover $\mathit{RootKey}$ when the threshold condition is satisfied, whereas an attacker with fewer than t shares cannot reconstruct it; and (4) the continuous decryptability of historical files is independent of the online availability of the ISP. Together, these properties show that CD-KLM can provide confidentiality, integrity, recoverability, and structurally sustained decryptability for long-running Hadoop-based agricultural big data storage environments.

5. Experimental Results and Analysis

To systematically evaluate the effectiveness and engineering practicality of the proposed CD-KLM architecture, this section conducts experiments from multiple perspectives, including service availability, key recovery capability, key storage overhead, and encryption/decryption performance, and analyzes the corresponding results.

5.1. Experimental Environment Configuration

This study employs a distributed Hadoop cluster to simulate a realistic agricultural big data processing environment. The cluster architecture consists of one NameNode, four data nodes (DSNs), four secret share storage nodes (SSNs), four independent salt providers (ISPs), and a native Hadoop KMS used for comparative experiments. The core hardware and software configurations of the system are as follows:

(1)

NameNode: 4-core CPU, 4 GB RAM (Ubuntu 22.04, Hadoop 3.3.4)

(2)

DataNodes (4 nodes): 4-core CPU, 4 GB RAM, 2 TB NVMe (Ubuntu 22.04, Hadoop 3.3.4)

(3)

Client node: 8-core CPU, 16 GB RAM (Java 11, CD-KLM core library)

(4)

SSN nodes (4 nodes): 4-core CPU, 4 GB RAM (Ubuntu 22.04)

(5)

ISP nodes (4 nodes): 4-core CPU, 4 GB RAM (Ubuntu 22.04)

(6)

KMS node (1 node): 8-core CPU, 8 GB RAM (Ubuntu 22.04)

In terms of cryptographic suite configuration, the data plane adopts AES-256-GCM to provide authenticated encryption and uses HKDF to implement stateless key derivation; the Shamir threshold parameters are configured as a total number of shares n=4 and a recovery threshold t=3. For service deployment, the ISPs adopt an Nginx-based load-balancing scheme to provide a unified external access entry. To ensure the comprehensiveness and realism of the experimental results, this study selects publicly available agricultural data resources as the base samples, including agricultural statistical data provided by FAOSTAT, cropland cover and crop-type raster data provided by the USDA NASS Cropland Data Layer (CDL), and agricultural research data resources collected by platforms such as CGIAR GARDIAN. Based on the granularity characteristics of different agricultural data objects, corresponding test file sets are constructed. Specifically, the small-file set ranging from 4 MB to 64 MB is mainly used to simulate fine-grained agricultural data objects such as field archives and statistical records, while the large-file set ranging from 512 MB to 4 GB is mainly used to simulate large-volume agricultural data objects intended for long-term preservation, such as cropland-cover rasters, remote sensing imagery, and other similar datasets. In this way, the experiments cover typical storage scenarios in agricultural big data platforms characterized by multi-source heterogeneity and significant granularity differences.

5.2. Service Availability Analysis

To evaluate the availability of the proposed CD-KLM architecture under service-abnormal scenarios, this section conducts experimental analysis from two aspects: the decryption correctness rate of historical files and the success rate of new-file creation. In the experiments, service-failure conditions in agricultural data platforms are simulated by shutting down the native KMS service and gradually reducing the number of available ISP instances, so as to observe the service availability of the system when the online key service is unavailable and when the salt-collaboration service is partially or completely unavailable.

As shown in Table 2, in the experiment on the decryption correctness rate of historical files, both native KMS and CD-KLM achieve a 100% decryption correctness rate when the service is in the normal state. However, once the online service fails, the decryption correctness rate of native KMS drops to 0%, whereas CD-KLM still maintains a 100% decryption correctness rate.This indicates that the two types of online services assume different responsibilities: the service status of the native KMS directly affects the historical-file decryption path, whereas the role of the ISP is restricted to the new-file creation phase, so the impact of its failure does not extend to the historical-file access path. Therefore, for historical agricultural data that require long-term preservation and continuous access, CD-KLM can avoid having the historical-file decryption path remain continuously constrained by the status of centralized online services.

Furthermore, in the success-rate experiment under different numbers of available ISP instances, CD-KLM exhibits clearly different dependency characteristics for new-file creation and historical-file access. When the number of available ISP instances decreases from 4 to 1, the success rate of new-file creation remains at 100%, indicating that the multi-instance ISP deployment combined with Nginx load balancing can effectively mitigate the impact of single-point failures. However, when the number of available ISP instances is further reduced to 0, the success rate of new-file creation drops to 0%. Meanwhile, the success rate of historical-file access remains at 100% throughout the entire process as the number of available ISP instances decreases from 4 to 0. This indicates that even if the encryption phase for newly generated data is affected by failures of the collaborative service, previously stored field archives, statistical records, or historical remote sensing data can still remain accessible.

5.3. Key Recovery Capability Analysis

To evaluate the recovery capability and continuous decryptability of CD-KLM under scenarios involving critical-key corruption, key loss, partial storage-node failures, and key rotation, this section conducts experimental analysis from four aspects: key splitting and distribution latency, key recovery latency, recovery success rate under SSN node failures, and the decryption correctness rate of historical files after rotation.

As shown in Figure 6Figure 7, under repeated experiments ranging from 100 to 500 runs, both the average key splitting-and-distribution latency and the average key recovery latency remain within the millisecond range and gradually converge as the number of repetitions increases. Specifically, the average key splitting-and-distribution latency decreases from approximately 3.38 ms and stabilizes at around 2.70 ms, while the average key recovery latency decreases from approximately 3.31 ms to about 2.41 ms.

For key recovery, as shown in Figure 8, by manually shutting down a subset of SSN instances to simulate SSN service failures, the key recovery success rate of CD-KLM exhibits a clear boundary that is strictly consistent with the threshold parameters. When the number of failed nodes is 0 or 1, the system recovery success rate remains at 100%; however, when the number of failed nodes increases to 2, the recovery success rate drops to 0%. This phenomenon is fully consistent with the Shamir threshold parameters used in the experiments, n=4 and t=3, meaning that the system can tolerate at most one share-storage node failure, whereas RootKey reconstruction cannot be completed once the number of available shares falls below the recovery threshold.

Furthermore, to further verify the recovery effectiveness of CD-KLM under key-version evolution scenarios, this study evaluates the decryption correctness rate of historical files after root-key rotation. As shown in Table 3, each experimental round generates a new RootKey and encrypts one file. As the epoch continues to increase, the client can re-derive the working key for historical-file access based on the fid, epoch, and salt recorded in the file metadata; when the corresponding version of the RootKey is no longer available locally, the client can further recover that version of the RootKey through SSN and then complete historical-file decryption. The experimental results show that, under multiple key-rotation rounds, the decryption correctness rate of historical files remains at 100%, indicating that root-key rotation does not compromise the accessibility of previously stored historical data. This shows that, for historical agricultural data accumulated across seasons and years, the system can restore its overall decryption capability after local critical keys are damaged or lost, thereby preventing long-term data assets from becoming unusable due to a single key incident.

5.4. Key Storage Overhead Analysis

To evaluate the control-plane storage overhead of CD-KLM under large-scale file scenarios, this section compares it with a file-level recovery model. In the experiments, the storage space directly related to key recovery is measured under different file scales for both schemes, while the storage occupied by ciphertext itself in the data plane is excluded. According to the current experimental configuration, each share record occupies approximately 132 B, and each secret corresponds to four shares; therefore, the recovery-state overhead of the file-level recovery model is about 528 B per file. In contrast, in CD-KLM, the threshold-custody object is unified at the root-key level. Under a single-epoch setting, the system only needs to maintain one set of threshold shares corresponding to the RootKey, while each file only stores lightweight public context information, namely fid, epoch, and salt, whose associated metadata size is about 69 B.

As shown in Table 4, when the file scale increases from ${10}^3$ to ${10}^6$, the control-plane storage overhead of the file-level recovery model is approximately 0.50 MB, 5.04 MB, 50.35 MB, and 503.54 MB, respectively, whereas that of CD-KLM is approximately 0.07 MB, 0.66 MB, 6.58 MB, and 65.80 MB, respectively. It can be observed that, at all tested scales, the storage overhead of CD-KLM remains significantly lower than that of the file-level recovery model, and the absolute gap between the two continues to widen as the file scale increases. When the file scale reaches ${10}^6$, the file-level recovery model requires about 503.54 MB of control-plane storage space, whereas CD-KLM requires only about 65.80 MB, reducing the overhead by approximately 437.74 MB. This is particularly important for agricultural big data platforms, where file scale continues to expand with production cycles and historical accumulation, because if the states related to key recovery keep growing, they will directly increase the burden of subsequent system deployment and operational management.

5.5. Encryption and Decryption Performance Analysis

To evaluate the encryption and decryption performance of CD-KLM in the data plane, this section conducts comparative experiments on both fine-grained agricultural data objects and large-volume agricultural data objects, and compares CD-KLM with the Plain baseline without encryption/decryption and the native Hadoop KMS. The experiments focus on the average encryption throughput and average decryption throughput under different file scales, so as to analyze the performance overhead of CD-KLM in practical agricultural data read/write scenarios and its variation trend.

As shown in Figure 9, under fine-grained agricultural data object workloads, the performance of the three mechanisms is more easily affected by fixed control overhead and runtime jitter, and thus the throughput fluctuations are relatively obvious. In the 4 MB scenario, the encryption throughput of CD-KLM is close to that of the Plain baseline and higher than that of Native KMS, indicating that under extremely small-file conditions, initialization overhead, cache status, and scheduling noise may mask part of the additional encryption overhead. However, when the file size further increases to 32 MB and 64 MB, the overall encryption and decryption throughput of CD-KLM becomes slightly lower than that of Plain and Native KMS.

Furthermore, as shown in Figure 10, under large-volume agricultural data object workloads, as the file size increases from 512 MB to 4096 MB, the performance differences among the three mechanisms gradually become stable, and the fixed control overhead is continuously amortized. For encryption throughput, CD-KLM exhibits a certain degree of degradation compared with Plain at 512 MB; however, when the file size increases to 4096 MB, the gap converges to approximately 3%. Compared with Native KMS, the performance gap of CD-KLM in large-file write scenarios remains consistently small and stays within the same order of magnitude overall. For decryption throughput, CD-KLM also shows only limited degradation compared with Plain and Native KMS, with a relatively small fluctuation range and no continuous deterioration as the file size increases.

6. Discussion

Compared with existing studies, it can be observed that this work is generally consistent with prior client-side encryption/decryption and dynamic key derivation approaches in its overall direction, in that all of them emphasize reducing the dependence on long-term persistent file-level keys in order to mitigate the risk of critical material exposure. However, unlike schemes that merely shift the execution of encryption and decryption to the client side, CD-KLM further consolidates historical-file decryption capability, root-key control, and recovery logic systematically at the client and root-key levels. As a result, client participation is no longer confined to the data plane, but is instead extended to the entire key lifecycle. This is particularly important in agricultural big data scenarios, because a large amount of data, such as field archives, remote sensing imagery, and breeding records, usually needs to be preserved across production cycles over the long term. Once the client cannot independently maintain the decryption capability for historical data, the continuous accessibility of existing agricultural data becomes more susceptible to the status of online services. Compared with multi-instance KMS, distributed KMS, or other multi-node collaborative schemes, the distinction of this work does not lie simply in improving the availability of online services, but rather in redefining their roles within the system: the service status of Native KMS directly affects the historical-file decryption path, whereas the role of the ISP in CD-KLM is restricted to the new-file creation phase, so the impact of its failure does not extend to the historical-file access path. This decoupling approach is better suited to agricultural scenarios in which distributed deployment, service fluctuations, and long-term operation coexist. Meanwhile, compared with file-level threshold recovery approaches, this work unifies the recovery object at the RootKey level, and experimental results show that the state overhead related to key recovery is significantly lower than that of file-level recovery models, indicating that the proposed scheme offers better scalability under large-scale file scenarios. This is especially important for agricultural big data platforms, where file scale continues to expand with production cycles and historical accumulation.

With regard to encryption and decryption performance, throughput fluctuations are more pronounced under small-file workloads, and at some data points CD-KLM is even close to Plain and locally outperforms Native KMS. This is because, under small-file conditions, fixed control overhead, cache state, and scheduling jitter are more likely to dominate end-to-end execution time. As file size increases, these disturbances are progressively amortized, and the performance gap of CD-KLM gradually converges and remains within a relatively low range, which also indicates that the proposed architecture is better suited to long-term, large-scale agricultural data storage scenarios.

Of course, CD-KLM still has certain limitations. First, new-file creation still depends on the ISP to provide random salt values, and therefore the stability of the ISP service is crucial to the encryption of new files. Second, since the RootKey is held by the client, the scheme places a relatively high level of trust in the client. Finally, the experiments in this paper are mainly conducted in a controlled Hadoop cluster environment, and further validation is still needed for more complex multi-tenant and cross-domain deployment scenarios.

7. Conclusions

This paper focuses on long-term secure storage scenarios for Hadoop-based agricultural big data and proposes a client-driven, server-lightweight-collaborative secure storage architecture, CD-KLM, to address several limitations of traditional online KMS-based architectures, including insufficient client participation in the key lifecycle, the risk that critical key corruption or loss may render the entire system undecryptable, and the tendency of file-level recovery states to become inflated under large-scale file scenarios. Built on a client-held RootKey, the proposed architecture combines a dynamic working-key derivation mechanism based on fid, epoch, and salt to enable on-demand generation of file-level working keys. At the same time, by unifying the recovery object at the root-key level, it constructs a key lifecycle management mechanism that supports both key recoverability and continuous decryptability of historical data. In particular, server-side collaboration is restricted to the new-file creation phase, during which collaborative salt values are issued only to authenticated clients, while the historical-file reading and decryption process no longer depends on a continuously online key service.

Based on the above design, this paper completes the system model, threat model, core workflows, and prototype implementation, and validates the security and performance of the proposed architecture in a Hadoop distributed environment. Experimental results show that, in terms of service availability, CD-KLM can still maintain correct decryption of historical files even when all ISP instances fail. In terms of key recovery, the system can stably recover the RootKey when the threshold condition is satisfied and maintains a 100% decryption correctness rate for historical files after multiple rounds of key rotation. In terms of control-plane overhead, the key storage overhead of CD-KLM is significantly lower than that of file-level recovery models. In terms of data-plane performance, although the system introduces a certain amount of encryption and decryption overhead, the overall impact on throughput remains low, and the system exhibits good stability under large-file workloads.

In summary, this paper demonstrates that CD-KLM can simultaneously achieve client-driven decryption control, root-key-level recoverability, and continuous decryptability of historical data without changing the original storage semantics of Hadoop/HDFS, thereby providing an implementable and verifiable architectural solution for the long-term secure storage of agricultural big data. Future work will further focus on adaptation and validation in more complex deployment environments, enhanced client-side key protection, and optimization of multi-version key management strategies.