Continuous Reorganization and Performance Preservation of Agent Memory Structure Under Distributed Change Environments

1. Introduction

Distributed changing environments refer to complex scenarios where the distribution of input data, task objectives, or environmental parameters dynamically shift during the operation of an agent. Typical applications include autonomous driving (sudden changes in road condition distribution), intelligent robot navigation (environmental feature drift), and distributed reinforcement learning (changes in multi-agent interaction patterns). In such environments, the traditional fixed memory structure of agents has inherent defects, failing to adapt to dynamic changes in distribution in real time. This leads to problems such as redundant memory accumulation, loss of key environmental information, and decreased generalization ability of decision models, directly resulting in decreased decision accuracy and increased response latency [1]. This research has clear theoretical and practical value: theoretically, it can improve the theoretical system of agent memory management in dynamic environments and fill the research gap in memory reorganization mechanisms with performance constraints; practically, it can improve the operational stability of agents in complex dynamic scenarios, providing technical support for engineering applications [2]. Existing memory recombination algorithms suffer from three core shortcomings: the recombination triggering mechanism relies on static thresholds, failing to perceive distribution changes in real time; the recombination process lacks performance constraints, easily leading to fluctuations in decision-making performance; and the memory unit retention/discarding judgment lacks an adaptive mechanism, making it difficult to balance memory efficiency and performance requirements.

2. Distribution-Aware Continuous Memory Reorganization Algorithm (DAMCR Algorithm)

The DAMCR algorithm uses “distribution awareness - dynamic recombination - performance feedback” as its core design logic [3]. Its core objective is to achieve real-time, smooth recombination of the agent’s memory structure under varying distribution environments, ensuring continuous stability of the agent’s decision-making performance while controlling recombination overhead. The algorithm adopts a modular architecture, including a distribution change perception module, a memory unit value evaluation module, a memory structure recombination execution module, and a performance constraint feedback adjustment module [4]. These modules work together to form a closed loop, achieving precise dynamic adaptation of the memory structure to the environment. The algorithm architecture diagram is shown in Figure 1.

2.1. Distribution Change Sensing Module

The core function of this module is to detect environmental distribution drift in real time, construct distribution change sensing factors, and trigger the memory reorganization mechanism [5]. First, the improved KL divergence is used to calculate the distribution drift of the input data, overcoming the defect of traditional KL divergence being sensitive to outliers. The improved KL divergence is defined as shown in Formula (1).

$$D_{KL}^{imp}(P\Vert Q)={\sum }_{i=1}^n P\left(x_i\right)\mathrm{l}\mathrm{n}{\displaystyle \frac{P\left(x_i\right)+\alpha }{Q\left(x_i\right)+\alpha }}-\beta \cdot \mathrm{v}\mathrm{a}\mathrm{r}\left(P-Q\right)$$

(1)

where $P\left(x_i\right)$ is the probability distribution of the current environmental data, $Q\left(x_i\right)$ is the baseline probability distribution (the distribution when the initial environment is stable), $n$ is the dimension of the data features, $\alpha$ is the smoothing coefficient (ranging from 0.01 to 0.05) to avoid the denominator being 0, $\beta$ is the outlier suppression coefficient (ranging from 1.2), and $\mathrm{v}\mathrm{a}\mathrm{r}(P-Q)$ is the variance of the difference between the two distributions, used to suppress the influence of outliers on the drift calculation.

Based on the drift, combined with the agent’s current decision error, a distribution change perception factor (DCF) is constructed, as shown in Formula (2), which comprehensively reflects the degree of environmental change and the impact on agent performance.

$$\mathrm{D}\mathrm{C}\mathrm{F}={\omega }_1\cdot {\displaystyle \frac{D_{KL}^{\mathrm{i}\mathrm{m}\mathrm{p}}(P\Vert Q)}{D_{KL}^{\mathrm{m}\mathrm{a}\mathrm{x}}}}+{\omega }_2\cdot {\displaystyle \frac{E-E_{\mathrm{m}\mathrm{i}\mathrm{n}}}{E_{\mathrm{m}\mathrm{a}\mathrm{x}}-E_{\mathrm{m}\mathrm{i}\mathrm{n}}}}$$

(2)

where ${\omega }_1,{\omega }_2$ are weight coefficients (satisfying ${\omega }_1+{\omega }_2=1$, with values ​​of 0.6 and 0.4 respectively), $D_{KL}^{\mathrm{m}\mathrm{a}\mathrm{x}}$ is the preset maximum drift threshold (with a value of 0.8), $E$ is the current decision error of the agent, and $E_{\min },E_{\max }$ are the minimum and maximum thresholds of the decision error respectively (calibrated according to the specific scenario). An adaptive recombination trigger threshold ${\theta }_{\mathrm{trig} }$ is set, and its value is dynamically adjusted with the stability of the environment, as shown in Formula (3). When $\mathrm{D}\mathrm{C}\mathrm{F}\ge {\theta }_{\mathrm{trig} }$, memory recombination is triggered.

$${\theta }_{\mathrm{trig} }={\theta }_0\cdot \left(1+0.3\cdot \exp \left(-\gamma \cdot \mathrm{stab} \right)\right)$$

(3)

where ${\theta }_0$ is the initial trigger threshold (value 0.5), $\gamma$ is the adjustment coefficient (value 0.7), and stab is the environmental stability coefficient, calculated from the drift variance over nearly 100 decision cycles.

2.2. Memory Unit Value Assessment Module

This module is responsible for assessing the value of each memory unit in the agent’s memory bank, calculating priority scores, and providing a basis for memory reorganization [6]. The value of a memory unit includes three dimensions: relevance, timeliness, and reliability. The comprehensive assessment function is shown in Formula (4).

$$V\left(m_k\right)={\lambda }_1\cdot R\left(m_k\right)+{\lambda }_2\cdot T\left(m_k\right)+{\lambda }_3\cdot K\left(m_k\right)$$

(4)

where $V\left(m_k\right)$ is the value score of the $k$ memory unit, ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are the value dimension weights (with values of 0.5, 0.3, and 0.2, respectively), $R\left(m_k\right)$ is the relevance score, $T\left(m_k\right)$ is the timeliness score, and $K\left(m_k\right)$ is the reliability score.

The relevance score reflects the degree of fit between the memory unit and the current environment distribution, and is calculated using cosine similarity, as shown in Formula (5); the timeliness score is calculated using a time decay model, as shown in Formula (6); and the reliability score is calculated based on the memory unit’s past decision contributions, as shown in Formula (7).

$$R\left(m_k\right)={\displaystyle \frac{\overrightarrow{m_k}\cdot \overrightarrow{P}}{‖\overrightarrow{m_k}‖\cdot \Vert \overrightarrow{P}\Vert }}$$

(5)

where ${\overrightarrow{m}}_k$ is the feature vector of the $k$ memory unit, $\overrightarrow{P}$ is the feature vector of the current environment distribution, and $\Vert \cdot \Vert$ is the L2 norm of the vector.

$$T\left(m_k\right)=\exp \left(-\tau \cdot \left(t_{\mathrm{now} }-t_k\right)\right)$$

(6)

where $\tau$ is the time decay coefficient (value 0.02), $t_{\mathrm{now} }$ is the current time, and $t_k$ is the generation time of the $k$ memory cell.

$$K\left(m_k\right)={\displaystyle \frac{1}{N_k}}{\sum }_{i=1}^{N_k} \mathrm{a}\mathrm{c}\mathrm{c}\left(m_k,x_i\right)$$

(7)

where $N_k$ is the number of times the $k$ memory unit participates in the decision-making process, and $\mathrm{a}\mathrm{c}\mathrm{c}\left(m_k,x_i\right)$ is the accuracy of the memory unit when participating in the $i$ decision-making process (1 for correct and 0 for incorrect).

2.3. Memory Structure Reorganization Execution Module

This module performs the retention, fusion, and discard operations of memory units based on their value scores, and introduces a buffer mechanism to achieve smooth reorganization.

Medium priority memory units adopt a feature fusion strategy to eliminate redundancy. The fusion formula is shown in (8), which merges multiple similar medium priority memory units into a new memory unit, retaining the core features.

$$m_{\mathrm{fusion} }={\displaystyle \frac{\sum _{k\in M_m} V\left(m_k\right)\cdot \overrightarrow{m_k}}{\sum _{k\in M_m} V\left(m_k\right)}}$$

(8)

where $m_{\mathrm{fusion} }$ is the new memory unit after fusion, $M_m$ is the set of medium-priority memory units, ${\overrightarrow{m}}_k$ is the feature vector of the $k$ memory unit in the set, and $V\left(m_k\right)$ is its value score. High-value features are retained by weighted averaging. A memory reorganization buffer pool $B$ is introduced to temporarily store the memory units to be reorganized [7].

$$\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}\left(B\right)=\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}\left(M\right)\cdot \left(0.1+0.05\cdot \mathrm{D}\mathrm{C}\mathrm{F}\right)$$

(9)

where $\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}\left(M\right)$ is the total capacity of the memory bank, and DCF is the distribution change sensing factor. The more drastic the environmental changes, the larger the buffer capacity, ensuring smooth recombination.

2.4. Performance Constraint Feedback Adjustment Module

This module monitors the agent’s decision-making performance indicators in real time, dynamically adjusts the memory recombination parameters, and forms a closed-loop adjustment to ensure stable performance during recombination. Decision accuracy Acc and recombination delay Lat are selected as core monitoring indicators, and the performance constraint function is constructed as shown in formula (10).

$$\mathrm{P}\mathrm{C}={\mu }_1\cdot \mathrm{A}\mathrm{c}\mathrm{c}-{\mu }_2\cdot {\displaystyle \frac{\mathrm{L}\mathrm{a}\mathrm{t}}{{\mathrm{L}\mathrm{a}\mathrm{t}}_{\mathrm{m}\mathrm{a}\mathrm{x}}}}$$

(10)

where PC is the performance constraint value, ${\mu }_1,{\mu }_2$ are weight coefficients (with values ​​of 0.8 and 0.2 respectively), and ${\mathrm{L}\mathrm{a}\mathrm{t}}_{\max }$ is the preset maximum recombination delay threshold (with a value of 50 ms). When $\mathrm{P}\mathrm{C}<{\mathrm{P}\mathrm{C}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\left({\mathrm{P}\mathrm{C}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right.$ is the minimum performance constraint threshold with a value of 0.75), the value assessment weight is dynamically adjusted, and the adjustment formula is shown in (11).

$${\lambda }_1^{\text{'}}={\lambda }_1\cdot \left(1+0.1\cdot {\displaystyle \frac{{\mathrm{P}\mathrm{C}}_{\mathrm{m}\mathrm{i}\mathrm{n}}-\mathrm{P}\mathrm{C}}{{\mathrm{P}\mathrm{C}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}}\right)$$

(11)

Wherein, ${\lambda }_1^{\text{'}}$ is the adjusted correlation weight. By increasing the correlation weight, the fit between the memory unit and the current environment is enhanced, the decision accuracy is improved, and the reconstruction latency is reduced, ensuring performance stability [8].

3. Experimental Simulation and Result Analysis

3.1. Experimental Environment Setup

The CPU used is an Intel Core i9-12900K (3.2GHz, 24 cores, 32 threads), the GPU is an NVIDIA RTX 3090 (24GB GDDR6X VRAM), the memory is 64GB DDR5 4800MHz, and the hard drive is a 1TB NVMe SSD, ensuring real-time performance for experimental data processing and simulation. The operating system is Ubuntu 20.04 LTS Server, the programming language is Python 3.9, the deep learning framework is PyTorch 1.12.0, the distribution drift simulation tool is PyDrift 0.5.2, the simulation environment is set up using OpenAI Gym 0.26.2, and data processing and plotting are performed using Matlab R2023a and Seaborn 0.12.2, ensuring reproducibility and accurate data calculation.

3.2. Baseline Algorithms

Eight baseline algorithms were selected for comparison with DAMCR, including three mainstream memory reorganization algorithms and five classic drift adaptation/memory management algorithms. All baselines used the same experimental conditions as DAMCR, differing only in their core mechanisms, as detailed below.

ST (Static Threshold Reorganization Algorithm): Reorganization is triggered by a fixed threshold, and old memory units are discarded based on their generation time; AR (Adaptive Random Reorganization Algorithm): The random reorganization frequency is adjusted according to environmental stability; DIR (Distribution Independent Memory Reorganization Algorithm): Random reorganization is performed at fixed time intervals; ADWIN (Adaptive Window Algorithm): Sliding window detects drift and deletes outdated windows; DDM (Drift Detection Method): Memory updates are triggered by an error rate threshold, and high-error memories are discarded; LRU (Least Recently Used Algorithm): Least recently used memories are discarded when memory is full; Reservoir Sampling: Old memory units are randomly replaced; PER (Preferred Experience Replay): Memory units are retained based on TD error priority, without multi-dimensional evaluation.

3.3. Main Results

The main experimental results are presented through quantitative tables and qualitative graphs. Four core evaluation indicators (decision accuracy, reorganization latency, performance fluctuation amplitude, and memory utilization) are used to verify the superiority of the DAMCR algorithm.

3.3.1. Quantitative Main Results

Table 1 shows the average performance (mean of 5 runs) of DAMCR and all baseline algorithms in two datasets and two drift scenarios. Bold text indicates the best performance, and italics indicate the second best. The results show that the DAMCR algorithm significantly outperforms all baseline algorithms in all four core metrics, achieving a decision accuracy of up to 92.3%, a recombination latency of only 12.7 ms, minimal performance fluctuation, and the highest memory utilization, validating its superiority in scenarios with changing distributions.

3.3.2. Qualitative Results

Figure 2 and Figure 3 are plotted based on the average of 5 random seed runs. The horizontal axis represents the number of training iterations (0-10000), and the vertical axis represents the corresponding evaluation metric. The shaded area represents the 95% confidence interval, reflecting the stability of the experimental results.

Figure 2 (Decision Accuracy Curve under Gradual Drift) shows that DAMCR maintains the highest decision accuracy and the slowest rate of decline during the 60% gradual drift process.

Figure 3 (Decision Accuracy Curve under Mutational Drift) shows that DAMCR’s accuracy decreases by ≤1.5% at the mutation point after 2000 iterations, and recovers within ≤500 iterations, outperforming all baselines.

4. Conclusions

To address the challenge of agent memory restructuring and performance preservation under varying distribution environments, this paper proposes the DAMCR algorithm, whose superiority is validated through system experiments (master experiment, ablation experiment, and statistical significance analysis). This algorithm achieves accurate distribution drift detection based on improved KL divergence, balances memory efficiency and performance through multi-dimensional memory evaluation and feature fusion, and ensures smooth and stable restructuring through buffering mechanisms and closed-loop feedback. In both gradual and abrupt drift scenarios, DAMCR significantly outperforms eight baseline algorithms in decision accuracy, restructuring latency, performance fluctuation, and memory utilization, with a maximum improvement in decision accuracy of 6.6% and a maximum reduction in restructuring latency of 55.1%. Current research has two limitations: it does not consider multi-agent collaborative memory restructuring, and its adaptability to extreme distribution drift needs improvement. Future research will extend this to multi-agent systems, optimize the drift perception factor, and conduct engineering validation in autonomous driving scenarios.