Memory in Vision-Language Models: Taxonomy, Mechanisms, and Applications

1. Introduction

Vision-language models (VLMs) have recently achieved remarkable progress across a broad spectrum of multimodal tasks [1], including image captioning [2], visual question answering (VQA) [3], video understanding [4], multimodal dialogue [5], and embodied intelligence [6]. By combining powerful visual encoders with large language models, modern VLMs exhibit remarkable capabilities in multimodal perception, reasoning, and generation. However, intelligent behavior extends beyond reasoning over the current input. Real-world applications require models to accumulate experiences across interactions, maintain long-term temporal consistency, personalize behaviors, adapt to dynamic environments, and continually integrate new knowledge. Despite recent advances in long-context modeling, most existing VLMs fundamentally operate in a stateless manner, where information processing is largely confined to the current input and a finite interaction context, limiting their ability to perform persistent reasoning over extended time horizons.

Memory has thus emerged as a fundamental paradigm for persistent multimodal intelligence. Rather than merely extending the effective context length, memory enables VLMs to accumulate experiences across interactions, retain knowledge over long time horizons, and continually adapt to dynamic environments. Inspired by cognitive memory theories [7,8,9] and facilitated by recent advances in neural architectures, a growing body of research has incorporated diverse memory mechanisms into VLMs, ranging from implicit parametric memory encoded within model representations to explicit external memory systems that organize and retrieve episodic, semantic, or multimodal knowledge. Representative approaches include key-value caches [10], latent memory representations [11], recurrent memory states [12], memory banks [13], external memory stores [14], and retrieval-augmented modules [15]. Beyond serving as passive repositories of information, these memory systems actively support information encoding, storage, retrieval, consolidation, updating, reflection, and forgetting, allowing models to selectively preserve useful experiences while incorporating new knowledge over time. Collectively, these capabilities enable VLMs to maintain long-term contextual consistency, retrieve relevant historical information, integrate multimodal experiences, and support continual reasoning, planning, and decision-making. Ultimately, memory transforms VLMs from stateless predictors into persistent multimodal systems, driving substantial improvements across applications like long-video understanding, multimodal dialogue, robotic manipulation, personalized assistants, and embodied agents.

Table 1. Positioning of our survey among related surveys on memory. •= explicit coverage; = partial or implicit coverage; ∘ = not covered.

	Memory Taxonomy (Sec. 2)				Vision & Multimodal		Evaluation & Analysis
Related Survey	Temporal Scope	Storage Location	Memory Entities	Memory Operations	Visual Memory	Multimodal Apps. (≥5)	Benchmarks / Metrics	Coupling Insights
Cognitive Architectures for Language Agents [16]	∘	∘	•	◐	∘	∘	∘	∘
Large Multimodal Agents [17]	∘	∘	◐	∘	•	◐	◐	∘
Agent Memory Mechanism [18]	◐	◐	•	•	∘	∘	∘	∘
World Models Survey [19]	∘	∘	◐	∘	◐	◐	∘	∘
Rethinking Memory in AI [20]	◐	◐	•	•	◐	•	∘	∘
Memory in the Age of AI Agents [21]	•	∘	•	◐	◐	∘	∘	∘
Memory for Autonomous LLM Agents [22]	•	•	◐	•	∘	∘	◐	∘
From Storage to Experience [23]	◐	∘	◐	∘	∘	∘	∘	∘
Video-LLM Survey [4]	∘	∘	∘	∘	•	∘	•	∘
Continual Learning of LLMs [24]	◐	∘	◐	•	∘	∘	•	∘
The AI Hippocampus [25]	◐	◐	•	◐	◐	•	∘	∘
Memory in Large Language Models [26]	◐	•	•	•	∘	∘	•	∘
Ours	•	•	•	•	•	•	•	•

Table 1. Positioning of our survey among related surveys on memory. •= explicit coverage; = partial or implicit coverage; ∘ = not covered.

	Memory Taxonomy (Sec. 2)				Vision & Multimodal		Evaluation & Analysis
Related Survey	Temporal Scope	Storage Location	Memory Entities	Memory Operations	Visual Memory	Multimodal Apps. (≥5)	Benchmarks / Metrics	Coupling Insights
Cognitive Architectures for Language Agents [16]	∘	∘	•	◐	∘	∘	∘	∘
Large Multimodal Agents [17]	∘	∘	◐	∘	•	◐	◐	∘
Agent Memory Mechanism [18]	◐	◐	•	•	∘	∘	∘	∘
World Models Survey [19]	∘	∘	◐	∘	◐	◐	∘	∘
Rethinking Memory in AI [20]	◐	◐	•	•	◐	•	∘	∘
Memory in the Age of AI Agents [21]	•	∘	•	◐	◐	∘	∘	∘
Memory for Autonomous LLM Agents [22]	•	•	◐	•	∘	∘	◐	∘
From Storage to Experience [23]	◐	∘	◐	∘	∘	∘	∘	∘
Video-LLM Survey [4]	∘	∘	∘	∘	•	∘	•	∘
Continual Learning of LLMs [24]	◐	∘	◐	•	∘	∘	•	∘
The AI Hippocampus [25]	◐	◐	•	◐	◐	•	∘	∘
Memory in Large Language Models [26]	◐	•	•	•	∘	∘	•	∘
Ours	•	•	•	•	•	•	•	•

Despite this rapid progress, the design space of memory mechanisms remains highly fragmented. Existing studies often investigate memory from task-specific [27,28], scenario-specific [29,30], or architecture-specific [31,32] perspectives, resulting in inconsistent terminology, diverse design choices, and limited understanding of how different memory paradigms relate to one another. Moreover, existing methods typically characterize memory through isolated viewpoints, such as temporal scope, storage location, memory representation, or retrieval strategy, without revealing how these design dimensions interact. Although several surveys have explored memory mechanisms in large language models, multimodal agents, and embodied systems [4,16,17,18,19,20,21,22,23,24,25], a systematic investigation dedicated to memory mechanisms in vision-language models remains lacking, even though representative memory-enabled VLMs have accumulated rapidly in recent years (Figure 2). Consequently, a unified understanding of memory design, architectural integration, evaluation methodologies, and application scenarios for VLMs has yet to be established.

To address this gap, we present the first comprehensive survey dedicated to memory mechanisms in VLMs. Rather than organizing prior work solely by architectures, tasks, or applications, we introduce a unified four-dimensional taxonomy, presented in Figure 1, that characterizes memory from four perspectives: when memory is maintained, where memory is stored, what information memory represents, and how memory is written, retrieved, updated, and utilized. Based on this taxonomy, we systematically review representative memory architectures and implementation mechanisms, including memory encoding, storage, retrieval, consolidation, and integration with multimodal backbones. We further summarize representative benchmarks and evaluation protocols, and analyze how memory facilitates a wide range of applications, including long-video reasoning, multimodal dialogue, robotics, and embodied intelligence. Finally, we discuss open challenges and future directions toward scalable, adaptive, and unified memory systems for persistent multimodal intelligence.

Figure 1. 4D Taxonomy of Memory Mechanisms

Figure 1. 4D Taxonomy of Memory Mechanisms

Figure 2. The proposed 4D taxonomy and the timeline of representative memory-enabled VLMs together with broader foundation-model milestones for context, illustrating the evolution of representative methods across the four taxonomy dimensions.

Figure 2. The proposed 4D taxonomy and the timeline of representative memory-enabled VLMs together with broader foundation-model milestones for context, illustrating the evolution of representative methods across the four taxonomy dimensions.

The main contributions of this survey are summarized as follows:

1.

4D Taxonomy. We propose a novel four-dimensional taxonomy that unifies existing memory mechanisms from the complementary perspectives of temporal scope, storage location, memory content, and memory operations, providing a principled framework for understanding and comparing existing approaches.

2.

Unified Perspective. We provide the first systematic survey dedicated to memory mechanisms in vision-language models, synthesizing fragmented research across architectures, applications, and evaluation protocols into a unified conceptual framework.

3.

Comprehensive Analysis. We systematically review representative memory architectures, implementation mechanisms, evaluation methodologies, and application domains, highlighting common design principles, advantages, and trade-offs across different memory paradigms.

4.

Challenges and Future Directions. We identify key challenges and emerging research opportunities toward scalable, adaptive, and cognitively inspired memory systems.

5.

Public Repository. To facilitate future research, we maintain a continuously updated repository containing representative publications, benchmarks, and resources related to memory-enhanced vision-language models.

The remainder of this survey is organized as follows. Section 2 introduces the proposed four-dimensional taxonomy, establishing a unified conceptual framework for understanding memory in VLMs. Section 3 reviews representative memory architectures and implementation mechanisms. Section 4 summarizes evaluation methodologies and benchmarks. Section 5 discusses representative application domains enabled by memory. Section 6 and Section 7 present current challenges and promising future research directions, respectively, followed by concluding remarks in Section 8.

2. 4D Taxonomy of Memory Mechanisms

Memory is the computational capability of an intelligent system to retain information beyond the computational step at which it is generated, such that the retained information can influence subsequent processing. Functionally, memory allows VLMs to carry forward representations of visual and textual observations, intermediate computational states, and prior multimodal experiences so that they can inform subsequent inference, reasoning, and action. The retained information may support transient reasoning within a single execution, episodic continuity across a bounded interaction, or persistent knowledge retention across sessions. It may reside in model-resident parameters, activations, latent states, and inference caches, or in independently addressable external stores such as memory banks, databases, and retrieval indexes. These temporal and architectural distinctions describe only part of the design space; our taxonomy therefore characterizes VLM memory jointly in terms of its temporal scope, storage location, memory entities, and memory operations.

From this perspective, memory should not be viewed merely as a storage component but as an evolving computational process. Regardless of its implementation, most memory systems involve a common lifecycle in which information is encoded from multimodal observations, selectively written into memory, organized or consolidated over time, retrieved when relevant to the current task, and subsequently updated, refined, or discarded through continual interaction. The effectiveness of a memory system therefore depends not only on what information is stored, but also on how memories are maintained, retrieved, and evolved throughout their lifetime.

To systematically compare diverse memory mechanisms without conflating distinct design choices, we organize existing approaches into a 4D taxonomy spanning four orthogonal dimensions:

1.

Temporal Scope (When): The temporal horizon over which stored information remains addressable.

2.

Storage Location (Where): Where memory is physically or logically stored.

3.

Memory Entities (What): The representational form of the stored information.

4.

Memory Operations (How): How memory is written, maintained, retrieved, and updated.

2.1. Temporal Scope (When)

Temporal scope, illustrated in Figure 3, classifies memory according to the boundary across which stored information remains available for reuse. Depending on when stored information ceases to be addressable, memory may be transient, remaining available only within a single inference; episodic, persisting across multiple turns or reasoning steps within a bounded interaction; or persistent, surviving beyond the current episode and remaining reloadable for future sessions.

2.1.1. Transient Memory

Transient memory refers to memory whose lifetime is limited to the current inference process. It functions as the model’s working memory, temporarily retaining task-relevant information so that intermediate computational states, visual evidence, and textual context remain available throughout ongoing reasoning. Unlike episodic or persistent memory, transient memory is discarded once inference concludes unless its state is explicitly preserved and transferred to a longer-lived memory system. Existing approaches primarily differ in how they construct, scale, and manage this temporary memory state.

Active Context Construction. Transient memory is formed through the representation and integration of multimodal information into the current model state. Early video-language systems explore this process through different strategies for representing and incorporating visual evidence. VideoBERT [33] places discretized video and language tokens within a bounded Transformer sequence, while ClipBERT [34] demonstrates that sparse clip sampling could improve the efficiency of visual evidence utilization. Flamingo [35] represents a prototypical example by integrating visual and textual information through interleaved few-shot prompts rather than an external memory store. Its Perceiver Resampler compresses visual features into a fixed number of visual tokens, which are incorporated into the language model through gated cross-attention during generation, making this visual evidence available only within the active context window and cache.

Active Context Scaling. Rather than introducing persistent memory, a subsequent line of work focuses on expanding transient memory capacity within a single inference. Representative efforts include adapting long-context language modeling to multimodal inputs [36], scaling active contexts toward million-token lengths [37], and jointly optimizing model architectures and system support to enable hour-scale video understanding [38,39]. Although these approaches substantially increase the capacity of transient memory during a single inference, enlarging the active context window alone does not establish episodic or persistent memory. Unless the resulting computational state is explicitly retained and made available for future interactions, the system remains stateless beyond the current inference. The same distinction applies to methods that maintain temporary buffers or online memory banks for streaming video analysis [40]; these mechanisms serve as transient workspaces rather than persistent stores, reflecting a broader trend in long-video modeling that substitutes expanded tracking for true memory retention [41].

Active Memory Management. Beyond scaling transient memory capacity, a critical challenge is determining which information should remain accessible within the limited active context during inference. Some recent works focus on improving the efficiency of transient memory by compressing multimodal KV caches [42], adopting modality-aware cache eviction strategies [43], and dynamically updating visual context during streaming generation [44]. These methods aim to preserve the most relevant information while operating within a finite context budget. Nevertheless, deciding what to retain remains challenging. Recent benchmarks show that simply increasing context length does not guarantee effective information utilization; models often struggle to retrieve and reason over relevant visual evidence when the active context becomes very large [45,46,47,48]. Consequently, transient memory management must balance two competing objectives: maximizing the amount of available evidence while maintaining efficient access to the information that matters most. Regardless of its capacity or management strategy, however, transient memory remains ephemeral and is discarded after inference unless its state is explicitly preserved and reused.

2.1.2. Episodic Memory

Episodic memory refers to interaction-specific information or internal state that captures context-rich experiences by preserving the sequential “what, when, and where” of an ongoing interaction. Unlike transient working context, which exists only during a single inference, episodic memory persists across multiple turns, observations, or decision steps within the same session, but is not intended to influence future independent interactions unless explicitly consolidated into persistent memory. Conceptually, it bridges transient context expansion and long-term knowledge retention.

Dialogue-bound State. In dialogue settings, episodic memory often takes the form of an evolving representation of grounded references and interaction history. VD-BERT [49] integrates image evidence with accumulated dialogue history, while multimodal dialogue state tracking explicitly models evolving visual states as new utterances and visual observations arrive [50]. RecFormer [51] maintains recurrent dialogue representations for visual dialogue, while MMCR [52] and Taking Notes [53] address multi-image, multi-turn scenarios in which previously observed visual information remains accessible across dialogue turns.

Streaming Video Sessions. Streaming video naturally exemplifies episodic memory because models must retain information from earlier video segments while processing subsequent segments within the same interaction. VideoLLM-online [54] maintains a rolling representation of previously observed video content to support later queries during an ongoing streaming session. Unlike transient context expansion, where information is discarded after a single inference, episodic memory preserves representations throughout the interaction. Mechanistically, this can be achieved either implicitly through recurrent state propagation or explicitly through event segmentation, which partitions continuous visual streams into discrete semantic episodes [55]. Sequence-focused benchmarks further evaluate whether models can reason over temporally ordered visual evidence within an episode rather than merely recognize isolated frames [56,57].

Embodied Episodes. Embodied agents provide another natural setting for episodic memory because completing a task often requires accumulating and binding information across multiple sequential decisions. During an episode, an agent must remember previous observations, actions, instructions, and spatial relationships to determine its next action. Vision-Dialog Navigation [58] is an early example in which the agent integrates language instructions, dialogue history, and visual observations throughout a navigation task. Subsequent approaches maintain similar episode-level state through navigation histories and topological representations, enabling agents to reason about previously visited locations and past interactions [59,60]. More recent embodied systems extend this idea beyond navigation to robotics and continuous environments, where agents continually reuse task-specific experiences during ongoing interactions [61,62,63].

Session closure provides the operational distinction between episodic and persistent memory. Dialogue histories, streaming summaries, and embodied trajectories remain episodic as long as they are confined to the temporal lifecycle of the current interaction. Once such information is deliberately consolidated, indexed, or retained to influence future independent sessions, it becomes persistent memory.

2.1.3. Persistent Memory

Persistent memory refers to information that survives beyond the temporal boundary of a single interaction and can be retrieved, reloaded, or reused to influence future independent sessions. Unlike episodic memory, which is tied to the current interaction, persistent memory remains available after session termination and supports continual accumulation of knowledge and experience.

Cross-session Retrieval. A common form of persistent memory stores raw or structured interaction history in an external repository that survives across sessions. Retrieval-augmented language models established this paradigm by retrieving textual knowledge from an external corpus rather than relying solely on parametric knowledge [15,64]. MuRAG [65] extended this paradigm to multimodal retrieval by incorporating image-text evidence into the external corpus. MemoryBank [13] further applied this principle to conversational agents, where previous interactions are archived and later retrieved to inform future conversations. These systems are persistent because the stored information remains reusable across decoupled interactions. Whether memory resides in an external database, a vector store, or directly within model parameters belongs to the storage substrate dimension rather than its temporal scope.

Cross-session Experience Accumulation. Persistent memory may also take the form of an evolving cognitive profile that accumulates experience over time instead of preserving raw interaction logs. Generative Agents [66] demonstrates this by storing observations, synthesizing them into higher-level reflections, and retrieving those reflections to guide planning and social interactions across simulated days. Unlike episodic memory, these abstract records survive session termination and continue to influence future behavior. Recent personalization systems make this distinction even more explicit by maintaining user preferences and behavioral profiles across highly disjointed interactions [67,68]. Similarly, Voyager [69] accumulates a reusable library of executable skills and experiences, enabling continual improvement over successive independent tasks rather than optimization within a single episode.

Embodied Long-term Memory. Embodied agents likewise benefit from retaining knowledge acquired through historical interactions to support a continuous operational lifetime. LifelongMemory [70] stores egocentric observations that can be retrieved to answer queries long after the original experiences have ended. This contrasts with episodic memory, where interaction-specific observations are discarded once the current task concludes. KARMA [71] explicitly combines short-term episodic and long-term persistent memory modules for embodied agents, while other robotic systems maintain reusable spatial and experiential memories to support later navigation, collaboration, and reasoning [72,73,74]. AtlasVA [75] similarly preserves visual skills as spatial heatmaps, visual exemplars, and symbolic representations that remain reusable across distinct embodied deployments.

Persistent memory does not need to reside in an explicit external repository. It can also be internalized through continual adaptation, localized fine-tuning, or dynamic parameter updates. Although such knowledge is encoded directly in model parameters rather than stored in a discrete memory structure, its defining property remains unchanged: it survives individual interactions and shapes future behavior. This further illustrates that temporal persistence (when) and storage substrate (where) are complementary but independent dimensions in our taxonomy.

2.2. Storage Location (Where)

Storage location partitions memory according to where remembered content resides and how it is accessed. Unlike temporal scope, which describes how long information persists, storage location distinguishes whether memory is maintained as model-resident state or as an independently addressable store. The same temporal behavior may therefore arise from different storage mechanisms: information that persists across requests may be encoded in model parameters or maintained in an external database, while short-lived state may appear either as intermediate activations or as temporary external records. Accordingly, we organize existing approaches into two categories. Internal memory stores information within the model’s computation, including parameters, hidden states, latent tokens, and inference caches that are accessed only through forward execution. External memory stores information in standalone repositories (such as documents, embeddings, records, and user profiles) that can be queried, updated, inspected, and replaced independently of the model itself.

Although storage location determines whether memory resides within or outside the model, each category can support diverse encoding substrates. Internal memory may be represented through parameters, hidden states, latent tokens, or activation caches, whereas external memory may be maintained as documents, embeddings, structured records, or knowledge graphs. These encoding choices affect properties such as interpretability, retrieval efficiency, and updateability, which are further analyzed in relation to memory entities and memory operations.

2.2.1. Internal Memory

Internal memory encompasses all forms of model-resident state. Although parameters, hidden activations, latent tokens, and KV caches differ in lifetime and functionality, they are unified by one property: remembered content exists only inside the model’s computation and is accessed through forward execution rather than through an independently queryable store. Internal memory may therefore be encoded through different substrates, including learned parameters, hidden states, latent representations, and activation caches.

Latent-token Memory. Learned latent tokens provide a compact internal representation for retaining task-relevant information within model computation. Perceiver [76] provides a canonical example by introducing a learned latent array that repeatedly attends to high-dimensional inputs while remaining entirely inside network computation. The latent array serves as an internal substrate that separates memory from raw observations without introducing an external store. BLIP-2 [77] extends this idea to VLMs by using learned query tokens to bridge a frozen image encoder and a frozen language model. Its Q-Former retains visual evidence as query-conditioned latent states rather than retrievable records. Flamingo [35] similarly employs resampled visual latents as internal state, while TokenLearner [78], learnable-memory image transformers [79], Perceiver IO [80], InstructBLIP [81], and mPLUG-Owl [82] instantiate related latent bottlenecks within model computation.

Hidden-state Memory. Sequential tasks require information from previous observations to remain available during subsequent computation. In these settings, hidden activations naturally function as memory by propagating contextual information across time:

$$m_t=h_t,h_t=f_{\theta }(h_{t-1},x_t),$$

(1)

where the hidden state $h_t$ summarizes previous observations while incorporating the current input $x_t$. This formulation captures memory mechanisms employed by MART [83], Multimodal Transformer with Variable-Length Memory [84], and Perceiver IO [80], where memory is carried through evolving hidden representations rather than a separate storage module.

Parametric Memory. Some methods preserve knowledge by modifying and protecting model parameters instead of maintaining explicit retrievable memories. Learning Without Forgetting for Vision-Language Models [85] treats previously acquired visual-language knowledge as parametric state that should survive continual updates. LLaMA-Adapter [86] stores task adaptation within lightweight attention and adapter parameters, while Memory-Space Visual Prompting [87] introduces a learned memory space embedded within model parameters for efficient adaptation. Although these mechanisms often support persistent or continually evolving memory, the remembered content remains internal because it is encoded in model weights and accessed only through model computation.

KV-cache Memory. During long-context inference, key-value (KV) caches constitute another form of internal memory. Unlike external retrieval databases, cached keys and values exist only as intermediate activation states generated during inference. AKVQ-VL [88] illustrates this category by quantizing cached keys and values while preserving their role within attention computation. Likewise, KV-cache compression and eviction methods improve inference efficiency by reorganizing or compressing activation states without introducing an independently addressable memory store [42,43]. Internal caches therefore improve computational efficiency while remaining tightly coupled to model execution, making their contents difficult to inspect, edit, or reuse outside the forward pass.

2.2.2. External Memory

External memory stores remembered content outside the model itself. Rather than encoding information in activations or parameters, these approaches maintain independently addressable repositories that can be queried, updated, inspected, and replaced without modifying the model’s internal computation. External memory may take diverse forms, including indexed retrieval corpora, structured knowledge repositories, and experience records.

Retrieval-Augmented Memory. Retrieval-augmented approaches explicitly separate memory storage from model computation by coupling a generator with an external memory store. REALM [64] established an early non-parametric memory in which retrieved documents complemented model parameters. RAG [15] subsequently became the canonical framework by retrieving passages from an external corpus instead of relying solely on parametric knowledge. Because retrieved content resides outside the model, the underlying memory can be refreshed, re-indexed, or replaced without changing generator weights. This retrieval process can be abstracted as

$$m_t=\mathrm{Retrieve}(q_t,\mathcal{M}),$$

(2)

where a query $q_t$ accesses an external memory store $\mathcal{M}$. MuRAG [65] extends this paradigm to multimodal image-text retrieval, while REVEAL [89] further incorporates multi-source multimodal knowledge stores during pretraining. More recent benchmarks and generalized retrieval frameworks continue to reinforce this separation between external storage and model computation [90,91].

Structured Knowledge Memory. For vision-language reasoning, external memory may consist of symbolic facts, entities, relations, and structured multimodal knowledge repositories rather than hidden activations or model parameters. KRISP [92] and KAT [93] illustrate explicit knowledge storage for visual question answering and vision-language transformers. Retrieval-Augmented Multimodal Language Modeling [94] similarly treats retrieved multimodal examples as external context for language modeling. Knowledge graphs, symbolic reasoning systems, and structured information repositories instantiate the same principle: remembered knowledge resides in an external store that can be queried independently of the model’s forward computation [95,96,97,98].

Open-domain Retrieval Memory. External memory can also originate from continuously evolving information sources beyond curated retrieval databases. SearchLVLMs [99] accesses the web as an up-to-date knowledge source for LVLMs. V* [100] treats visual search results as externally retrieved evidence. R4 [101] and VisRAG [102] further extend retrieval to spatio-temporal environments and multimodal documents, while End-to-End Optimization for Multimodal RAG [103] jointly optimizes retrieval and generation without altering the fundamental separation between external storage and model inference.

Experience Memory. Long-lived agents introduce another motivation for external storage: remembered experiences must remain editable, inspectable, and reusable long after an interaction has ended. LifelongMemory [70] stores egocentric experiences as external records rather than latent activations. MemoryBank [13] similarly maintains conversational histories as an independently accessible memory store. Although these systems are categorized as persistent memory under the temporal dimension, their defining property here is that remembered content exists as external data that can be queried independently of the model’s forward computation.

2.3. Memory Entities (What)

Memory entities describe the representational form of remembered information in a VLM, independent of where it is stored or how long it persists. These entities can be organized along a continuum of transformation, ranging from high-fidelity observations to increasingly processed representations. As summarized in Figure 4, we distinguish four complementary forms: raw observations, which preserve input evidence with minimal processing; structured episodic records, which organize observations into temporally or structurally defined events; abstract semantic knowledge, which captures reusable concepts, facts, and preferences; and latent representations, which encode information as distributed neural states optimized for computation. These entity types are not mutually exclusive, and modern multimodal foundation models often combine multiple forms simultaneously.

2.3.1. Raw Observations

Raw observations preserve multimodal evidence in a form that remains close to the original input, including image frames, video clips, audio tokens, text tokens, and patch-level visual streams before semantic abstraction or latent compression. Their defining characteristic is fidelity to the incoming signal rather than explicit structure or semantic interpretation. Unlike higher-level memory entities, raw observations preserve information before decisions about summarization, compression, or abstraction are made, making them valuable whenever downstream reasoning requires access to original sensory evidence.

Direct Sequence Storage. To mitigate premature information loss, several VLMs retain visual and textual tokens directly within the context window rather than converting observations into higher-level memory records. VideoBERT [33] exemplifies this design by representing multimodal experience as a discretized video-language token sequence, allowing visual units and words to occupy a shared Transformer sequence before any durable record or semantic abstraction is formed. ClipBERT [34] similarly treats sparse video clips as the primary visual evidence for downstream reasoning, while HERO [104] preserves raw video-language streams through hierarchical pretraining. Flamingo [35] interleaves images directly into the prompt before resampling, and Long Context Transfer [36] propagates visual tokens across extended contexts without first converting them into structured memories. More recent long-context VLMs extend the same principle to much larger temporal scales. LongVITA [37] preserves multimodal token streams across long contexts, Video-XL [39] retains hour-scale frame evidence, and LongVILA [38] keeps large collections of frames directly addressable as raw video evidence.

Streaming Raw Observations. In online and streaming environments, incoming observations may arrive faster than their long-term importance can be reliably determined, making raw buffers an essential intermediate memory form. TimeChat [105] maintains high-fidelity video observations for temporal grounding; VideoLLM-online [54] processes streaming chunks sequentially as raw evidence; and StreamChat [44] maintains an active rolling visual context during inference. Although later stages may compress, retrieve, or discard these observations, the underlying memory entity remains tied to the original sensory input.

2.3.2. Structured Episodic Records

Structured episodic records organize past multimodal experiences into bounded and addressable units such as dialogue turns, state slots, route trajectories, graph nodes, video segments, and event logs. Unlike raw observations, these memories introduce explicit temporal, spatial, or interaction structure that allows individual experiences to be selectively accessed. An observation becomes episodic once it is associated with a specific event, interaction, or temporal context rather than being retained as unstructured input evidence.

Dialogue Records. Dialogue-oriented VLMs naturally organize conversational history into bounded multimodal records instead of maintaining a flat prompt history. VD-BERT [49] exemplifies this design by jointly encoding dialogue history and visual context into a unified vision-dialogue representation that subsequent turns can systematically revisit. Multimodal dialogue state tracking [50] further formalizes this by introducing structured visual-object slots that evolve across turns, effectively transforming the conversational context into an explicit episodic state rather than raw dialogue text. This design choice, preserving structured interaction history over raw context windows, is mirrored across several recent paradigms. These include context-aware visual dialogue reasoning [106], recurrent dialogue history in RecFormer [51], cross-modal conversational representations [107], focused dialogue notes in Taking Notes [53], and multimodal turn-level context in MMCR [52].

Navigation Records. Embodied navigation systems similarly convert continuous perception into route-indexed records that associate observations with positions, actions, and traversal history. Vision-Dialog Navigation [58] stores route episodes that combine visual observations with conversational guidance, while history-aware navigation methods [59] explicitly maintain trajectory histories for subsequent planning. Dual-scale graph transformers [60] organize navigation experience into complementary global and local route representations, and ETPNav [63] maintains topological plans that remain tied to a specific traversal episode. More broadly, embodied-memory systems preserve action trajectories, object-goal routes, spatial-cognition memories, and robot spatiotemporal logs as reusable episodic records [61,62,72,108].

Streaming Records. Streaming video architectures transition raw observations into ordered session records once data frames acquire a stable temporal identity. Rather than processing isolated clips, these systems format historical context sequentially. For instance, VideoLLM-online [54] maintains rolling, causally linked video chunks across a continuous live session, while (author?) [55] propagate condensed chunk-level summaries recursively across extended video horizons to form a coherent chronological session log. Similarly, VideoLLaMB [109] introduces a recurrent memory bridge layer combined with a semantic segmentation algorithm to structurally link successive video segments without losing historical context. This requirement for ordered structure is highlighted by the Mementos framework [56], which evaluates a model’s sequential visual reasoning strictly across temporally ordered image sequences where the explicit order defines the episodic memory. By shifting from ephemeral inputs to ordered series, these approaches construct a coherent session-level archive of the stream.

Agent Logs. Long-lived autonomous agents frequently structure historical context into event-centered records that pair raw observations with the discrete interactions or metadata that generated them. Rather than treating history as an undifferentiated stream, these architectures serialize memory as discrete episodic units. For instance, LifelongMemory [70] indexes continuous egocentric video feeds by extracting localized visual descriptions into distinct, searchable text-based event traces. Similarly, Generative Agents [66] structures an agent’s memory stream as a ledger of timestamped event observations, preserving the chronological ground-truth before triggering higher-level reflection or planning layers. This pairing of raw interaction histories with structural metadata is also utilized by MemoryBank [13], which buffers sequential user-agent dialogues alongside personality and mood tracking before applying cognitive forgetting models to abstract the history into long-term user profiles. By anchoring memory to specific situational contexts, agent logs preserve the raw environmental history in a queryable, object-oriented format.

2.3.3. Abstract Semantic Knowledge

Abstract semantic knowledge stores information generalized beyond individual experiences, including facts, concepts, rules, preferences, summaries, and symbolic relations. Unlike episodic records, which preserve information about specific events, semantic knowledge represents reusable information whose validity is independent of the original observation or interaction from which it was derived. Its defining characteristic is generalization and cross-context applicability rather than event-specific recollection.

Retrieval Knowledge Stores. Retrieval-augmented systems represent semantic memory by storing interpretable knowledge objects, such as documents, text passages, image-text pairs, or structured knowledge bases, rather than opaque latent representations. Text-centric foundations such as RAG [15] and REALM [64] pioneered retrieving discrete textual passages as reusable semantic knowledge. Modern vision-language systems extend this paradigm to multimodal knowledge sources. Reveal [89] integrates image-text corpora and external knowledge into a unified multimodal memory for vision-language pretraining, while MuRAG [65] extends retrieval to multimodal image-text knowledge entries. Retrieval-augmented multimodal language modeling [94] further stores retrieved examples as explicit semantic priors. SearchLVLMs [99] grounds generation using real-time web knowledge, whereas VisRAG [102] retrieves complete multimodal documents as reusable knowledge objects rather than relying solely on textual evidence. More recent multimodal RAG frameworks further generalize this paradigm by integrating graph-structured memories, heterogeneous knowledge sources, and unified retrieval engines for multimodal knowledge access [91,97,98].

Symbolic and Explicit VQA Memories. Knowledge-based VQA makes semantic memory explicit by leveraging structured facts, symbolic relations, and localized knowledge evidence that cannot be reliably inferred from visual observations alone. KRISP [92] exemplifies this paradigm by linking detected visual concepts with external symbolic knowledge graphs to support open-domain reasoning. Similarly, KAT [93] augments vision-language reasoning by retrieving and integrating explicit textual knowledge aligned with visual concepts. Related approaches further enrich VQA by representing context as interpretable evidence rather than relying entirely on opaque latent features. These include architectures that incorporate plug-and-play textual knowledge blocks into the prompt space [95], systems that optimize visual prompting through localized external references [96], and frameworks that maintain explicit visual references through guided search and region-level retrieval mechanisms [100]. By representing memory through discrete knowledge tokens, symbolic relations, or explicit visual references, these approaches make stored context more interpretable and auditable than purely latent representations.

Personalized Knowledge. Long-lived agents often transform repeated interactions into durable knowledge about users, environments, and social conventions. MERLOT RESERVE [110] learns script-like multimodal knowledge that supports reasoning beyond individual video clips. Generative Agents [66] converts accumulated event traces into higher-level semantic reflections, while MemoryBank [13] abstracts interaction histories into persistent user facts and preferences. Similarly, AUGUSTUS [67] maintains contextual user profiles, and Ella [68] transforms embodied social experiences into long-term personalized knowledge.

Semantic Knowledge Maintenance through Editing and Pruning. Post-training knowledge maintenance further reveals the properties of semantic memory by examining how stored visual-language associations can be modified, removed, or preserved after initial training. Multimodal knowledge editing [111] directly targets specific semantic associations in VLMs while minimizing unintended changes to unrelated concepts. Complementing editing approaches, unified knowledge-maintenance frameworks [112] study how semantic capabilities encoded in model parameters can be selectively pruned, evaluated, and recovered after intervention. Together, these paradigms demonstrate that semantic associations are not fixed after training but can be explicitly targeted, refined, and maintained through controlled post-training operations.

2.3.4. Latent Representations

Latent representations store remembered information as distributed neural states rather than directly interpretable observations, episodic records, or symbolic knowledge. Unlike the previous categories, which characterize memory according to the semantic nature of stored information, latent representations describe the computational form through which information is maintained. Their defining property is representational opacity: downstream computation can exploit these states, but the preserved information is not directly accessible as a human-readable record or explicit fact. Latent representations include encoder outputs, learned tokens, query states, compressed activations, recurrent hidden states, and inference-time cache tensors.

Latent Arrays. Latent-array architectures mediate high-dimensional inputs through compact sets of learned memory slots that serve as intermediate computational states. Perceiver IO [80] exemplifies this design by introducing a structured latent array that decouples computational complexity from input size, mediating arbitrary modalities through a fixed number of learned latent variables. Rather than retaining dense, uncompressed visual tokens, TokenLearner [78] dynamically constructs a sparse set of learned visual tokens from image and video features to provide compact representations for downstream processing. Related approaches similarly employ learned vectors as task-specific latent memory states, including the original Perceiver architecture [76], learnable-memory image transformers [79], and Memory-Space Visual Prompting [87], which parameterizes prompt information within a highly compressed latent memory space.

Query-token Visual States. Many vision-language models maintain visual information through learned query tokens that bridge vision encoders and language models. Flamingo [35] uses a Perceiver Resampler to transform variable-length visual features into a small, fixed set of interface tokens consumed by the language model. BLIP-2 [77] similarly introduces a Q-Former to extract compact visual query states from a frozen image encoder. Building on this paradigm, InstructBLIP [81] conditions these visual queries on task instructions to produce instruction-aware latent tokens, while mPLUG-Owl [82] employs a visual abstractor module to compress dense visual features into a compact sequence of modular latent tokens.

Compressed Video Latents. Long-video understanding increasingly relies on compressing dense frame-level observations into compact latent states rather than retaining all raw visual tokens. MovieChat [113] exemplifies this paradigm by compressing dense video sequences into sparse token representations for efficient long-video reasoning. Similarly, LongVU [114] stores adaptive spatiotemporal representations that reduce spatial and temporal redundancy while preserving long-range video dependencies, whereas VideoChat-Flash [115] introduces a hierarchical video compression strategy that transforms long-duration visual context into compact latent representations for extended video understanding.

Recurrent Latent Bridges. Recurrent memory mechanisms propagate latent states across successive observations, allowing models to access historical context without replaying the complete input sequence. Variable-Length Memory architectures [84] instantiate this design by encoding embodied experiences into adaptive latent states for autonomous navigation. Similarly, MA-LMM [40] maintains long-video histories through a memory-augmented latent bank that supports information accumulation over extended temporal horizons, while VideoLLaMB [109] extends this recurrent latent bridging paradigm to long-context video-language modeling over extended temporal sequences.

Cache Representations. Transformer-based key-value (KV) cache mechanisms expose latent memory during inference because the stored information consists of activation tensors rather than interpretable records. Cache-centric frameworks explore how these latent states can be selectively retained, evicted, compressed, or quantized to extend multimodal context windows [42,43,88]. Although these caches do not correspond to explicit semantic entities, they function as transient memory states that preserve task-relevant intermediate information required for continued sequence generation and multimodal inference.

2.4. Memory Operations (How)

Memory operations describe the actions through which a VLM interacts with memory, independent of the memory content and storage location. Although memory architectures differ substantially across VLMs, most systems follow a common operational lifecycle consisting of four fundamental operations (Figure 5): memory writing, which admits new information into a memory-bearing representation; memory consolidation, which reorganizes, compresses, or stabilizes existing memories; memory retrieval, which selects and accesses relevant information for the current task; and memory updating, which revises, strengthens, weakens, or removes previously stored information. Together, these operations determine not only what information is retained, but also how efficiently and effectively memory supports long-term multimodal reasoning.

2.4.1. Memory Writing

Memory writing is the operation through which new information becomes part of a memory system. Unlike retrieval or updating, which operate on existing memories, writing determines whether newly observed information should be preserved in the first place. Since memory capacity is finite and not every observation contributes equally to future reasoning, writing mechanisms must address two fundamental questions: when information should be written and how it should be incorporated into a memory-bearing representation. Different writing policies therefore directly influence the quality, diversity, and long-term utility of stored memories, as information that is never written cannot be consolidated, retrieved, or updated later.

Depending on the architecture, writing may produce transient working memory, episodic records, or persistent knowledge. Although implementations differ considerably, existing approaches can be broadly categorized according to how information is admitted into memory.

Encoding Controllers. One common writing strategy employs dedicated encoding controllers that transform incoming multimodal observations into memory-bearing representations suitable for subsequent computation. Rather than directly storing raw visual or textual inputs, these architectures progressively integrate multimodal evidence into latent states that preserve information throughout the reasoning process. HERO [104] provides an early example of hierarchical video-language representation learning by integrating clip-level and video-level contexts into unified latent representations for downstream grounding tasks. Although these representations are not maintained as persistent external memory, they nevertheless function as memory-bearing computational states that preserve information across multiple stages of inference. MART [83] extends this idea through a recurrent memory mechanism that continually updates latent representations during video caption generation, enabling sequential visual observations to accumulate over time. More recent long-video systems further develop this paradigm by progressively admitting streaming observations into memory representations. For example, MA-LMM [40] maintains long-video context through a memory-augmented architecture, VideoLLM-online [54] continuously converts streaming video segments into reusable memory representations, while TimeChat [105] and related streaming frameworks [55] progressively incorporate temporal observations for subsequent reasoning.

Query-token-based Admission. Query-token mechanisms provide a compact interface for writing visual information into memory rather than storing raw, dense feature maps. By projecting high-dimensional visual observations into a small set of learned latent tokens, these methods preserve task-relevant context while ensuring compatibility with LLMs. Foundational architectures like the Perceiver [76] and TokenLearner [78] established this paradigm by using attention-based bottlenecks to extract compact tokens from dense visual features. Building on this principle for VLMs, Flamingo [35] introduced a cross-modal resampler to map visual inputs into latent representations. Similarly, BLIP-2 [77] utilized Q-Former queries to selectively extract task-relevant visual states for frozen language models, with InstructBLIP [81] further conditioning this admission process on task instructions. Together, these approaches demonstrate how incoming visual evidence can be efficiently transformed into compact, accessible memory states.

Recording Interaction History. Beyond encoding perceptual observations, many memory systems explicitly record information generated during interaction. Rather than preserving raw inputs alone, these approaches transform conversations, user feedback, planning trajectories, or behavioral experiences into reusable memory entries that can influence future interactions. MemoryBank [13] exemplifies this process by explicitly writing conversation-derived memories into a long-term memory structure to support personalized response generation across sessions. Generative Agents [66] similarly records observations, reflections, and behavioral experiences into persistent memory streams that can later support planning and decision-making. Taking Notes [53] extends this paradigm to multimodal dialogue by treating automatic note generation as an explicit writing mechanism that transforms long interaction histories into concise and reusable summaries. Multimodal dialogue state tracking frameworks [50] further formalize interaction-time writing by encoding turn-level evidence into structured slots that represent evolving dialogue states.

Implicit Parametric Writing. Not all memory writing creates explicitly addressable memory entries. Some approaches instead incorporate newly acquired information directly into model parameters or latent representations through parameter-efficient adaptation mechanisms. In these systems, writing modifies the computational behavior of the model itself rather than constructing an external memory store. Adapter-based methods, prompt tuning, and related parameter-efficient learning approaches [86,87,116] exemplify this implicit writing paradigm, where stored information becomes distributed across model parameters and can only be recalled indirectly through subsequent inference.

Regardless of implementation, memory writing establishes the foundation upon which all subsequent memory operations depend. Decisions made during writing determine not only what information becomes available for future retrieval, but also the effectiveness of later consolidation, updating, and long-term reasoning. Consequently, writing policies must balance information preservation against storage efficiency, ensuring that important experiences are retained without overwhelming finite memory resources.

2.4.2. Memory Consolidation

Unlike memory writing, which determines how newly observed information enters a memory system, memory consolidation operates on information that has already been retained. Its objective is to improve the long-term utility of existing memories by reorganizing them into forms that are more compact, structured, stable, and efficient to retrieve, thereby increasing scalability without sacrificing information useful for future reasoning. Inspired by memory consolidation in cognitive psychology, where newly acquired experiences are gradually transformed into stable long-term memories [7,8,9], computational consolidation extends beyond simple stabilization to encompass a broader family of operations, including compression, abstraction, hierarchical organization, semantic indexing, and graph restructuring. Although these approaches differ in implementation, they share a common objective: improving the accessibility, efficiency, and long-term usability of stored information as memory continues to grow.

Compression-based Consolidation. One of the most common consolidation strategies reduces redundancy within existing memory representations while preserving information most relevant for future reasoning and retrieval. Rather than modifying incoming observations during memory writing, these methods operate on previously stored representations to improve storage efficiency, increase memory capacity, and extend the effective reasoning horizon under constrained computational budgets. Such approaches typically consolidate memory by selecting informative observations, merging redundant representations, summarizing dense sequences, or transforming fine-grained token-level features into compact memory units. MovieChat [113] provides a representative example by organizing long-video understanding through a two-stage memory mechanism that aggregates dense visual observations into compact short-term and long-term memory representations. LongVU [114] similarly introduces adaptive spatiotemporal compression by pruning redundant visual tokens after encoding, while VideoChat-Flash [115] further extends this principle through hierarchical token compression across clip- and video-level representations. Although these methods differ in implementation, they share a common objective: reorganizing previously encoded information into more compact representations that preserve essential semantics while reducing storage and computational costs. Compression therefore functions not as an admission mechanism, but as a post-writing maintenance process that improves the long-term utility of stored memory.

Transient Memory Maintenance. Consolidation is not restricted to long-term memory stores. During inference, active working memory and KV caches grow continuously and must be regulated dynamically. Unlike long-term representation consolidation, which restructures stored memories after encoding, transient memory maintenance operates online throughout execution by continuously adapting the content and size of active memory. It regulates context by retaining informative states, compressing redundant representations, refreshing evolving information, and discarding entries with diminishing utility. The objective is not indefinite preservation, but maximizing the effectiveness of bounded working memory under strict computational and context constraints. Although these operations occur during inference rather than after explicit memory formation, they serve a similar consolidation function by allocating limited memory resources toward the representations most valuable for ongoing reasoning.

Online memory systems provide representative examples of this paradigm. MA-LMM [40] maintains an adaptive memory bank that continuously updates historical visual representations during long-video understanding, enabling previously observed content to remain accessible without repeatedly processing the entire video. Streaming VLMs such as StreamChat [44] and VideoLLM-online [54] similarly refresh memory representations as new visual observations arrive, ensuring that active memory evolves together with the incoming video stream. More recently, multimodal KV-cache management has emerged as an effective mechanism for maintaining transient memory under limited computational budgets. These methods typically estimate the importance of cached tokens through attention patterns, modality-aware heuristics, or learned retention policies, and then apply eviction, compression, or quantization to preserve critical information while reducing memory usage. LOOK-M [42] formulates cache management as an importance-aware retention problem by selectively preserving informative visual tokens, while MadaKV [43] introduces modality-aware cache eviction strategies and AKVQ-VL [88] further improves runtime efficiency through adaptive cache quantization. Although these methods differ in implementation, they share the common objective of maintaining an informative and computationally efficient working memory throughout long-context inference.

Structural Organization and Indexing. As memory repositories continue to grow, efficient memory utilization depends not only on what information is retained, but also on how it is organized. Simply accumulating memories eventually leads to inefficient retrieval, increased redundancy, and poor scalability. Consequently, many memory systems perform structural consolidation by transforming previously stored information into searchable, semantically meaningful structures that facilitate efficient future access. Unlike compression-based consolidation, which reduces redundancy within memory representations, structural organization preserves memory content while improving its accessibility through indexing, hierarchical organization, graph construction, embedding-based clustering, or dedicated retrieval databases. Through these operations, isolated memory entries are transformed into reusable repositories that can support scalable retrieval across future tasks.

Representative retrieval-augmented systems illustrate this organizational perspective. REALM [64] constructs an indexed retrieval corpus by associating stored documents with searchable representations, enabling efficient access to external knowledge during inference. RAG [15] similarly organizes large document collections into retrieval-ready indexes that can be queried across different tasks. Extending these concepts to multimodal settings, MuRAG [65] organizes textual and visual evidence into a unified retrieval memory, and REVEAL [89] maintains large-scale visual-linguistic knowledge within an external store for scalable access. Rather than emphasizing retrieval in isolation, these architectures demonstrate how memory can be consolidated into structured, reusable repositories over extended interactions. Recent frameworks push this paradigm further through increasingly sophisticated organization mechanisms: VisRAG [102] indexes document images directly as visual retrieval units to avoid information loss introduced by OCR-based conversion, whereas MegaRAG [97] and RAG-Anything [91] generalize structural consolidation to graph-based heterogeneous stores capable of representing relationships among text, images, tables, and auxiliary modalities within unified retrieval frameworks.

From the perspective of memory operations, structural organization transforms memory from a collection of isolated records into an organized knowledge repository that supports efficient and scalable retrieval. As memory repositories continue to expand in both size and modality, designing effective organizational structures becomes increasingly important for maintaining retrieval quality and long-term memory scalability.

Consolidating Embodied Experience. Embodied agents continuously accumulate vast perceptual observations, interaction histories, spatial trajectories, and behavioral experiences throughout extended operations. Retaining these experiences as isolated episodes is neither scalable nor effective for downstream decision-making. Consequently, embodied memory systems consolidate transient experiences into structured representations that preserve context critical for future planning, reasoning, and interaction. Such consolidation typically involves extracting meaningful events from continuous observations, summarizing past interactions, constructing spatial or semantic representations, and transforming individual experiences into reusable behavioral knowledge. For personalized interactive agents, MemoryBank [13] and LifelongMemory [70] maintain user histories and multimodal behavioral traces across multiple sessions, transforming isolated interactions into persistent profiles that support long-term assistance. In spatial navigation, frameworks like Mem4Nav [108] and KARMA [71] consolidate streaming observations, spatial trajectories, and environmental maps into structured episodic representations that can be efficiently queried during localization and task execution, avoiding reconstruction from raw observations. Beyond preserving individual episodes, recent architectures consolidate experience into higher-level behavioral abstractions. AtlasVA [75], for instance, decouples memory into complementary visual and symbolic layers, distilling trajectory histories into hierarchical skill atlases for long-horizon planning. Similar principles underpin modern robotic architectures, including ReMEmbR [72], RoboMemory [74], and RoboOS-Next [73]. These frameworks integrate multimodal interaction histories, spatio-temporal observations, and behavioral traces into structured stores designed for sustained manipulation and continuous control. Unlike compression-based consolidation, which primarily improves storage efficiency, or structural organization, which facilitates scalable retrieval, embodied memory consolidation transforms low-level interaction histories into increasingly abstract and reusable knowledge that enables agents to accumulate experience and improve behavior over extended lifetimes.

Taken together, these consolidation strategies demonstrate that effective memory depends not only on what information is stored, but also on how previously acquired memories are transformed and maintained over time. Whether through compression of redundant representations, structural organization for efficient access, or abstraction of embodied experiences into reusable knowledge, consolidation enables memory systems to remain scalable while preserving information most valuable for future reasoning, interaction, and decision-making.

2.4.3. Memory Retrieval

Memory retrieval is the operation through which a model accesses previously stored information during reasoning. Unlike memory writing and consolidation, which determine how information enters and is maintained within memory, retrieval governs which subset of stored knowledge becomes available for the current task. Since only a small fraction of accumulated memory can typically participate in inference under finite computational and context budgets, retrieval must selectively identify information that is both relevant and informative while avoiding unnecessary or distracting evidence. Consequently, retrieval largely determines whether stored memories can be effectively translated into improved reasoning and decision-making.

Depending on the memory architecture, retrieval may involve searching external databases, recalling episodic experiences, accessing structured knowledge, or attending over internal memory representations. Although these mechanisms differ substantially in implementation, they share the common objective of selecting useful information from potentially large and heterogeneous memory repositories.

Search-based Retrieval. Search-based retrieval formulates memory access as a query-driven search problem over external memory repositories. Given the current multimodal context, the model first constructs a retrieval query and then selects the most relevant information from large collections of textual, visual, or multimodal knowledge before incorporating the retrieved evidence into subsequent reasoning. This paradigm enables models to overcome the limited capacity of parametric memory by dynamically accessing external information only when it is needed.

Retrieval-augmented language models provide the foundation for this paradigm. REALM [64] learns to retrieve relevant documents from an indexed corpus during pretraining, while RAG [15] combines dense document retrieval with sequence generation to enable scalable knowledge access during inference. RETRO [117] further scales this paradigm by retrieving relevant document chunks from a large external corpus and integrating them into generation through chunked cross-attention. Extending retrieval beyond text, MuRAG [65] introduces a unified multimodal memory containing both textual and visual evidence, and REVEAL [89] scales multimodal retrieval through large external visual-linguistic knowledge stores. More recent VLMs continue to expand retrieval capabilities by incorporating web-scale search and multimodal evidence integration. SearchLVLMs [99] introduces plug-and-play web retrieval for vision-language models, while VisRAG [102] retrieves visually encoded document images directly instead of relying on OCR-based text conversion. Recent retrieval-augmented multimodal frameworks further improve retrieval quality through enhanced visual search, retrieval control, and multimodal evidence integration [94,100,101,103].

Knowledge-guided Retrieval. Not all relevant information is contained within perceptual observations or previously recorded experiences. Many reasoning tasks require external factual, commonsense, or relational knowledge that cannot be reliably inferred from visual evidence alone. Knowledge-guided retrieval therefore augments memory retrieval by selectively accessing structured knowledge bases, knowledge graphs, or symbolic repositories that complement multimodal perception.

Representative approaches include KRISP [92], which combines visual reasoning with symbolic knowledge retrieval for visual question answering, and KAT [93], which retrieves external knowledge to support knowledge-intensive vision-language reasoning. Plug-and-Play VQA [95] similarly demonstrates that external knowledge can be incorporated at inference time without requiring end-to-end retraining of the underlying vision-language model. Together, these approaches illustrate how retrieval extends beyond recalling previous experiences to incorporating structured knowledge unavailable from the current visual context.

Attention-based Retrieval. Attention-based retrieval accesses retained information by conditioning the current computation on memory representations through attention mechanisms. Depending on the architecture, these representations may consist of recurrent hidden states, compressed activations, dedicated memory tokens, cached key-value states, or entries obtained from a separately maintained memory. In model-resident variants, retrieval is performed through differentiable attention over states propagated from earlier segments or processing steps. Hybrid variants instead combine neural attention with non-differentiable lookup over an auxiliary memory. Attention-based retrieval is therefore characterized by how retained information is selected and integrated into the current computation, rather than by requiring the underlying memory to be exclusively internal.

Representative model-resident approaches include RMT [12], which propagates dedicated memory tokens across long sequences, and Transformer-XL [118], which applies segment-level recurrence and attention over cached hidden states to extend context beyond a fixed input window. Compressive Transformer [11] further maintains compressed representations of older activations, allowing attention to access both recent and more distant context. Memorizing Transformers [10] instantiate a hybrid design: they augment self-attention with approximate nearest-neighbor lookup over a separately maintained, non-differentiable memory of previous key-value pairs. Together, these approaches illustrate a spectrum ranging from attention over model-resident recurrent states to attention augmented by non-parametric memory lookup.

Episodic Memory Retrieval. Whereas search-based and knowledge-guided retrieval primarily access external information sources, episodic retrieval recalls experiences accumulated through previous interactions. These memories may include user preferences, dialogue histories, observations, navigation trajectories, or behavioral experiences that enable models and embodied agents to adapt future reasoning according to past events.

MemoryBank [13] retrieves previously stored user memories to support personalized responses across long-term interactions, while Generative Agents [66] recall observations, reflections, and prior experiences to guide planning and social behavior. In embodied settings, ReMEmbR [72] and Mem4Nav [108] retrieve previously encountered spatial-temporal experiences to support localization, navigation, and long-horizon decision-making. These systems demonstrate that retrieval transforms accumulated experiences from passive storage into actionable evidence that continuously informs future perception, reasoning, and action.

Taken together, these retrieval mechanisms demonstrate that effective memory depends not only on what information is stored, but also on how previously acquired knowledge and experiences are accessed during inference. Whether through search, knowledge integration, attention-based recall, or episodic retrieval, these mechanisms allow models to selectively leverage relevant information while maintaining computational efficiency. Effective retrieval therefore serves as the bridge between stored memory and downstream reasoning, enabling memory systems to transform accumulated information into actionable evidence.

2.4.4. Memory Updating and Forgetting

Memory updating and forgetting describe processes that modify retained information after its initial acquisition, enabling memory systems to adapt to evolving environments while controlling the persistence of outdated or less useful content. These operations include revising stored knowledge, incorporating new information, reorganizing existing representations, preserving important information against interference, and selectively removing obsolete memories. Unlike writing, which introduces new information into memory, updating transforms the content, representation, or accessibility of previously retained information.

Editing Stored Knowledge. Editing mechanisms aim to modify specific knowledge encoded within multimodal models without retraining the entire model. Such methods typically identify the parameters, representations, or external memory components responsible for a target association and apply localized modifications while minimizing unintended changes to unrelated knowledge. Can We Edit [111] provides a representative study of whether visual knowledge encoded in multimodal models can be selectively revised while controlling collateral effects. This line of work formulates updating as a targeted intervention problem, where the objective is to incorporate corrected information while preserving existing capabilities. Visual-oriented fine-tuning and unified knowledge maintenance approaches further investigate how VLMs can integrate updated visual knowledge through parameter adaptation or controlled fine-tuning strategies while reducing interference with previously learned representations [112,119]. These approaches highlight the challenge of balancing editability with knowledge consistency, as overly localized updates may fail to propagate necessary changes, whereas broad modifications may introduce unintended side effects.

Continual Learning and Forgetting Control. Continual learning considers memory updating as a sequential adaptation problem, where models continuously incorporate new multimodal information while maintaining previously acquired knowledge. A central challenge is catastrophic forgetting, where optimization toward new tasks overwrites representations required for earlier capabilities. Existing approaches address this problem through several mechanisms, including replaying representative historical data, constraining parameter updates, isolating task-specific knowledge, or generating synthetic examples that approximate previous experiences. Learning without Forgetting for VLMs [85] provides a representative example by balancing adaptation to new multimodal tasks with preservation of prior knowledge. Other approaches investigate interference reduction, replay-based learning, and synthetic data generation to improve knowledge retention during sequential updates [120,121]. Instruction-aware preservation and visual conditioning strategies further explore how models can selectively incorporate new instructions or visual information while maintaining compatibility with existing capabilities [122,123]. Together, these methods reveal the fundamental tradeoff between plasticity, which enables adaptation to new information, and stability, which preserves previously acquired knowledge.

Updating Transient Inference Memory. Updating also occurs within short-lived memory states used during inference, where limited computational resources require models to continuously refresh, compress, or discard intermediate representations. KV-cache management methods modify the key-value states retained by attention layers by estimating the importance of cached tokens and selectively applying eviction, compression, merging, or quantization strategies. LOOK-M [42] updates cache contents through importance-based retention policies, where less informative tokens are removed to reduce memory consumption while preserving relevant context. MadaKV [43] further adapts cache eviction according to modality-aware importance, recognizing that visual and textual tokens may contribute differently to downstream reasoning. AKVQ-VL [88] updates transient memory through adaptive quantization, reducing the precision of less critical representations while maintaining high-fidelity storage for important information. Streaming systems such as StreamChat [44] and VideoLLM-online [54] continuously update inference memory as new observations arrive, replacing or compressing older information to support long-context interaction in dynamic environments. These approaches demonstrate that forgetting can serve as an intentional memory management strategy rather than merely an undesirable failure mode.

Updating Persistent Memory Stores. Persistent memory systems require mechanisms for revising previously recorded experiences while maintaining consistency over long-term interactions. Unlike transient inference memory, persistent memory updates must determine when new information should create a new memory, modify an existing memory, or replace outdated information. MemoryBank [13] represents long-term user information as explicit memory entries stored outside model parameters. During interaction, newly observed information is compared with existing memories and used to update, merge, or create memory entries, enabling personalized responses across extended conversations. KARMA [71] similarly updates embodied memories by integrating newly acquired experiences to improve future action selection. Beyond explicit memory stores, parameter-efficient adaptation approaches update persistent knowledge implicitly by modifying lightweight parameter components rather than maintaining separate external memory structures [79,86,87]. These approaches reduce update costs but trade explicit memory control for less interpretable and less directly editable representations.

Taken together, these updating mechanisms demonstrate that effective memory depends not only on acquiring and storing information, but also on continuously adapting retained knowledge as new evidence emerges. Whether through knowledge editing, continual learning, transient state management, or persistent memory revision, updating enables memory systems to remain adaptive while preserving information that remains valuable over time. Controlled forgetting further ensures that limited memory capacity is allocated to the knowledge and experiences most relevant for future reasoning, interaction, and decision-making.

3. Memory as a Coupled System

In practical VLMs, the four dimensions of our taxonomy are analytically separable but operationally coupled. The preceding subsections introduced each dimension independently to establish a clear vocabulary, whereas concrete systems instantiate combinations in which choices along one dimension influence, but do not uniquely determine, choices along the others. Figure 6 presents eight representative such combinations. These dependencies arise jointly from task requirements, architectural constraints, computational budgets, and the intended lifetime of retained information. Temporal scope provides a useful organizing lens because it often constrains feasible storage and maintenance strategies; nevertheless, it should be interpreted as describing recurrent design patterns rather than imposing hard admissibility rules on the other dimensions. We therefore examine transient, episodic, and persistent memory in turn while explicitly allowing multiple storage locations, memory entities, and operation sets within each temporal regime.

When information is required only within a single execution, such as answering questions over an extended video or conditioning generation on interleaved demonstrations, the temporal scope is transient. Existing systems commonly realize transient memory through model-resident states, including raw observations retained near the input [33,38,39,104] and latent representations that compress such evidence into compact token sets [35,76,77]. Writing and online maintenance operations, such as compression, eviction, and reorganization of key-value caches, are therefore particularly common [42,43,88]. However, transient scope does not require memory to be internal: session-local scratchpads, temporary external caches, and tool-generated records may also be written and retrieved within a single execution. Updating and forgetting may likewise occur as active states are refreshed, compressed, or evicted. Internal storage and a restricted operational lifecycle are thus prevalent design choices for transient memory, rather than defining requirements.

When information must remain available throughout a bounded interaction, such as a multi-turn grounded dialogue or a streaming video session, the temporal scope becomes episodic. Episodic memory may be maintained internally through recurrent or compressed latent states [54,109,113], or externally through addressable interaction records such as dialogue turns, state slots, observations, and navigation steps [49,50,63]. Its operation set consequently depends on the implementation: internal recurrent states may emphasize writing and consolidation across turns, whereas explicit episodic stores may additionally require indexing, retrieval, updating, and capacity-aware forgetting. External retrieval is therefore compatible with episodic memory even when the retained information is discarded at the end of the interaction.

When information must persist across multiple sessions, for example to support long-term personalization or lifelong embodied agents, systems commonly employ explicit external stores, parametric memory, or combinations of the two. External stores may retain raw multimodal observations, structured episodic records, abstract semantic knowledge, or latent encodings, and make them independently addressable through retrieval mechanisms [15,64,65,102]. Parametric memory instead encodes retained knowledge in model weights and accesses it implicitly through forward computation; modifying such memory typically requires continual training, parameter-efficient adaptation, or model editing [85,86,111]. Persistent systems often benefit from a broader lifecycle comprising writing, consolidation, retrieval, updating, and forgetting [13,66,71], but persistence does not imply that every system implements every operation or adopts a particular memory entity.

Viewed from this perspective, the taxonomy defines a structured and non-uniform design space rather than either an unconstrained Cartesian product or a set of hard compatibility rules. Some combinations are common because they offer favorable trade-offs, whereas others remain uncommon because of implementation cost, limited editability, or insufficient retrieval interfaces. Latent representations illustrate this distinction: transient activation states are model-resident, whereas derived embeddings or key-value representations can be materialized in independently addressable external stores. Similarly, a transient hidden state cannot itself survive beyond its execution lifetime, but its content may be transformed and written into an episodic or persistent representation. Characterizing these couplings as empirical design tendencies, rather than intrinsic impossibilities, both improves the descriptive accuracy of the taxonomy and exposes under-explored combinations for future research.

4. Benchmarks & Evaluation Metrics

4.1. Benchmarks and Datasets

Memory-related evaluation spans diverse tasks that probe different aspects of memory, ranging from long-context reasoning to continual adaptation and personalized interaction. To provide a structured overview, we categorize existing benchmarks and datasets into eight groups according to the memory being evaluated, the task setting, and the primary evaluation objective: long video understanding and generation, image reasoning and captioning, continual learning, embodied AI, multimodal RAG, egocentric vision, personalized assistants, and virtual-environment interaction, mirroring the application domains of Figure 7. Table 2 summarizes representative benchmarks and datasets together with their data scales and task types.

Long Video Understanding and Generation. These benchmarks evaluate memory over long or streaming videos, where relevant information may be separated by substantial temporal gaps. They assess temporal retention, cross-segment retrieval, long-range reasoning, and generation or prediction conditioned on historical observations.

Image Reasoning and Captioning. These benchmarks evaluate memory across multiple images, image sequences, or visual contexts. Compared with long-video tasks, they place less emphasis on continuous temporal dynamics and instead focus on cross-image reasoning, visual context integration, and memory-aware description generation.

Continual Learning. Continual learning benchmarks evaluate memory across sequential training stages, where models must acquire new knowledge, domains, tasks, or instructions while preserving previously learned capabilities. They primarily measure knowledge retention, adaptation to new tasks, and resistance to catastrophic forgetting.

Embodied AI. Embodied AI benchmarks evaluate memory in agents operating in physical or simulated environments. They require models to maintain and utilize historical observations, object states, goals, and action histories to support navigation, manipulation, autonomous driving, and long-horizon decision-making.

Multimodal RAG. Multimodal RAG benchmarks evaluate retrieval-based external memory by measuring how effectively models retrieve, integrate, and reason over multimodal evidence, such as images, documents, charts, and structured knowledge sources, with evaluation emphasizing evidence grounding and cross-modal reasoning.

Egocentric Vision. Egocentric vision benchmarks evaluate memory from first-person visual streams, particularly those capturing daily activities and personal experiences. They emphasize episodic retrieval and first-person assistant capabilities, including recalling previously observed people, objects, events, and contextual information.

Personalized Assistants. Personalized-assistant benchmarks evaluate persistent user memory, including personalized visual concepts, preferences, experiences, and interaction histories. They assess whether models can retain user-specific information across interactions and use it to generate personalized responses.

Virtual-environment Interaction. Virtual-environment benchmarks evaluate memory in digital environments such as games, graphical user interfaces, web applications, and computer-use tasks. They assess whether agents can maintain task states, remember previous actions, and leverage long-term interaction histories during multi-step or multi-session execution.

4.2. Metric Taxonomy

Existing memory benchmarks employ diverse evaluation metrics because they target different aspects of multimodal memory, including long-context reasoning, egocentric recall, continual adaptation, embodied interaction, retrieval-augmented generation, personalization, and virtual-environment tasks. Consequently, their metrics are not directly comparable under a single unified scale. Instead, they capture complementary properties of memory, such as long-context utilization, information retention, knowledge preservation, action execution, evidence retrieval, generation consistency, and task success.

Based on their evaluation objectives, we organize existing metrics into the following eight categories:

• Task Performance measures overall task success, including accuracy, QA accuracy, MCQ accuracy, task score, and pass rate.
• Retrieval and Grounding measures the ability to retrieve, localize, and correctly ground relevant visual, textual, or multimodal evidence.
• Memory Recall and Retention evaluates whether models can successfully recover and utilize information from previous observations, sessions, or user interaction histories.
• Temporal and Long-Context Evaluation measures performance across varying video durations, context lengths, temporal dependencies, and streaming settings.
• Continual Learning measures the ability to preserve previously acquired knowledge while adapting to new tasks, domains, instructions, or knowledge.
• Generation Quality and Consistency evaluates the fidelity, visual quality, and temporal consistency of generated multimodal content.
• Embodied and Agentic Execution measures how effectively memory supports decision-making and action execution in embodied, robotic, or interactive environments.
• Efficiency and Robustness evaluates computational efficiency and robustness, including memory overhead, latency, computational cost, and performance under distracting or challenging memory conditions.

Representative metrics within each category are summarized in Table 3.

5. VLM Memory Applications

5.1. Long Video Understanding and Generation

Offline long-video understanding assumes the complete video is available before inference, allowing memory to be constructed over the full temporal context rather than incrementally during streaming. Within this setting, memory primarily supports retrieving, organizing, and reasoning over information distributed across extended temporal horizons (Figure 8).

5.1.1. Offline Long-Video Understanding

Long-form Video QA. Long-form Video QA requires preserving and retrieving question-relevant evidence across extended video histories, rather than merely understanding individual clips. Early work already connected this task with explicit memory access, as exemplified by DeepStory [188], which represents video history as a retrievable memory. Subsequent methods externalized long videos into textual histories [189] and further organized them into more concise and structured language repositories for long-range reasoning and question answering [190]. Later work shifted the focus from retaining more history to retaining the history most relevant to the query. MovieChat [113] and MovieChat+ [191] improve long-term memory through sparse compression and query-aware retention, respectively, while Glance and Focus [192] progressively retrieves evidence from global event summaries to local details. More recent methods, including MA-LMM [40] and MemVid [193], further organize memory into more structured and selective representations that facilitate retrieval and reasoning. Overall, recent methods increasingly shift from passively storing long-video history toward selectively organizing and retrieving question-relevant evidence.

Long-video Event Understanding. Long-video event understanding requires organizing long videos at the event level rather than treating them as flat temporal histories. Rather than compressing an entire video into a single representation, earlier work introduced event-centered memory structures. HERMES [194] builds long-form understanding through episodic accumulation and semantic retrieval, while hierarchical event-based memory [195] explicitly models event segmentation together with intra- and inter-event memory. Later methods emphasize persistent event memory through active memory construction and continuous-time consolidation. AMEGO [163] builds a self-contained active memory for very long egocentric videos, and ∞-Video [196] continuously consolidates observations for training-free long-video understanding. Recent work further develops structured episodic memory for modeling event relationships. HippoMM [197] combines episodic and semantic consolidation for long audiovisual event understanding, Video-EM [198] represents key moments as temporally ordered episodic events, and GCAgent [199] models schematic and narrative episodic memory to capture causal and temporal relationships across events. Overall, recent work suggests that organizing memory around semantically meaningful events provides a more effective representation than compressing entire videos into flat histories.

Temporal Grounding. Temporal grounding requires retrieving the video moments relevant to a language query, making effective temporal memory organization essential for locating evidence across long videos. Early efforts primarily improved temporal awareness through timestamp-aware encoding, boundary-aware training, and explicit temporal representations, as demonstrated by TimeChat [105], VTimeLLM [200], and Seq2Time [201]. Later work increasingly organized temporal evidence into more structured representations. Grounded-VideoLLM [202] introduces an additional temporal stream together with discrete temporal tokens to improve grounding, TRACE [203] represents videos as causal event sequences, and Number it [204] improves temporal retrieval through explicit frame indexing. More recent methods increasingly combine structured temporal evidence with reasoning. VideoExpert [205] separates temporal grounding from content generation through specialized temporal and spatial experts, Time-R1 [206] improves grounding via reasoning-oriented post-training, VideoITG [207] adaptively selects relevant frames through instruction-guided temporal grounding, and E.M.Ground [208] further improves grounding through holistic event perception and matching. Overall, recent work increasingly treats temporal grounding as retrieving and reasoning over structured temporal evidence rather than simply localizing timestamps.

5.1.2. Online / Streaming Video Understanding

Unlike offline long-video understanding, streaming video understanding requires models to process continuously arriving video without access to the complete sequence, making memory essential for retaining and retrieving information over time. Early work primarily adapted Video-LLMs to streaming settings through sequential processing and temporally aligned interaction, as demonstrated by Streaming Long Video Understanding with Large Language Models [55] and VideoLLM-online [54]. Subsequent work made explicit memory a central component of online understanding. Flash-VStream [209] maintains memory for real-time long-video understanding, while Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge [210] supports multi-round interaction through hierarchical memory organization. Later methods increasingly emphasized selective and efficient memory management under limited computational budgets. Streaming VQA with In-context Video KV-Cache Retrieval [211] retrieves relevant historical context from cached representations, StreamMem [212] dynamically manages streaming memory, Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding [213] reduces memory consumption through efficient memory design, and CacheFlow [214] maintains fixed-size caches for long-duration streaming. More recent work further integrates memory with adaptive reasoning and response planning. WAT [215] decouples observation from reasoning to better utilize accumulated memory, while StreamReady [216] jointly learns what information to retain and when to respond during continuous interaction. Overall, recent methods increasingly treat streaming memory as an actively managed resource that selectively retains, retrieves, and reasons over historical observations rather than simply buffering incoming video.

5.1.3. Reasoning over Long Videos

Reasoning over long videos has increasingly emerged as a distinct challenge beyond generic long-video understanding. Rather than only recognizing events or retrieving relevant moments, the objective is to connect temporally dispersed evidence across extended video histories to support multi-step, causal, and narrative reasoning. Recent benchmarks have made this transition more explicit. MoviePuzzle [217] studies visual narrative reasoning through multimodal order learning, while VRBench [218] and VCRBench [219] explicitly evaluate multi-step and causal reasoning over long videos. More recent benchmarks further expand long-video reasoning to larger-scale and more diverse scenarios, including NA-VQA together with its reasoning framework Video-NaRA [220], MINERVA-Cultural [221], and VBVR [222]. Motivated by these increasingly demanding reasoning tasks, recent methods have shifted from storing long-video context toward organizing memory for evidence integration and inference. TemporalVLM [223] builds temporally structured representations for long-range reasoning, while WorldMM [224] and LongVideoAgent [225] organize and retrieve relevant evidence across extended video histories. More recent agentic methods, including A4VL [226] and SAGE [227], further integrate memory with multi-stage planning and reinforcement-trained decision making to support complex long-video reasoning. Overall, recent work increasingly treats memory as an active reasoning substrate that organizes, retrieves, and integrates temporally distributed evidence rather than simply retaining long-video context.

5.1.4. Long Video Generation

Long video generation aims to synthesize extended video sequences while maintaining scene, entity, motion, and camera consistency over long horizons. This requires preserving and reusing historical context beyond locally plausible clip generation. Consequently, memory has become a key mechanism for maintaining coherent visual narratives across long sequences. In scene- and world-consistent generation, GEN3C [228], Context as Memory [229], and WorldWeaver [230] leverage 3D caches, retrieved historical frames, and depth-informed memory banks to reduce structural and temporal drift. For long-form autoregressive generation, Pack and Force Your Memory [231], LongLive [232], Context Forcing [233], and Relax Forcing [234] preserve long-range dependencies through memory packing, context reuse, and KV-memory designs. More recent methods make memory increasingly structured and semantically meaningful: VideoMemory [235] maintains entity-centric memories across shots, while AnchorWeave [236] and MemCam [237] retrieve spatial and camera-aware memories to preserve world consistency. Collectively, these approaches illustrate a transition from reusing recent visual history to maintaining structured representations of scenes, entities, geometry, and camera state that support coherent long-form video generation.

5.1.5. Long Video Prediction

Long video prediction focuses on forecasting future visual observations over extended horizons, with recent work increasingly framing the problem as learning interactive world models. Rather than merely generating plausible future clips, these models must preserve and update an internal representation of the environment under actions, viewpoint changes, and evolving scene dynamics. EVA [238] couples reasoning and generation for embodied future anticipation, while Vid2World [239] repurposes pre-trained video diffusion models into action-conditioned interactive world models. As prediction horizons increase, persistent memory becomes essential for maintaining world state over time. Video World Models with Long-term Spatial Memory [240], RELIC [241], and LIVE [242] introduce geometry-grounded spatial memory, compressed camera-aware KV memory, and cycle-consistent long-horizon training to mitigate forgetting and error accumulation. More recent work further expands the scope of memory beyond static scene structure. PAN [243] and Astra [244] advance more general and controllable action-conditioned world simulation, while VerseCrafter [245] and hybrid-memory methods [246] integrate geometric world representations with dynamic subject memory. Together, these developments reflect a shift from maintaining short-term temporal context to explicitly modeling persistent, evolving world states that enable long-horizon interactive prediction.

5.2. Image Reasoning and Captioning

5.2.1. Image Reasoning

Long-chain multimodal reasoning often suffers from visual forgetting, where attention to the visual input gradually diminishes as the reasoning trajectory grows longer, causing models to rely increasingly on previously generated text rather than the image itself [123]. Memory mechanisms, summarized in Figure 9, address this problem by preserving visual evidence throughout inference. Take-along Visual Conditioning (TVC) maintains access to visual information by dynamically pruning redundant visual tokens while retaining informative ones during reasoning [123]. In multi-image reasoning, CMMCoT [247] caches decoder-layer key-value representations for each image in an external memory bank and retrieves them on demand, enabling models to revisit relevant visual evidence across multiple images. Rather than relying on external memory, VisMem [248] incorporates both short-term perceptual memory and long-term semantic memory within the model to preserve fine-grained visual details while progressively consolidating higher-level concepts during reasoning and generation. Collectively, these approaches demonstrate that memory helps maintain visual grounding over long reasoning chains by preserving perceptual evidence, semantic representations, and cross-image correspondences throughout inference.

5.2.2. Image Captioning

Image captioning benefits from memory by augmenting visual features with semantic information accumulated beyond the current image, including previous training examples and external knowledge sources. Standard attention mechanisms compute representations solely from the input image, limiting their ability to exploit knowledge acquired from previous examples or external resources [249]. The Meshed-Memory Transformer introduces learnable memory vectors into the visual encoder to encode dataset-level prior knowledge that complements region-level visual representations [250]. Moving beyond static dataset-level priors, Prototypical Memory Networks summarize representations from previously observed training samples into compact prototype vectors, allowing the model to retrieve discriminative visual patterns learned from similar examples [249]. Memory can also be maintained externally. Sarto et al. [251] retrieve captions from visually similar images in an external corpus and combine the retrieved text with the input image using a kNN-augmented language model. Overall, memory in image captioning has evolved from learnable global memory toward retrieving instance-level visual and textual knowledge that enriches image understanding and improves caption generation.

5.3. Continual Learning

5.3.1. Incremental Recognition

As illustrated in Figure 10, incremental recognition requires VLMs to continually expand their recognition space as new categories, domains, and open-vocabulary concepts are introduced, while preserving previously acquired vision–language semantic alignment. Early studies on frozen CLIP suggest that its pretrained vision–language embedding space already exhibits considerable robustness to continual recognition, indicating that the embedding space itself functions as an implicit semantic memory [252]. However, continual finetuning can perturb this space and degrade zero-shot transfer. ZSCL [134] therefore shifts the focus from mitigating forgetting of old classes to preserving CLIP’s open-vocabulary prior, reducing semantic drift through feature distillation and weight regularization. Subsequent methods move from protecting the original CLIP space toward more controllable incremental retention mechanisms. One line of work combines CLIP’s zero-shot semantics with exemplar memory to balance unseen-class generalization and seen-class consolidation [253]. MoE-Adapters [254] organize stage-wise knowledge into callable expert modules to reduce interference across incremental stages, while CLAP4CLIP [255] and RAPF [256] further stabilize incremental adaptation through probabilistic finetuning and representation adjustment with parameter fusion, respectively. More recent methods, including CoLeCLIP [257], LADA [258], and incremental prompt tuning with intrinsic textual anchors [259], further extend memory into scalable representations at the category, open-vocabulary, and semantic-anchoring levels. Overall, the object of memory in incremental recognition evolves from preserving CLIP’s pretrained semantic embedding space toward scalable mechanisms that retain and organize category-level, open-vocabulary, and semantic knowledge across incremental stages.

5.3.2. Continual Knowledge Updating

Building on category-level recognition, continual knowledge updating requires VLMs to adapt their pretrained cross-modal representations from continually arriving image–text corpora, evolving data distributions, and newly emerging domains, while retaining prior knowledge, cross-modal alignment, and previously acquired capabilities. Early work primarily viewed memory as a means of preserving CLIP’s pretrained capabilities during continual pretraining. VR-LwF [260] maintains zero-shot and image–text matching abilities through replayed vocabulary, while IncCLIP [261] uses generative negative text replay and multimodal knowledge distillation to retain previous pretraining knowledge. Subsequent methods move beyond replaying samples or vocabularies and instead treat the geometry of the cross-modal representation space itself as the primary object of memory. Mod-X [262] preserves off-diagonal information to maintain multimodal alignment on previous domains, whereas CTP [263] combines compatible momentum contrast with topology preservation to retain cross-task embedding structures when absorbing new image–text data. As continual pretraining moves closer to real deployment, memory must also accommodate temporally evolving data streams and accumulated model states. TiC-CLIP [264] and FoMo-in-Flux [265] push the problem toward web-scale temporal data streams and realistic deployment constraints, while TIME [266] provides a model-level memory route through temporal model merging to integrate multimodal expert knowledge accumulated at different stages. Recent methods further preserve prior knowledge without fully storing historical data. GIFT [121] uses synthetic image–text pairs as replay surrogates, while GNSP [267] protects existing representations through gradient null-space projection and cross-modal alignment constraints. Overall, the object of memory evolves from replaying historical data or vocabularies, to preserving cross-modal representation geometry, and ultimately to maintaining temporally accumulated multimodal knowledge under limited access to historical data.

5.3.3. Continual Task Adaptation

Unlike continual updating of pretrained representations, continual task adaptation requires VLMs to acquire new multimodal capabilities as new tasks, interaction patterns, and instruction formats emerge, while retaining previously learned visual reasoning, question-answering strategies, and instruction-following behaviors. CLiMB [138] and VQACL [268] characterize this problem from continual vision–language tasks and VQA skills, respectively, making task interaction patterns a primary object of memory. Later methods transform these interaction patterns into more structured memory. Scene-graph-based symbolic replay [269] converts visual relations and question–answer patterns into scene graph prompts, while QUAD [270] uses questions-only memory to retain question patterns with reduced dependence on stored images. Subsequent methods increasingly parameterize memory as reusable task-adaptation states. TRIPLET [271] preserves modality- and task-specific factors through decoupled prompts, whereas CL-MoE [272] selectively retrieves previous task abilities through expert routing. As VLMs are increasingly adapted through instruction tuning in MLLM–style architectures, the emergence of continual instruction tuning [273] further highlights forgetting in multimodal instruction following, extending the object of memory to instruction-following behavior, visual reasoning, and task-specific parameters. SMoLoRA [274] mitigates the dual forgetting of visual understanding and instruction following through separable routing, while HiDe-LLaVA [275] separates task-private knowledge from cross-task general ability through task-specific expansion and task-general fusion. Overall, the object of memory evolves from replaying previous task experiences to maintaining reusable question structures, prompt and expert states, and instruction-adaptation parameters that support continual acquisition of new multimodal skills.

5.4. Embodied Navigation and Action

Figure 11. Memory architecture for embodied navigation and action.

Figure 11. Memory architecture for embodied navigation and action.

5.4.1. Vision-Language Navigation

A. Instruction-following and Spatial Memory Navigation

Instruction-following Vision-Language Navigation (VLN) with spatial memory requires agents to ground language instructions in accumulated observations and spatial context under partial observability, rather than relying solely on the current visual view. Early approaches introduced lightweight navigation memory through progress estimation and trajectory tracking. Self-Monitoring [276] estimates navigation progress to align the agent’s internal state with instruction execution, while Regretful Agent [277] exploits progress signals for backtracking and path correction. Subsequent methods move beyond compact state summaries toward explicit history modeling: VLN-BERT [278] and HAMT [59] incorporate past observations, actions, and instruction context to improve sequential decision making. As navigation tasks become more challenging, agents move beyond storing observation histories and begin to organize memory into spatial representations, such as topological graphs and environment maps. DUET [60] and BEVBert [279] organize explored trajectories as topological or map-based representations, while GridMM [280] maintains a dynamically expanding grid memory to preserve spatial relationships and instruction-relevant information. Recent studies further integrate memory with high-level reasoning and compact representation. MC-GPT [281] constructs a memory-based topological map over viewpoints, objects, and spatial relations, whereas JanusVLN [282] separates geometric and semantic knowledge through dual implicit memory representations. Overall, instruction-following VLN memory has evolved from trajectory-level progress tracking toward structured, semantics-aware spatial representations that support planning and reasoning.

B. Dialogue-based Navigation

Dialogue-based navigation extends VLN memory beyond instruction execution by requiring agents to maintain conversational context, user intent, environmental observations, and spatial state across multiple interaction rounds. Talk the Walk [283] and CVDN [284] formulate navigation as grounded dialogue, where conversation history assists localization, clarification, and action selection. CMN [58] explicitly separates language and visual memory to associate current dialogue with previous utterances, observations, and actions, while RMM [285] further models collaborator state by reasoning about the knowledge shared between the guide and navigator. More realistic collaborative settings emphasize the alignment among linguistic descriptions, environmental semantics, and agent location. RobotSlang [286] and DRAGON [287] model correspondences between dialogue and embodied states, while DialNav [288] considers remote collaborative navigation where the guide must infer the navigator’s location through dialogue alone. Overall, memory in dialogue-based navigation has progressed from storing conversation history to maintaining cross-modal interaction states that connect dialogue, perception, and spatial reasoning.

C. Long-horizon and Persistent Navigation

Long-horizon and persistent VLN focuses on accumulating and reusing knowledge across extended routes, multi-stage instructions, and repeated interactions with the same or related environments. Early work explored future-aware and map-based memory mechanisms: Look Before You Leap [289] predicts future navigation states to support planning, Chasing Ghosts [290] formulates instruction following as Bayesian state estimation with an online semantic spatial map, and Talk2Nav [291] uses spatial memory to associate landmarks with directional transitions in large-scale urban navigation. Memory subsequently became important for managing long and compositional instructions. M-TRACK [292] tracks intermediate milestones to support subgoal completion and reduce navigation failures caused by missed targets or premature termination. Beyond individual episodes, persistent navigation methods focus on reusable environmental knowledge. Structured Scene Memory [293] organizes historical observations into persistent scene representations, IVLN [294] and ESceme [295] retrieve prior experiences in revisited environments, and OVER-NAV [296] leverages open-vocabulary perception with structured representations to integrate visual, semantic, and spatial information over long tours. More recently, MSNav [297] combines dynamic memory management with LLM-based spatial reasoning, using selective updates to reduce memory redundancy in long-horizon tasks. Overall, persistent VLN memory extends beyond episode-specific state tracking toward reusable environmental knowledge that supports navigation across stages, routes, and experiences.

5.4.2. Vision-Language Action

Memory in Vision-Language Action (VLA) models enables robots to perform long-horizon tasks in partially observable and dynamic environments, where the appropriate action depends not only on the current observation but also on previous interactions, task progress, and execution feedback. Unlike short-horizon control, real-world manipulation is inherently non-Markovian: identical visual inputs may require different actions depending on prior actions, hidden states, object changes, or completed subtasks. Early episodic memory approaches for robotic manipulation demonstrate the importance of retaining state transitions, action trajectories, and subtask structures from demonstrations [298]. Building on this idea, subsequent VLA methods incorporate interaction histories to monitor task progress, recover from failures, and improve action generation.

Recent approaches increasingly transform raw interaction histories into structured perceptual and spatial memories. SAM2Act+ [32] introduces a memory bank with attention mechanisms to enhance spatial reasoning, while MemoryVLA [299] combines working memory with perceptual-cognitive memory to preserve both visual details and high-level task semantics. Beyond storing past observations, memory also enables experience retrieval and reuse across tasks. MAP-VLA [300] retrieves stage-specific memory prompts from previous demonstrations to guide action prediction, whereas EchoVLA [301] integrates scene memory with episodic memory for mobile manipulation by maintaining spatial-semantic representations and task-level experiences.

More recent VLA models focus on scalable and temporally consistent memory mechanisms for extended real-world deployment. OptimusVLA [302] separates memory into Global Prior Memory and Local Consistency Memory to balance reusable knowledge with current-task adaptation. MEM [303] introduces complementary short-term visual memory and long-term textual memory for sustained interaction, while ReMem-VLA [304] maintains multi-scale temporal context through frame-level and chunk-level recurrent memory queries. HiMe [305] introduces hierarchical memory across different temporal scales to support long-horizon VLA control, while MemoryVLA++ [306] extends memory-based action generation with future-state imagination. Overall, memory in VLA systems has evolved from storing demonstrations and interaction histories toward structured perceptual-spatial memory, experience-based retrieval, hierarchical multi-scale memory, and predictive temporal modeling that support robust long-horizon robot behavior.

5.4.3. Autonomous Driving

Autonomous driving requires memory mechanisms that support continuous perception, reasoning, and decision making in dynamic and safety-critical environments. Unlike embodied navigation and manipulation, where memory primarily helps maintain spatial and task context, autonomous driving must track rapidly changing traffic conditions, preserve interactions with surrounding agents, and incorporate prior experience to handle rare or long-tail scenarios. Since driving decisions often depend on events that are not fully observable from the current sensory input, memory provides a mechanism for integrating historical observations, contextual knowledge, and previously encountered situations.

Recent vision-language driving approaches explore memory as a bridge between perception and high-level reasoning. VLP [307] incorporates language-model-based reasoning into an end-to-end driving planner, where a query-based transformer decoder selectively accesses memory representations containing task-relevant contextual information. By combining retrieved memory with the current scene understanding, VLP improves generalization to uncommon scenarios beyond those directly observed during training. Beyond implicit contextual memory, recent methods introduce explicit experience-based memory for scenario retrieval and decision support. LeapVAD [308] adopts a dual-process architecture consisting of a heuristic process and an analytic process. Its reflection mechanism stores previous driving decisions and outcomes in a memory bank, allowing the analytic process to accumulate scenario knowledge while the heuristic process retrieves relevant experiences for efficient decision making. Similarly, MTRDrive [309] proposes a memory-tool synergy reasoning framework, where a VLM interacts with an external memory bank and an active toolkit engine to gather complementary information. Retrieved experiences and tool-generated evidence are then integrated through reasoning to produce driving decisions.

Overall, memory in autonomous driving has evolved from maintaining temporal scene context toward explicit experience-based and reasoning-oriented memory systems that enable adaptive planning, long-tail scenario handling, and safer decision making in complex environments.

5.4.4. 3D Spatial Scene Memory

Beyond trajectory and map-based navigation memory, 3D spatial scene memory preserves semantic and geometric knowledge of explored environments for subsequent reasoning and interaction. Early approaches established language-grounded 3D representations: VLMaps [310] integrates vision-language features into spatial maps, while ConceptGraphs [311] and HOV-SG [312] organize objects and spatial relations into open-vocabulary 3D scene graphs for language-guided reasoning and navigation. Subsequent methods move toward persistent and dynamically maintained scene memory. DovSG [313] locally updates 3D scene graphs as the environment changes, while 3D-Mem [314] introduces Memory Snapshots and Frontier Snapshots for incremental memory construction, retrieval, and exploration. Recent work further develops more flexible spatial memory mechanisms: MTU3D [315] constructs spatial memory online from streaming RGB-D observations, and 3DSPMR [316] reuses accumulated visual, geometric and relational knowledge across sequential embodied tasks. Memory Over Maps [317] instead retrieves posed RGB-D keyframes as lightweight visual memory and reconstructs spatial information on demand, while SoftNav [318] directly injects persistent 3D scene representations into the VLM hidden space. Overall, 3D spatial scene memory is evolving from language-grounded maps and scene graphs toward incrementally updated, selectively retrieved, and VLM-accessible representations for long-horizon embodied reasoning and action.

5.5. Multimodal RAG

Figure 12. Memory architecture for multimodal retrieval-augmented generation (RAG).

Figure 12. Memory architecture for multimodal retrieval-augmented generation (RAG).

5.5.1. Knowledge-Augmented Visual QA

Knowledge-augmented visual QA requires models to retrieve external knowledge beyond the image content, since many questions depend on named entities, commonsense relations, world knowledge, or fine-grained factual information. Benchmarks such as KVQA [319], OK-VQA [320], A-OKVQA [321], and InfoSeek [322] collectively demonstrate that visual evidence alone is often insufficient, motivating external knowledge as a retrievable memory that complements visual perception. Early retrieval-augmented approaches organize structured knowledge bases or textual corpora as searchable external memory for answer generation [323]. Subsequent work improves the alignment between image-question representations and retrieved knowledge by enabling fine-grained retrieval over visual entities and semantic relations [324]. More recently, retrieval has become tightly integrated with multimodal generation. ReAuSE [325] enables MLLMs to autoregressively generate document identifiers before reasoning over the retrieved evidence, coupling retrieval directly with generation. MI-RAG [326] further introduces iterative multimodal retrieval, maintaining an evolving reasoning memory that progressively refines retrieval queries and integrates evidence from heterogeneous knowledge sources. Overall, memory in knowledge-augmented VQA has evolved from static external knowledge repositories to retrieval memory that is increasingly entity-aware, tightly coupled with multimodal generation, and iteratively updated throughout the reasoning process.

5.5.2. Multimodal Document Retrieval and QA

Multimodal document retrieval and QA require models to retrieve and integrate evidence from visually rich documents, where critical information may reside not only in text but also in layouts, tables, charts, figures, and page-level visual structures. Early work on multimodal document understanding, including LayoutLM [327], DocVQA [328], InfographicVQA [329], and ChartQA [330], demonstrates that document representations must preserve both textual semantics and visual layout. Hierarchical document modeling [331] further highlights the need to aggregate evidence across multiple pages as documents become longer and more complex. Building upon these representations, recent multimodal RAG methods organize document collections as retrievable visual memory. ColPali [332] learns multi-vector embeddings directly from page images, enabling retrieval without relying on OCR pipelines. M3DocRAG [333] extends this idea to multi-page and multi-document settings by jointly retrieving relevant visual pages from document memory and generating grounded responses. VDocRAG [334] further unifies PDFs, presentation slides, and other visually rich documents as image-based retrieval units, reducing information loss introduced by text parsing. MMDocRAG [156] systematically evaluates cross-modal evidence retrieval and grounded generation across long, multi-page documents, highlighting visual grounding as a central challenge. Overall, memory in multimodal document RAG has evolved from layout-aware document representations to retrievable visual page memory that enables evidence aggregation across pages, documents, and modalities.

5.5.3. Domain-Specific and Scenario-Grounded Multimodal RAG

Domain-specific and scenario-grounded multimodal RAG extends retrieval memory to specialized environments where reasoning depends on domain knowledge, scenario experience, or structured contextual evidence. In long-video understanding, Video-RAG [335] retrieves query-relevant auxiliary evidence from audio transcripts, OCR, and object detections to complement visual information, while VideoRAG [336] extends retrieval to external video corpora by retrieving relevant videos together with their visual and textual content. Vgent [337] further organizes long videos into structured graphs and combines retrieval with intermediate reasoning to aggregate relevant evidence across clips. Similar retrieval paradigms have emerged in medical vision-language tasks, where datasets such as VQA-RAD [338], SLAKE [339], and PMC-VQA [340] demonstrate that accurate reasoning requires integrating medical terminology, structured clinical knowledge, and visual evidence. MED-VRAG [341] further advances medical RAG by retrieving document page images rather than isolated text passages and iteratively accumulating retrieved clinical evidence within a memory bank to support diagnosis-oriented reasoning. In autonomous driving, DriveLM [342] formulates driving reasoning across perception, prediction, and planning, highlighting the need to retrieve and reuse structured scenario memory. RAG-Driver [343] retrieves similar driving cases as external memory for in-context learning, improving both interpretability and generalization across traffic scenarios, while RealGen [344] retrieves real driving scenes to guide controllable traffic scene generation, extending retrieval memory beyond question answering to multimodal synthesis. Beyond these domains, RS-RAG [345] organizes multimodal geographic knowledge as retrievable memory for remote sensing reasoning. Overall, multimodal RAG in domain-specific settings is evolving from generic external knowledge retrieval toward scenario-grounded retrieval memory that organizes and reuses temporally structured video evidence, clinical evidence, driving experience, and geographic knowledge for grounded multimodal reasoning.

5.6. Egocentric Vision

Figure 13. Memory architecture for egocentric vision.

Figure 13. Memory architecture for egocentric vision.

5.6.1. Episodic Retrieval

First-person videos provide a natural setting for studying episodic memory, requiring models to recall where an object was previously observed, when an interaction occurred, or what happened earlier in a long stream of experience. The Ego4D dataset introduces an episodic memory benchmark that requires localizing visual evidence answering a natural-language query within a user’s past video [162]. Moving beyond evaluation, AMEGO [163] constructs an online memory of object-centric interaction events using structured representations without explicit semantic labels, enabling efficient retrieval from long egocentric videos. Embodied VideoAgent [346] further extends episodic memory beyond retrieval by maintaining a persistent scene memory that is continuously updated through observed object interactions and subsequently used for reasoning and long-horizon planning in embodied environments.

5.6.2. Wearable Assistants

Wearable assistants require long-term memory that retains previous observations and retrieves relevant experiences in real time. EgoLife builds such an assistant, EgoButler, around an explicit two-stage memory architecture: EgoGPT continuously converts multimodal observations into structured memory entries, while EgoRAG organizes them into hierarchical hourly and daily summaries and retrieves time-stamped evidence for answering questions about events that occurred days earlier [168]. The benchmark highlights remaining challenges, including robust identity recognition and retrieval over extended temporal horizons. Complementing this direction, TeleEgo evaluates egocentric assistants on continuous video streams under real-time constraints, explicitly assessing long-term memory retention as a core capability alongside response quality [167].

5.7. Personalized Assistants

Figure 14. Memory architecture for personalized multimodal assistants.

Figure 14. Memory architecture for personalized multimodal assistants.

5.7.1. Concept Personalization

Personalized assistants require VLMs to acquire user-specific concepts and reason about their visual appearance, textual descriptions, and semantic relationships. Early approaches encode personalized knowledge directly into model parameters. MyVLM [173] augments a VLM with concept-specific classification heads and learned embeddings for personalized concepts, while Yo’LLaVA [174] learns latent representations of personalized subjects from only a few example images. Although effective, these methods require additional optimization whenever new concepts are introduced, making continual personalization increasingly expensive as the number of users and personalized concepts grows.

Recent methods instead externalize personalized knowledge into explicit memory. Retrieval-Augmented Personalization (RAP) [347] stores personalized concepts in an external key–value memory, allowing user-specific knowledge to be updated by modifying the memory rather than retraining the underlying VLM. Online-PVLM [348] further treats personalization as dynamic memory, generating concept embeddings on the fly at inference time without per-instance optimization and continuously storing and retrieving user-specific visual features across interactions. By decoupling personalized concepts from model parameters, these memory-augmented approaches improve scalability to large numbers of users and continually expanding concept vocabularies.

5.7.2. Conversational Memory

Recognizing personalized concepts alone is insufficient for long-term assistance; an assistant must also remember what has been said and observed throughout its interactions with the user. The challenge is that conversational history is open-ended and inherently multimodal. Relevant information may have appeared weeks earlier, in a different conversation session, or within an image or video, making it impractical to retain the entire history in the context window. Consequently, recent systems increasingly maintain explicit long-term memory that stores historical interactions separately from the language model and retrieves only the most relevant information at inference time. M2A [349] combines an immutable message log with a higher-level semantic memory that is updated online, while M3-Agent [350] extends this idea to streaming audiovisual observations by organizing them into an entity-centric memory graph for retrieval and reasoning. PersonaVLM [177] similarly focuses on maintaining long-term multimodal personal context within a personalized vision-language assistant. Evaluating such capabilities remains challenging because conventional short-context benchmarks rarely assess retrieval over extended interaction histories. Mem-Gallery [181] addresses this gap by directly measuring multimodal long-term conversational memory, evaluating whether assistants can accurately recall information introduced many dialogue turns earlier.

5.8. Virtual-Environment Interaction

Figure 15. Memory architecture for virtual environment interaction.

Figure 15. Memory architecture for virtual environment interaction.

5.8.1. Open-world Agents

In games and open-world environments, agents perceive the environment through visual observations and human instructions while performing tasks over extended horizons. A key challenge is enabling agents to leverage information beyond the current observation by reflecting on previous interactions. Memory can operate at multiple levels: within a single trial, recent observations and actions provide a working context for immediate decision-making; across trials, stored experiences can be retrieved to guide future planning and improve task execution. In addition to experiential memory, agents can incorporate external knowledge sources, such as crafting rules and world descriptions from the Minecraft wiki through API-based access [351].

JARVIS-1 incorporates a multimodal memory module that stores successful interaction experiences by associating task scenarios with corresponding action plans. During inference, the agent retrieves relevant experiences as in-context examples to support reasoning and planning, improving decision consistency and performance over long-horizon tasks [352]. Optimus-1 [353] further introduces a Hybrid Multimodal Memory module that summarizes multimodal observations collected during execution into a compact experience pool for efficient storage and retrieval. Beyond episodic experiences, it represents important environmental knowledge, such as crafting requirements, using a hierarchical knowledge graph, enabling agents to reason based on both prior interactions and structured knowledge of the virtual world.

5.8.2. GUI Agents

GUI agents provide an important application domain for memory-augmented multimodal systems because successful interaction with graphical interfaces often requires more than interpreting the current screen. Agents must maintain intermediate interaction states, remember previously encountered interface structures, reuse successful action strategies, and adapt based on prior failures. Consequently, memory in GUI agents has evolved from short-term working context within individual tasks toward persistent memory that stores episodic experiences, semantic knowledge, and reusable interaction procedures across tasks.

Web Agents Web agents interact with complex websites by interpreting visual interfaces and executing sequences of actions. Early systems primarily rely on short-term interaction history as working memory, maintaining previous observations and actions to support ongoing task execution. WebVoyager [354] uses trajectory-based context, including previous actions and annotated screenshots, as temporary memory during web navigation. However, such within-session memory is discarded after task completion and therefore cannot support cross-task knowledge transfer or prevent repeated exploration of previously encountered failures.

ICAL [355] addresses this limitation by transforming episodic interaction histories into reusable semantic knowledge. Instead of directly replaying previous trajectories, it uses a VLM to extract higher-level abstractions from past interactions, including causal relationships, object state transitions, and temporal subgoals. These semantic memories can be retrieved during future tasks, allowing the agent to transfer previously acquired knowledge across different environments and improve performance on visually grounded web benchmarks such as VisualWebArena [356] and VideoWebArena [357]. This illustrates a transition from storing raw interaction traces toward consolidating experiences into task-relevant knowledge.

Mobile Agents Mobile agents operate in dynamic environments involving multiple applications and long sequences of UI interactions. Compared with web agents, mobile agents must frequently maintain context across application transitions and changing interface states. Mobile-Agent-v2 introduces a memory unit that stores task-relevant historical information extracted from previous screenshots. During execution, the agent updates this memory with newly identified focus content, enabling it to preserve important interface states and interaction context across multiple steps. Such working memory is particularly valuable for long-horizon tasks involving multiple applications, where recalling the locations and states of relevant UI elements improves action localization and execution reliability [358].

To support longer-horizon interaction, Mobile-Agent-E [359] extends this framework with persistent memory components consisting of Tips and Shortcuts. Tips capture generalized knowledge extracted from previous tasks, such as strategies for completing specific operations or avoiding common failures, whereas Shortcuts encode reusable sequences of atomic actions for frequently occurring subtasks. Through this separation, the agent transforms episodic interaction histories into procedural knowledge that can be directly reused during future tasks. This design reflects the distinction between episodic memory and procedural memory, where past experiences are converted into reusable skills rather than repeatedly replayed.

Computer Agents Computer-use agents extend GUI interaction beyond individual applications by enabling autonomous control of general-purpose computer environments through keyboard and mouse actions. These agents require persistent memory because computer tasks often involve complex workflows that cannot be completed through immediate perception alone. Agent S [360] introduces a self-supervised memory mechanism that continuously updates from interaction experiences. Its episodic memory stores detailed task trajectories, while narrative memory extracts higher-level task knowledge that can guide future planning. By maintaining both concrete experiences and abstract knowledge, the agent can refine future behavior while avoiding repeated mistakes.

Similarly, UI-TARS [361] is a native computer-use agent that directly interprets screenshots and generates keyboard and mouse actions. Through iterative training and reflection, it leverages accumulated interaction experiences to improve decision-making and adapt to new tasks. Together, these approaches demonstrate how GUI agents are evolving from reactive systems that rely primarily on immediate visual context toward memory-augmented agents capable of accumulating experiences, extracting reusable knowledge, and continuously improving interaction strategies.

6. Challenges & Limitations

Memory extends VLMs beyond isolated image–text inference by enabling models to accumulate, organize, and reuse multimodal experiences. However, current memory-augmented VLMs remain limited by challenges that span four fundamental dimensions: temporal scope, storage location, memory entity, and memory operation. These challenges become particularly critical in long videos, retrieval-augmented generation, embodied environments, and personalized assistants, where memory must be scalable, grounded, editable, secure, and reliably evaluated.

Scalable Memory for Long-horizon Multimodal Experiences. Long-context VLMs must process increasingly long streams of visual observations, language interactions, audio signals, and intermediate reasoning states. Recent systems, including Long Context Transfer, LongVILA, Long-VITA, and Video-XL, demonstrate substantial progress toward extending active context windows and processing hour-scale videos [36,37,38,39]. However, increasing context length alone does not guarantee effective memory. A model may accept more tokens while still failing to retrieve a brief but critical event, identify its temporal location, or use it efficiently under realistic latency and storage constraints. This challenge becomes more pronounced in streaming settings, where the memory horizon is potentially unbounded and the model must continuously decide what information should remain accessible [362]. The central problem is therefore not simply enlarging context windows, but developing mechanisms for selective retention, indexing, and access over long multimodal histories.

Capacity–efficiency Tradeoffs. Memory capacity inevitably introduces computational and storage costs. KV-cache based approaches make this tension explicit: methods such as LOOK-M, MadaKV, and AKVQ-VL reduce inference memory through token pruning, adaptive eviction, or quantization, but these strategies rely on assumptions about which visual or linguistic states can be safely discarded [42,43,88]. Full-stack systems such as LongVILA further suggest that effective long-context memory requires joint optimization across model architectures, training strategies, and inference systems [38]. A fundamental challenge remains that memory importance is often query-dependent: information that appears redundant during storage may later become essential for answering questions about a specific object, event, or temporal relationship.

Faithful Compression and Temporal Preservation. Because storing every observation is impractical, memory systems must compress multimodal experiences while preserving future utility. Approaches such as MovieChat, LongVU, and VideoChat-Flash improve scalability through sparse, adaptive, or hierarchical video memory mechanisms [113,114,115]. However, compression decisions are inherently difficult because future queries are unknown at storage time. Existing approaches often preserve high-level semantics while losing fine-grained temporal structure, including event order, duration, and visual details. Although TimeChat and VideoLLaMB introduce mechanisms for temporal awareness and recurrent context modeling [105,109], maintaining both semantic abstraction and precise temporal grounding under limited memory remains unresolved.

Reliable Retrieval and Trustworthy External Memory. External memory expands the knowledge available to VLMs but introduces new failure modes in retrieval, ranking, and evidence integration. RAG established the retrieval–generation paradigm, while MuRAG and REVEAL extended this framework to multimodal knowledge sources [15,65,89]. More recent systems, including SearchLVLMs and VisRAG, incorporate web-scale retrieval and visually grounded document memory [99,102]. Nevertheless, retrieved evidence may be irrelevant, outdated, incomplete, or inconsistent with the current visual context. Unlike traditional databases, VLMs may generate plausible responses even when retrieved memory is incorrect, making retrieval reliability and evidence verification critical challenges. MRAG-Bench highlights this issue by evaluating whether retrieved visual evidence genuinely improves multimodal reasoning [90].

Grounding, Provenance, and Semantic Alignment. As memory representations become increasingly abstract, maintaining a clear connection between stored information and original observations becomes challenging. Latent memory slots, compressed summaries, and user profiles improve efficiency but reduce interpretability and traceability. Knowledge-grounded approaches such as KRISP and REVEAL demonstrate the benefit of explicit evidence connections [89,92], while VisRAG preserves visual document representations to avoid excessive loss during textual conversion [102]. Meanwhile, benchmarks such as LongVideoBench, Video-MME, and MMLongBench show that models may fail to associate answers with the correct visual evidence even when relevant information exists in memory [45,46,363]. Future memory systems require not only compact storage and accurate retrieval, but also provenance tracking, evidence attribution, and uncertainty estimation.

Memory Updating, Forgetting, and Privacy. Persistent memory requires models to continuously incorporate new information while avoiding harmful accumulation of outdated or incorrect knowledge. Multimodal editing and continual learning studies identify updating and forgetting as fundamental challenges rather than secondary concerns [85,111,112]. These issues become more sensitive in personalized assistants, where long-term memory may contain user preferences, social interactions, and private experiences. Systems such as MemoryBank, personalized multimodal LLM frameworks, and AUGUSTUS demonstrate the value of persistent user memory [13,67,364]. However, persistent memory also creates risks including unauthorized retention, adversarial memory injection, profile drift, and unintended personalization. Designing memory systems therefore requires mechanisms for controllable updating, selective forgetting, user consent, and memory integrity verification.

Comprehensive Evaluation of Memory Behavior. Evaluation of multimodal memory remains fragmented across individual capabilities. Needle-style benchmarks examine hidden information retrieval, long-video benchmarks measure temporal understanding, and multimodal RAG benchmarks evaluate evidence retrieval and integration [45,46,48,90]. MMLongBench and OVO-Bench further investigate long-context reasoning and online video understanding [57,363]. However, real-world memory failures often involve multiple stages: incorrect storage, excessive compression, unreliable retrieval, and improper updating or forgetting. Future benchmarks should therefore evaluate the complete memory lifecycle across multiple sessions, modalities, environments, and evolving user objectives.

7. Future Directions

Future progress in memory-augmented VLMs will require more than longer context windows or larger retrieval stores. Instead, memory should be viewed as a controllable lifecycle that determines what information is acquired, how it is represented, when it is reused, how it is revised, and when it should be forgotten. Future memory-native VLMs will likely require coordinated designs across memory formation, representation, retrieval, consolidation, update, and governance, rather than treating memory as an external module attached to a pretrained model.

Cognitive-inspired Memory Architectures. Future VLMs may benefit from cognitively inspired but engineering-oriented memory hierarchies that separate transient perception, episodic experience, semantic abstraction, and reflective reasoning. Generative Agents demonstrated how observations and reflections can be organized into a persistent memory stream for later behavior [66], while MemoryBank converted interaction histories into long-term conversational memory [13]. KARMA introduced explicit short- and long-term memory modules for embodied agents [71], and brain-inspired multi-memory systems suggest that lifelong embodied intelligence may require several interacting memory stores [365]. Future VLMs could move beyond flat context accumulation by maintaining fast perceptual buffers, episodic records, semantic summaries, and reflective memory that are continuously consolidated over time. An important open question is how raw multimodal experiences should be transformed into reusable knowledge without losing critical evidence.

Hybrid and Adaptive Memory Substrates. Future VLMs will likely require hybrid memory architectures that dynamically combine multiple substrates rather than relying on a single storage mechanism. Raw observations preserve visual evidence, episodic memory captures event structure, semantic memory enables abstraction, latent memory provides computational efficiency, and external retrieval expands knowledge beyond model parameters. RAG and MuRAG established retrieval-based augmentation for textual and multimodal knowledge [15,65], while Reveal and retrieval-augmented multimodal language modeling demonstrated how multimodal knowledge memory can support reasoning [89,94]. VisRAG further showed that some information is better preserved as visual documents rather than converted into text-only representations [102]. Future systems should learn how to route information among different memory substrates while maintaining provenance, compression efficiency, and retrieval reliability.

Learned Memory Formation and Consolidation. Most existing VLM memory systems rely on manually designed rules for selecting, summarizing, or caching information. However, deciding what deserves to become memory is itself a fundamental intelligence problem. Future models should learn memory formation policies that determine when to write new information, what level of abstraction to preserve, and how repeated experiences should be consolidated into reusable knowledge. Such mechanisms could enable lifelong adaptation while preventing uncontrolled memory growth. Similar to human consolidation, future VLMs may need mechanisms that periodically reorganize accumulated experiences, resolve redundancy, and transform episodic memories into more general semantic knowledge.

Task-adaptive Memory Lifecycle Management. Memory policies should adapt to downstream objectives rather than rely on fixed compression ratios, static retrieval thresholds, or predefined eviction rules. VQA may require fine-grained object details, long-video understanding may require event-level summaries, navigation may require persistent spatial representations, and personalization may require stable yet editable user profiles. Model editing, continual VLM learning, and knowledge maintenance have begun to treat memory update as a central operation [85,111,112]. LOOK-M and MadaKV further demonstrated that online inference memory can be revised through retention and replacement strategies [42,43]. Future VLMs require task-adaptive controllers that jointly determine what to write, compress, retrieve, revise, and forget. Since long-lived memory inevitably accumulates outdated or conflicting information, future systems must also learn when to trust, reconcile, or discard existing memories.

Grounded and Provenance-aware Memory. Future memory representations should preserve links to their original evidence rather than storing only compressed vectors or summaries. A memory entry should ideally include its source image, video segment, dialogue history, timestamp, confidence, and retrieval trajectory. KRISP and Reveal demonstrated the value of explicit evidence for knowledge-grounded reasoning [89,92], while VisRAG preserved document pages as visual evidence during retrieval [102]. LongVideoBench, Video-MME, and MMLongBench further indicate that answer accuracy alone is insufficient when models cannot identify the supporting visual evidence [45,46,363]. Provenance-aware memory could therefore improve not only grounding but also hallucination diagnosis, error analysis, and user trust.

Efficient Long-context Memory Systems. Efficiency should be incorporated into memory mechanisms from the beginning rather than treated as a post-processing optimization. LongVILA demonstrated the importance of model–system co-design for long-context VLMs [38], while LongVU and VideoChat-Flash explored complementary strategies for long-video compression and efficient reasoning [114,115]. LOOK-M, MadaKV, and AKVQ-VL showed that KV-cache pruning, eviction, and quantization can substantially influence multimodal inference efficiency [42,43,88]. Future architectures should move beyond simply extending context length by combining video-token compression before reasoning, hierarchical memory retrieval during inference, adaptive cache management during decoding, and streaming memory updates as new observations arrive.

Unified Cross-modal and Cross-task Memory Frameworks. Future memory systems should become reusable across modalities and tasks rather than being redesigned separately for images, videos, language, navigation, and robotics. Perceiver and Perceiver IO introduced general latent representations for high-dimensional inputs [76,80], while Flamingo and BLIP-2 demonstrated how compact visual representations can interface with large language models [35,77]. PaLM-E extended this interface toward embodied multimodal reasoning [366], and RoboOS-NeXT suggested shared memory infrastructures for long-lived collaborative agents [73]. A common memory interface across vision, language, audio, action, and personalization could enable knowledge transfer across tasks while preserving modality-specific grounding and update mechanisms.

Memory-centric Evaluation and Governance. Future benchmarks should evaluate complete memory lifecycles rather than isolated retrieval or generation accuracy. Needle-in-a-haystack evaluations expose long-context retrieval failures, MRAG-Bench targets multimodal retrieval-augmented generation, OVO-Bench evaluates online video understanding, and MMLongBench expands long-context VLM evaluation [48,57,90,363]. Future benchmarks should jointly evaluate memory writing, compression, retrieval, updating, forgetting, provenance tracking, and cross-session reuse. Beyond accuracy, evaluation should measure whether users can inspect, correct, delete, and control persistent memories. This is particularly important for personalized multimodal assistants and contextualized user-memory systems [67,364]. Security mechanisms against malicious memory injection, stale information propagation, and unintended retention should become fundamental components of memory governance rather than deployment-time patches.

8. Conclusion

In this survey, we presented a systematic study of memory mechanisms in VLMs and introduced a unified 4D taxonomy that characterizes memory along four fundamental dimensions: when memory is formed and used, where it is stored, what information it represents, and how it is accessed and updated. Building upon this taxonomy, we reviewed representative memory architectures, learning strategies, evaluation benchmarks, and applications across multimodal understanding, long-video reasoning, retrieval-augmented generation, embodied intelligence, and personalized assistants.

Despite rapid advances, current memory-augmented VLMs remain far from achieving reliable lifelong multimodal intelligence. Key challenges include scalable memory management, adaptive memory formation and consolidation, efficient long-context reasoning, multimodal grounding and provenance, continual updating, and trustworthy evaluation. We envision that future VLMs will evolve from systems with extended context or external retrieval modules into memory-centric architectures capable of selectively acquiring, organizing, revising, and reusing multimodal experiences over time. Such adaptive and unified memory frameworks will be critical for building intelligent agents that can reason persistently, interact naturally, and continuously improve from experience.

To facilitate future research, we accompany this survey with a continuously maintained public repository of representative papers, benchmarks, and resources. We hope this survey provides a structured perspective on the emerging field of VLM memory and serves as a foundation for developing next-generation multimodal systems with persistent, grounded, and trustworthy intelligence.