Native Real-Time Interactive Omni Intelligence: A Unified Framework

1. Introduction

Artificial intelligence has progressed from language-centered models toward multimodal foundation models that can perceive, reason over, and generate content across multiple modalities such as text, images, audio, video, 3D geometry, dynamic 4D representations, and other forms of contextual information [1,2,3,4,5,6,7,8,9,10]. Recent omni models integrate several of these modalities within a shared backbone [11,12,13,14]. We use Omni Intelligence to describe this breadth of multimodal perception, reasoning, knowledge, and generation.

Many current omni models still formulate multimodal interaction as completed prompts or conversational turns [11,12,14]. The model receives a sufficiently complete context and produces a response afterward. However, in real-world interaction, audio and video continue to arrive while the model is reasoning or generating a response, and results from retrieval, tools, or external computation may arrive after the interaction state has changed.

We define Interactive Omni Intelligence as the ability to maintain and use omni intelligence while multimodal input, system output, and interaction state continue to evolve. The model must judge when the available evidence is sufficient, whether silence signals hesitation or turn completion, whether speech is addressed to the system, and how new input should affect an active response. Pauses, overlap, backchannels, interruptions, and revisions therefore carry both semantic and control information [15,16,17].

Current systems support live interaction through three primary ways. Adapted systems add streaming interfaces, modality modules, and interaction controllers to pretrained language or omni models. Cascaded VAD–ASR–LLM–TTS systems are a common instance of this category [18,19,20]. Model-native systems represent temporal events explicitly and learn speech timing, overlap, interruption handling, or multimodal generation within the model. This group includes speech-native full-duplex models and streaming omni input-output models [15,21,22,23,24]. Foreground–background systems connect a low-latency interaction process to slower reasoning, retrieval, memory, or tool execution [25,26,27]. Such systems can be combined with adapted or model-native systems to coordinate fast interaction with slower computation.

These designs balance intelligence and interactivity differently. Adapted systems retain strong pretrained reasoning and knowledge, although their interaction loop may depend on coordination across several interfaces. Model-native systems incorporate temporal events directly into their state updates. Their requirements include synchronized interaction data, additional streaming computation, and training that preserves existing capabilities [28,29]. Foreground–background systems allow acknowledgements, turn management, and interruption handling to proceed while slower computation continues. Returned results must then be checked against the latest interaction state. These competing demands during training and runtime give rise to the Intelligence–Interactivity Trade-off [30,31,32].

The central challenge is therefore to build systems that can continuously update their understanding of the world, coordinate processes operating at different temporal scales, and express their capabilities at the appropriate moment without sacrificing reasoning quality, responsiveness, or safety. By organizing existing work around this joint intelligence–interactivity perspective, we aim to provide a common conceptual foundation for comparing current systems, identifying missing capabilities, and guiding the development of omni models that can participate naturally and reliably in persistent, open-ended interaction.

We analyze two closely related requirements. Omni Intelligence concerns the system’s ability to integrate information across modalities, understand the situation, and reason about it. Omni Interactivity concerns its ability to respond appropriately as the interaction unfolds: deciding when to respond or act, when to wait, and when to stop or revise an ongoing response. These requirements are coupled because incoming information can change both what the system understands and how it should respond. Figure 1 summarizes the framework, from real-time interaction requirements and streaming representations to model construction, execution, evaluation, and multi-rate coordination.

We first define the field and position it relative to multimodal language models, spoken dialogue systems, streaming models, and interactive agents in Section 2. With this foundation, we further address three questions. First, how should continuous multimodal streams be represented, and how should models be built and trained to process them? Section 3 and Section 4 examine representations, architectures, data, and training objectives. Second, how can a system maintain multimodal understanding and coordinate its responses during ongoing interaction? Section 5 and Section 6 cover evolving state, grounding, reasoning, memory, responsiveness, turn-taking, interruption, and serving. Third, how should intelligence and interactivity be evaluated together, and how can their competing demands be managed? Finally, Section 7 covers evaluation, Section 8 analyzes capability and runtime tensions, and Section 9 develops the open research agenda.

2. A Framework for Interactive Omni Intelligence

As shown in Figure 1, our proposed conceptual framework organizes Interactive Omni Intelligence around two coupled components: Omni Intelligence and Omni Interactivity. This section develops the framework by distinguishing modality coverage from real-time interaction requirements and by positioning related system families according to how they implement interaction control.

2.1. From Omni Intelligence to Real-World Interactivity

Omni Intelligence concerns the breadth and integration of model capabilities across language, vision, audio, and other modalities [2,3]. Interactive Omni Intelligence adds a temporal dimension: the ability to use these capabilities and adapt responses as new information arrives and interaction unfolds. Research toward this goal draws on spoken dialogue systems [33,34,35,36], multimodal foundation models [1,2], streaming generation [21,22,37], low-latency serving [38,39], and interactive agents [40,41]. Although these areas often share modalities and components, they differ in their assumptions about input availability, response timing, and whether ongoing responses can be interrupted or revised. We therefore distinguish the modalities a system supports from how it processes incoming information and coordinates its responses over time.

We characterize real-time Interactive Omni Intelligence through three properties. First, Streaming Input: input can arrive incrementally through audio and video streams, text updates, changes in screen state, tool feedback, and other signals. Second, Evolving State: the system can incorporate new information into its interaction state while processing earlier input or producing a response. Third, Online Response Control: it can initiate, revise, or stop outputs while interaction is ongoing, including speech, text, tool calls, and interface actions, without waiting for a fixed end-of-turn boundary. Together, these properties distinguish continuous interaction from conventional turn-based multimodal inference, in which a system receives a completed input before generating a response. Figure 2 illustrates this distinction in the context of spoken interaction.

These properties describe interactive behavior without specifying how it is implemented. An adapted system can meet these requirements through streaming interfaces, external controllers, or orchestration around a pretrained model  [42,43]. We use native interactive omni model for models that learn temporal alignment, incremental multimodal processing, and interaction control as integral parts of their architecture and training. Interactivity describes system behavior during ongoing interaction, while nativeness identifies where the mechanisms supporting that behavior are implemented and learned.

This distinction positions the survey at the intersection of two research directions: full-duplex spoken language modeling and general omni and multimodal modeling. Full-duplex spoken language modeling studies temporal coordination, overlapping speech, interruption, and turn-taking, primarily in spoken interaction [34,36]. General omni and multimodal modeling supports a broader range of perceptual and generative capabilities, but many systems still assume completed inputs or discrete conversational turns [1,2]. Interactive Omni Intelligence brings these directions together: it asks how models can maintain broad multimodal understanding and generation while continuously incorporating new information, coordinating their outputs, and adapting to changes in the interaction. This interaction couples updates to the system’s understanding with response selection based on the updated state [44].

Within this conceptual framework, characterizing Omni Interactivity requires clarifying the relationship between response latency and continuous interaction. Endpoint detection [33] and inference optimizations such as KV-cache management [38,45], speculative decoding [46], and latency-aware serving [39] can reduce response latency without changing a system’s turn-based operation. Conversely, a system can process streaming inputs and support interruption while still requiring substantial computation. We therefore examine which input streams can update the system’s state, when outputs can begin or change, and which components control multimodal fusion, response timing, and interruption. This also distinguishes our scope from surveys of spoken dialogue, spoken language models, and full-duplex speech interaction [34,35,36], and provides a common framework for comparing speech-native, multimodal, modular, and agentic systems.

2.2. Taxonomy of System Families

We use the conceptual framework to examine four overlapping system families, followed by examples of how deployed systems combine them. They illustrate different combinations of modality coverage, temporal interaction capabilities, and the locus of interaction control. Figure 3 summarizes their relationships.

Cascaded real-time systems.

A conventional real-time assistant chains separately trained components such as voice activity detection, streaming ASR, an LLM, and incremental TTS [34,36]. Streaming recognition can build on online mechanisms such as RNN-Transducer and strong speech front-ends such as Whisper [18,47]. These systems are modular, deployable, and easy to debug, and sufficiently aggressive streaming can provide low perceived latency. However, timing, turn-taking, interruption, and recovery are typically coordinated outside the language model [36,42]. Textual interfaces between components may also discard information carried by prosody, laughter, hesitation, overlap, and backchannel timing [15,21]. Cascaded systems therefore provide a modular baseline in which real-time interaction is coordinated primarily by components outside the pretrained model.

Speech-native full-duplex models.

Speech-native full-duplex models represent user and model speech as synchronized or parallel streams and learn aspects of turn-taking, overlap, interruption, and backchanneling directly from conversational sequences. dGSLM [15], Moshi [21], and SyncLLM [48] illustrate this progression through two-channel dialogue modeling, parallel user–model speech streams, and language modeling synchronized to a continuous interaction clock. These models directly learn temporal coordination, primarily through speech, and provide methods that can inform broader multimodal interaction.

Omni input and output models.

Omni models broaden perceptual and generative capabilities across text, images, audio, and video. Qwen2.5-Omni [22], for example, uses block-wise audio and visual processing, temporally aligned multimodal position encoding, and a Thinker–Talker architecture supporting streaming text and speech generation; Qwen2-VL provides an important vision–language predecessor [49]. LLaMA-Omni [50], LLaMA-Omni2 [51], AnyGPT [52], Mini-Omni [37], Mini-Omni2 [53], VITA [54], and MiniMind-O [55] further illustrate efforts to unify multimodal perception with text or audio generation. However, modality breadth alone does not imply Interactive Omni Intelligence. Streaming output, for example, does not by itself establish that new input can continuously modify an ongoing response, that overlap and interruption are modeled, or that the system can revise its behavior as the interaction evolves. Our core scope therefore focuses on systems that combine these multimodal capabilities with the real-time interaction properties defined above.

Foreground–background and multi-rate systems.

This system family cuts across the preceding categories and focuses on coordinating components that operate at different computational rates. A fast foreground process remains synchronized with the interaction, while slower processes perform retrieval, reasoning, memory access, planning, or tool use asynchronously. Building on retrieval-augmented generation [56], MoshiRAG [26] and Stream RAG [27] demonstrate how retrieval or tool use can proceed while spoken interaction remains active. STITCH interleaves speech generation with internal reasoning, while EgoMem coordinates asynchronous retrieval, dialogue, and long-term memory processes [57,58].

We view this family as a broader pattern of Multi-Rate Interactive Intelligence. It directly addresses the Intelligence–Interactivity Trade-off developed in this survey: conversational interaction may require decisions on the scale of tens or hundreds of milliseconds, whereas reasoning, retrieval, memory consolidation, or tool execution may require substantially longer computation. Separating these processes allows slower computation to continue without blocking immediate responses, but requires synchronization, validation of returned results, cancellation, and updates to the interaction state. Multi-rate designs can appear in both adapted and model-native systems.

Deployed real-time systems.

Recent industry systems support ongoing multimodal interaction. Google’s Gemini Live [59] supports low-latency interaction over speech and visual inputs, allowing the system to respond as new speech and visual information arrives. Thinking Machines Lab’s Interaction Model [60] advances this direction by representing interaction as continuous, time-aligned multimodal streams, processing audio, video, and text through fine-grained micro-turns so that pauses, interruptions, overlapping speech, and visual events remain part of the model’s temporal context. OpenAI’s GPT-Realtime-2 [61] targets real-time conversational applications with native text and audio input/output, image input, tool use, and low-latency streaming, although its currently documented interface does not include native video input. Alibaba’s Qwen3.5-Omni-Realtime [62] extends real-time interaction to text, image, audio, and video inputs, supporting streaming spoken responses and multimodal grounding within a persistent interactive session. Similarly, ByteDance’s SeedRealtime [63] is designed as a native audio-visual full-duplex model that jointly processes continuous audio, video, and text streams, with an emphasis on natural conversational timing, visual awareness, and integrated tool use. These systems increasingly integrate multimodal perception, response generation, and temporal coordination within real-time models.

3. Streaming Multimodal Representations

Native interactive models must process information as it arrives, without assuming that all relevant input is already available. Audio, video, text, and tool feedback, therefore, require representations that preserve their timing and support incremental processing by the model backbone. This section reviews these representations and four consequences of their design: the token rate the model must sustain, the information it can retain, the minimum delay before new input can influence a response, and the data and objectives available for training. These choices affect both pillars of the survey. Token rate and encoding delay constrain omni interactivity, while the preservation of content and temporal structure supports omni intelligence, including prosodic understanding and temporal grounding. Training data and objectives shape both. Representation design is therefore an early source of the intelligence–interactivity trade-off, with implications for subsequent architectural and training choices. Each subsection examines how its mechanisms support the two pillars and what computational, informational, or latency costs they introduce.

We proceed from audio representations in Section 3.1 and streaming video representations in Section 3.2 to representations of time, silence, and overlap in Section 3.3. Section 3.4 then examines how multiple streams are serialized into model input sequences, while Section 3.5 covers the text, tool, and background reasoning streams that accompany ongoing interaction. The last group provides the representations needed to coordinate foreground interaction with background computation in the dual-model pattern discussed in Section 4.

3.1. Audio Representations

An interactive omni model converts continuous microphone input into features or tokens that the language-model backbone can process under a latency constraint. The audio front-end determines the token rate, the paralinguistic information retained, and the minimum encoding delay before the model can respond. Speech representations have been surveyed for spoken dialogue broadly [34,35] and for discrete audio tokens specifically [64,65]. We review mel-spectrogram features, self-supervised speech units, and neural codec tokens, then examine semantic and acoustic information, tokenization granularity, and the choice between discrete tokens and continuous features.

Mel features and their discretization.

Log-mel spectrograms provide frame-based time-frequency features for audio encoders such as Whisper [18] and the block-wise audio towers of omni models [22]. They retain paralinguistic information and are commonly used for speech understanding. dMel directly discretizes mel filterbank energies into intensity bins, producing a training-free tokenizer that supports ASR and TTS within a single decoder [66]. MELLE instead predicts continuous mel frames with a regression head, providing an alternative to vector quantization [67].

Self-supervised speech units.

A second family uses self-supervised learning to obtain representations of linguistic content without transcripts. Contrastive and masked-prediction objectives such as CPC [68], wav2vec 2.0 [69], HuBERT [70], w2v-BERT [71], and WavLM [72] produce frame-level embeddings whose intermediate layers align strongly with phonetic content [73,74]. Quantizing these embeddings, either with learned codes [75] or with offline k-means as popularized by HuBERT, yields discrete semantic units. GSLM demonstrated that a language model trained purely on such units can generate intelligible speech, establishing the textless pipeline of speech encoder, unit language model, and vocoder [76]. Semantic units provide compact representations of linguistic content, but their compression may remove speaker and prosodic information needed for expressive full-duplex conversation.

Neural codec tokens.

Neural audio codecs optimize waveform reconstruction at low bitrates, and their discrete code indices can be used as model tokens. SoundStream uses a fully convolutional causal encoder and decoder with residual vector quantization (RVQ), reconstruction and adversarial losses, and quantizer dropout. It compresses 24 kHz audio to 3–18 kbps in a single model and supports real-time streaming [77]. Subsequent codecs refine different parts of this design. EnCodec introduces multi-scale spectrogram discriminators and produces a 75 Hz token stream [78], DAC addresses RVQ codebook underuse with factorized, $L_2$-normalized codes and supports high-fidelity coding of 44.1 kHz general audio [79], and group-RVQ variants reduce the number of quantizers required [80], while lightweight inverse-STFT decoders accelerate token-to-waveform generation [81].

Because codec indices form a finite vocabulary, they can be modeled autoregressively. AudioLM applies autoregressive modeling to audio tokens [82], and VALL-E formulates zero-shot TTS as codec-token language modeling [83].

Semantic versus acoustic tokens.

Semantic tokens emphasize linguistic content and achieve relatively low token rates, but retain less information for waveform reconstruction. Acoustic codec tokens prioritize reconstruction fidelity, although their linguistic structure is often less explicit [34,64]. AudioLM combines the two in a coarse-to-fine hierarchy, generating semantic tokens for long-range coherence before acoustic tokens for waveform fidelity [82]. Subsequent tokenizers incorporate semantic information directly into codec representations. SpeechTokenizer distills HuBERT representations into the first RVQ level, leaving later levels to encode residual acoustic detail [84]. Mimi uses a split-RVQ design with a WavLM-distilled semantic codebook and parallel acoustic codebooks [21]. X-Codec incorporates SSL features to strengthen the semantic information carried by acoustic codes [85]. FACodec separates content, prosody, and timbre into quantized subspaces [86], while SemantiCodec represents general audio at fewer than 100 tokens per second [87]. For interactive models, these representations must often support both understanding and generation within the same audio-token interface.

Codebook depth, frame rate, and the latency versus quality trade-off.

When codebooks are serialized, frame rate and codebook depth determine the token rate, while total sequence length additionally depends on audio duration. Deeper RVQ stacks can improve reconstruction fidelity but increase the number of tokens that the model must process. VALL-E, for example, applies autoregressive prediction to the first codebook and non-autoregressive refinement to seven additional codebooks [83]; the serialization of multiple codebooks is discussed in Section 3.4. Other designs reduce token rates by using fewer codebooks. WavTokenizer represents speech with a single codebook at 40–75 tokens per second [88], while BigCodec achieves 1.04 kbps with a single codebook by increasing codec capacity [89]. TS3-Codec provides streaming single-codebook coding without convolutional layers [90], SNAC distributes codebooks across multiple temporal resolutions [91], and GLM-4-Voice uses a single-codebook tokenizer operating at 12.5 Hz for streaming spoken dialogue [92]. Native interaction also requires causal processing and low encoding latency. Mimi, the codec used by Moshi, provides one example through causal convolutions, a 12.5 Hz frame rate at 1.1 kbps, and approximately 80 ms of codec latency [21]. Moshi uses this codec within a full-duplex system designed to respond within approximately 200 ms.

Discussion: discrete tokens or continuous features?

The choice between discrete tokens and continuous features remains open. Controlled comparisons report advantages for continuous features on semantic understanding tasks and attribute the gap to quantization loss [93,94]. Dedicated benchmarks now quantify these gaps along complementary axes: Codec-SUPERB for reconstruction quality [95], DASB for the downstream utility of discrete tokens [96], and STAB for intrinsic tokenizer properties [97]. Discrete tokens support autoregressive speech generation, while continuous output requires an appropriate prediction head [67]. Some omni models therefore use continuous features for understanding and discrete codec tokens for generation [22]. Others tokenize both input and output to model the two speech streams jointly [21]. Continuous input features can preserve semantic and paralinguistic detail, while discrete output tokens support streaming generation. Fully tokenized designs must balance understanding quality with joint speech modeling. Section 4 extends this separation to foreground and background models. How these per-modality choices are arranged across model input and output streams is the subject of Section 3.4.

3.2. Video Representations

Streaming video models process frames as they arrive and may need to respond before the stream ends. This changes the focus from understanding a completed video [98,99] to determining when visual information becomes available and when it should trigger a response [100]. We review frame tokenization, sampling, temporal position encoding, bounded visual memory, and response triggering. Section 3.4 discusses synchronization with audio and text.

Patch tokens and connectors.

A standard video front-end divides each sampled frame into patches, encodes them with a vision transformer, and compresses the resulting tokens before passing them to the language model [101]. A single frame can produce hundreds of patch tokens, so video models use connectors to reduce the number processed by the language model. Cross-attention resamplers map variable-length spatial and temporal features to a fixed number of latent tokens [102,103,104]. Query-based bottlenecks such as the Q-Former provide another form of token compression [105]. More aggressive per-frame pooling can represent an entire frame with as few as two tokens, making hour-long video inputs more practical [106]. Multiplying the number of tokens per frame by the number of frames processed per second gives the visual token rate that the backbone must process. This is the video counterpart of the frame-rate and codebook-depth calculation discussed for audio in Section 3.1.

Frame sampling.

Uniform sampling is computationally simple but can miss brief events. A common baseline samples approximately one or two frames per second, so events occurring between sampled frames may not be represented. Adaptive keyframe selection prioritizes frames according to query relevance and temporal coverage under a fixed token budget [107]. Streaming systems must make sampling decisions online as frames arrive, which favors lightweight policies that do not depend on future frames. In systems that do not retain raw frames, discarded observations cannot be recovered [100].

Temporal position encoding.

Temporal position encoding represents when each observation occurred. Multimodal rotary position embeddings decompose position indices into temporal, height, and width components [49]. VideoRoPE examines which frequency bands should represent time and how temporal indices should be spaced between consecutive frames [108]. TMRoPE aligns temporal positions across audio and video by assigning the same temporal index to video patches and audio frames that share a timestamp [22]. This allows observations from different modalities to be aligned by absolute time. Section 3.3 examines explicit representations of time, silence, and overlap.

Streaming ingestion, memory, and token compression.

Streaming video is processed incrementally, so stored context grows unless it is bounded by memory or compression. Attention sinks with contiguous position indices support long streams when training follows the same streaming setting [109], while memory modules condense earlier observations into queryable spatiotemporal states [110]. Compression exploits redundancy across frames. Removing unchanged tokens can eliminate more than 80% of visual tokens with little reported accuracy loss [111], while training-free token merging combines redundant visual tokens [112]. These methods support long-running streams while limiting the stored context. Memory and compression determine which past observations remain available and how much memory and computation are required.

When to respond.

Streaming video also requires a decision about when to respond. Methods include per-frame prediction, dedicated response heads, separate perception and response modules, and activation thresholds. VideoLLM-online predicts either silence or the start of a response at each frame [113]. MM-Duet uses a dedicated response head that allows text to be inserted at different points during video playback [114]. Dispider separates perception, decision, and response generation [115], while StreamBridge and LiveStar apply activation or uncertainty thresholds to determine when a response should begin [116,117]. Early triggering can produce irrelevant responses, while conservative triggering can delay useful information. Section 8 examines this distinction through response-onset latency and time-to-informative-answer.

Discussion: coverage versus timeliness.

Video sampling and compression balance information retention against processing cost and response delay. Discarding tokens can remove evidence needed later. Reliable response timing depends on knowing when relevant events occur, making temporal grounding central to proactive interaction. Streaming benchmarks evaluate response timing alongside answer accuracy on timestamp-anchored queries [118,119], and current systems remain behind human performance on these evaluations [120]. Section 5 and Section 6 further examine temporal grounding and response timing. The mechanisms reviewed here determine which visual events are encoded and when their representations become available to the model. Section 3.3 next examines how time, silence, and overlap are represented as model inputs.

3.3. Time, Silence, and Overlap as Model Inputs

Cascaded systems commonly use external speech segmentation and endpoint detection before passing a speaker-attributed transcript to the language model. Native models can instead represent pauses, overlapping speech, and timing within their input streams, as illustrated in Figure 4. Turn-taking has a long literature in conversational systems [33,121], and full-duplex spoken language models have recently been surveyed [36]. This subsection examines the information required to learn response timing, while Section 6 discusses turn-taking behavior. In what follows, we first ground the discussion in human conversation science. We then trace the shift from reactive endpointing to predictive turn-taking, examine silence and overlap in turn, and review the mechanisms that make models time-aware; we conclude by discussing how temporal behavior is evaluated.

What human conversation implies for machines.

Human turn-taking is systematic: turns are constructed from recognizable units, and speaker change becomes relevant at projectable completion points [122]. Across ten languages, response offsets typically fall within a narrow window of 0 to 200 ms, with both long gaps and substantial overlap generally avoided [123]. Because utterance planning takes considerably longer than 200 ms, human listeners must anticipate turn completion from prosodic and linguistic cues and prepare responses before the speaker finishes [124]. Interactive models similarly benefit from retaining these cues and preparing responses before an endpoint is detected.

From reactive endpointing to predictive turn-taking.

Cascaded systems commonly reduce turn-taking to endpointing by declaring a turn complete after a fixed silence interval. However, a fixed silence threshold cannot reliably distinguish hesitation from turn completion across all conversational contexts. Continuous prediction provides a more flexible alternative. Recurrent models estimate future speech activity frame by frame [125], multiscale models combine acoustic and lexical cues at their respective rates [126], language models predict turn shifts from pragmatic completeness [127], and generative models learn distributions of response offsets [128]. Voice Activity Projection (VAP) predicts future speech activity for both speakers directly from two-channel audio without requiring turn annotations [16]. Unlike VAD, it predicts upcoming activity instead of detecting current activity and models both speakers jointly. Real-time and multilingual variants extend this approach to additional deployment and language settings [17,129].

Silence as an interaction state.

Acoustically similar silences can serve different interaction functions. A pause within a turn may hold the floor, while a gap between turns may transfer it. The floor-transfer offset measures the interval from the end of one speaker’s speech to the start of the other’s, with overlap represented by negative values [33]. Speakers can also use fillers such as “uh” to hold the turn during a pause [130]. Systems can use semantic and user-state information to decide whether a pause permits a response [42], a policy considered further in Section 6. In frame-based full-duplex models, silence can be represented as an output state, allowing the model to learn when not to generate speech [21].

Overlap and backchannels as normal input.

Overlapping speech is a normal part of conversation. Backchannels such as “yeah” and “mm-hmm” signal engagement without claiming the floor. Models can predict their timing and type by combining acoustic and language-model cues [131] or by adapting VAP for continuous backchannel prediction [132]. During model speech, the system must distinguish user acknowledgments, interruptions, and background speech [36,42]. This requires representations that distinguish incoming speech from the model’s output. Codec-free designs such as SALMONN-Omni explicitly account for the model’s own speech during simultaneous listening [133]. dGSLM demonstrates that some overlap and backchannel behavior can be learned from two-channel conversational audio without explicit turn labels [15].

Mechanisms for time awareness.

Models represent temporal information in several ways. Temporal position encodings associate tokens with timestamps. TMRoPE, for example, assigns a temporal index to each fixed audio frame and aligns audio and video using absolute timestamps [22], as discussed in Section 3.2. Fixed-duration chunks provide another way to align input and output over time. SyncLLM inserts synchronization tokens at intervals of approximately 160 to 240 ms and predicts upcoming user chunks to compensate for processing delay [48]. Parallel layouts align user and model tokens at corresponding time steps. Moshi represents both speakers as parallel token streams [21], LSLM integrates a streaming listening channel with its speaking channel [134], and NTPP jointly predicts token pairs across the two dialogue channels [135]. Models can also be trained directly for incremental decoding, as demonstrated by Speech ReaLLM [136]. Synchronization may be learned within the model or implemented by an external controller that determines whether the system should continue listening, respond, or stop [36,43]. Temporal representations support the reasoning about event timing examined in Section 5 and the response-timing decisions discussed in Section 6. Section 3.4 examines how these streams are serialized.

Discussion: measuring temporal behavior.

Conventional turn-based benchmarks often emphasize response content without evaluating continuous input processing or response timing. Dedicated evaluations provide broader coverage. Full-Duplex-Bench measures pause handling, backchanneling, turn-taking latency, and interruption behavior [137]; later versions add overlap handling [138] and multi-turn interaction with an automated examiner [139]. Talking Turns compares the turn-taking behavior of audio foundation models with conversational corpus statistics [140], while spoken-dialogue surveys summarize measures such as floor-transfer offsets, interruption success, and response delay [34]. Silence is usually studied within turn-taking and full-duplex systems. Evaluations should distinguish response onset from the delivery of useful content, since a fast initial response does not necessarily provide a timely answer. Learning temporal behavior requires representations that retain both speakers’ timing, including pauses and overlap. Stream formats that discard either timeline lose this information. Representing both streams at each time step increases token count, including during silence, and requires training data beyond conventional text turns. Section 3.4 examines the resulting trade-offs.

3.4. Unified Stream Formats

The preceding sections reviewed modality-specific streams and their timing. We now examine how these streams are arranged as model inputs and outputs. We group related formats into turn-level concatenation [34,141], time-aligned interleaving [35], and parallel streams [36], and compare their serialization mechanisms and trade-offs in Table 1.

Turn-level concatenation.

Turn-level concatenation appends complete text or speech segments. SpeechGPT sequences recognized text, a textual response, and speech units [142], while Spectron transcribes speech, generates a textual continuation, and then synthesizes it as audio [143]. VoxtLM combines task tokens with text and speech units in a single decoder [144], while USDM continues pretraining on speech-text data before turn-based dialogue fine-tuning [145]. Complete segments allow these systems to reuse standard text-language-model objectives and turn-based datasets. However, waiting for a segment or textual answer to finish delays speech output and can remove detailed information about speaker timing.

Time-aligned interleaving.

Time-aligned interleaving orders tokens from different streams within a sequence using timestamps or alignment boundaries. SUTLM compares speech-only, text-only, concatenated, and alternating layouts, and reports the strongest cross-modal transfer with word-level alternation [146]. SpiRit-LM interleaves text and speech units at word boundaries [147], while GLM-4-Voice uses chunk-level interleaving for streaming synthesis and trains on large-scale synthetic interleaved data [92,150]. A fixed text-to-speech chunk ratio controls look-ahead and latency, as demonstrated by IST-LM [148]. Interleaved models can be initialized from pretrained text models [151]; recent work also studies their scaling behavior [152] and dynamic turn-level interleaving [153]. These models require aligned training data and an explicit representation of both speakers to support overlap.

Parallel user and model streams.

Parallel layouts represent both participants at aligned time steps, thereby preserving the timing relationship between streams described in Section 3.3. dGSLM uses a dual-tower architecture with cross-attention between two conversation channels [15], while Moshi represents the user’s speech, the model’s speech, and a time-aligned inner-monologue text stream as parallel sequences [21]. SyncLLM combines interleaving with synchronization of fixed-duration chunks from both speakers [48]. NTPP jointly predicts token pairs across two dialogue channels [135], PSLM uses parallel text and speech sequences to reduce latency [149], Mini-Omni decodes text and audio in parallel, with text guiding audio generation [37], and LSLM integrates a streaming listening channel with its speaking channel [134]. Parallel representations also precede LLM-based dialogue, as pGSLM models speech units, pitch, and duration in separate channels [154]. These categories can overlap. OmniFlatten, for example, progressively converts parallel dialogue streams into a single GPT sequence [155]. Parallel-stream models require synchronized two-channel data and architectures adapted beyond conventional text language models [36].

Serialization mechanics.

Three serialization mechanisms recur across these formats. Multi-codebook outputs require a defined ordering. MusicGen’s delay pattern offsets codebook sequences so that they can be generated in a single pass [156], while stack-and-delay changes their ordering to balance latency and reconstruction quality [157]. Moshi instead combines a temporal Transformer with a smaller depth Transformer that predicts tokens across codebooks [21]. Text and audio sequences generally differ in length, requiring padding or delay policies. These include fixed-length padding in LLaMA-Omni [50], trailing padding in Mini-Omni [37], and a learned delay between Moshi’s text and audio streams [21]. Chunk ratio and look-ahead control how much text is available during speech generation [148]. Temporal position encodings can additionally align modalities through absolute time, as illustrated by TMRoPE in Section 3.2 [22].

Discussion: serialization trade-offs.

The formats impose different buffering requirements, including complete segments, aligned chunks with look-ahead, and synchronized frames. End-to-end latency also depends on implementation and compute. For example, Moshi operates with 80 ms codec frames, while SyncLLM uses synchronized chunks of approximately 160 to 240 ms [21,48]. Temporal detail depends on alignment granularity, and linguistic consistency can depend on text supervision. Textless parallel models can capture overlap and laughter but may lose semantic information [15]. Text guidance, as in Moshi’s inner-monologue stream, can improve linguistic quality within a parallel-stream design [21]. Concatenated formats use conventional turn-based data, interleaved formats require aligned sequences, and parallel formats require synchronized conversational streams [36]. Serialization therefore affects response timing, information retention, and compatibility with pretrained text models. The balance among these factors depends on the model architecture, training data, and implementation.

3.5. Speech Synchronization and User Interruption

Ongoing interaction can involve partial transcription, internal text generation, tool calls, and retrieval. These topics have dedicated surveys [40,41,158,159,160,161]. Here, we examine how these streams are synchronized with speech and handled when the user interrupts. Table 2 summarizes the stream types and their synchronization requirements.

Partial transcript streams.

Incremental transcripts can change as new audio arrives. Transducer models support streaming recognition through monotonic alignment [47,162]. Stabilizing partial recognition results balances transcription accuracy against the delay before results are treated as final [163]. In addition to producing transcripts through a separate recognizer, native models can align text directly with an audio stream. Delayed Streams Modeling pairs input audio with a time-aligned delayed transcript [164]. Moshi uses a related alignment mechanism for a different purpose, pairing inner-monologue text with the model’s own speech [21]. Systems must decide when a transcript prefix is reliable enough to use for retrieval or tool selection.

Inner monologue: reasoning while listening and speaking.

Text generated just ahead of audio can improve linguistic consistency, as demonstrated by Moshi [21]. Some models also generate separate, unspoken reasoning sequences while input or audio playback continues. SHANKS generates chain-of-thought while the user is speaking and reports that more than half of its required tool calls can be issued before the user finishes [165]. Think-While-Listening evaluates text-space reasoning for spoken responses and estimates when a question is sufficiently complete to begin reasoning [166]. STITCH alternates unspoken reasoning chunks with spoken chunks, allowing reasoning to continue during audio playback [57]. Other methods reason in hidden states during the user’s turn [167] or use pauses in the model’s speech for additional reasoning [168]. These methods perform reasoning in text or hidden states while receiving input or playing speech, allowing computation to overlap with ongoing interaction.

Tool-call and tool-result streams.

Tool use adds outgoing call streams and incoming result streams. Text-based agents commonly interleave reasoning, actions, and returned observations [169,170], while Toolformer and ToolLLM train models to decide when and which tools to invoke [171,172]. Tool-learning surveys further organize this process into planning, tool selection, invocation, and result integration [159]. Spoken interfaces introduce additional errors. Speech recognition can produce incorrect tool arguments, and evaluations compare the resulting tool-use performance with text-based inputs. VoiceAgentBench reports that cascaded ASR-plus-LLM systems outperform end-to-end speech models on voice-based agentic tasks [173], while conversions of text tool benchmarks into spoken form reveal a consistent modality gap [174]. Full-duplex benchmarks also evaluate multi-step tool use under realistic speech disfluency [175].

Background retrieval and reasoning.

Retrieval augmentation grounds responses in external knowledge [56]. Retrieval timing determines when external information becomes available. FLARE triggers retrieval when a look-ahead draft is uncertain [176], while Self-RAG uses reflection tokens to decide when to retrieve or critique generated content [177]. MoshiRAG launches retrieval asynchronously and supplies the returned text to the foreground model while speech interaction continues [26]. This can overlap retrieval with early response generation. Section 8 discusses the resulting distinction between response-onset latency and time-to-informative-answer. Stream RAG predicts a retrieval query while the user is still speaking [27]. Speculative interaction agents issue tool calls early and process input and output asynchronously [178], while streaming retrieval methods estimate when a speculative query is sufficiently stable to execute [179]. Other asynchronous designs allow a voice foreground to continue speaking while a background model reasons [180]. These systems coordinate a fast foreground interaction process with slower background retrieval, tool use, or reasoning. The design can reduce blocking delays, but requires a mechanism for validating and incorporating results during ongoing interaction. Section 4 examines this foreground-background pattern [25,26,27,60].

Interruption: cancel, delay, or roll back.

Interruptions require explicit handling of generated but unplayed speech, running tool calls, and pending retrieval. Systems may cancel pending work, retain useful results, or undo reversible effects. ChipChat tracks which generated tokens have been played, allowing the context to be rolled back to the output that the user actually heard [181]. Interruptible systems may stop spoken output, steer or abandon ongoing reasoning, and update their behavior after new user input  [165,180], while full-duplex benchmarks evaluate tool use under interruption [175]. Open problems include preventing duplicate calls and managing state after interrupted execution. Section 6 discusses serving-side operations such as stopping playback and rolling back caches, while Section 9 considers cancellation, deduplication, and reversible side effects as broader research problems.

Discussion: speech-aligned and asynchronous streams.

Table 2 summarizes these streams by function, timing relative to speech, and interruption handling. Text streams can be aligned with speech, while tools and retrieval complete asynchronously. Their requests and results must be tied to the interaction state and inserted at appropriate points. Partial transcripts and inner monologues follow the speech-aligned formats discussed in Section 3.4. Dual-model architectures discussed in Section 4 separate immediate interaction from slower computation, introducing coordination and cancellation requirements. Failures include premature commitment, delayed reasoning, outdated intent, and inconsistent state after interruption. Representation choices determine which activities can be interrupted and resumed consistently. Section 9 examines the remaining cancellation and state-management problems.

4. Building Native Omni Models for Real-Time Interaction

Native real-time omni models operate while interaction is still unfolding. User speech may continue after response generation begins, visual evidence may become relevant midway through a response, and results from reasoning or tool use may arrive after the conversational state has changed. The central design problem is to update the interaction state with new multimodal evidence while controlling ongoing output, interruption, and revision. This section examines the problem through architecture, data, and training. We first compare architecture families and fusion mechanisms, with attention to where the live state is maintained and how low-latency interaction is coordinated with slower computation. We then discuss the data and objectives required to learn temporally grounded behavior, including silence, overlap, interruption, and foreground–background coordination.

4.1. Architecture Families and Modality Fusion

We compare architectures by which components observe incoming events, maintain the interaction state, and control response timing. Four families organize the current design space: cascaded baselines, speech-native full-duplex models, omni input-output models, and dual-model interaction systems. Figure 5 shows how these four families implement fusion, speech generation, temporal control, and background computation.

Cascaded baselines.

The most widely deployed real-time voice architecture chains independently trained modules: voice activity detection (VAD) or endpointing, streaming ASR, an LLM, incremental TTS, and playback control. Online recognition mechanisms such as RNN-Transducer support streaming input [47], while ASR front-ends such as Whisper provide strong transcription quality for speech interfaces [18]. Incremental TTS can begin synthesis from partial text and reduce audible response delay [182,183]. The strength of this design is modularity: each component has a defined interface, can be improved independently, and can be monitored or replaced without retraining the whole interaction system.

These interfaces also determine where the live interaction state can reside. At the ASR-to-LLM boundary, converting audio into text hypotheses and metadata can remove prosody, hesitation, laughter, overlap, and backchannel timing before they reach the language model. At the LLM-to-TTS boundary, generated text becomes an audio queue whose timing is further shaped by synthesis and playback buffering. Interruption is usually handled by runtime logic that stops playback, discards queued speech, and cancels or restarts generation [36,184]. Cascades can therefore be responsive when their components are carefully scheduled, but turn-taking and overlap remain forms of engineered synchronization outside the model [36]. Work such as Voice Activity Projection shows that conversational timing can instead be learned through predictions of future voice activity [16], while full-duplex benchmarks directly evaluate pause handling, backchanneling, turn-taking, and interruption management [137]. A useful middle ground consolidates the front end itself: UAF combines several front-end functions within a single model, including VAD, turn detection, speaker recognition, ASR, question answering, and interruption-related control signals, predicting them within a single streaming autoregressive model while retaining a separate generation module [185]. Native architectures can learn these functions within shared stream representations and interaction state.

Speech-native full-duplex architectures.

Speech-native full-duplex models represent user speech and model speech as time-aligned streams that can be active during the same interval [15,21,48]. Silence, pauses, overlap, backchannels, and interruptions therefore become temporal conditions visible to the model. Their state can update from incoming audio while speech generation continues, allowing the model to condition turn-taking and interruption responses on both streams. Interaction events can consequently be represented as stream events: a backchannel is aligned with ongoing user speech, an interruption occurs while the model-speech stream is active, and yielding changes the model-speech stream in response to the user.

dGSLM provides an early foundation by modeling two-channel dialogue directly from conversational audio, showing that turn-taking can be learned from paired speech streams without first reducing the exchange to text [15]. Moshi jointly models user speech, model speech, and an inner-monologue text stream over neural codec tokens [21], while SyncLLM uses fixed-duration dialogue chunks [48]. Freeze-Omni preserves a text-capable LLM backbone and adds streaming speech modules with chunk-level state prediction [43]. SALMONN-omni uses continuous speech embeddings without codec tokens [186]. Mini-Omni and Mini-Omni2 provide smaller open-source implementations with speech encoders, speech-token decoders, delayed parallel generation, and lightweight interruption signals [37,53]. BayLing-Duplex shows that full-duplex interaction control need not require a redesigned representation: it is learned within an otherwise standard autoregressive speech–language model using only a small set of control tokens, so that a single model decides when to listen, speak, and stop without a separate turn-taking module while retaining the backbone’s response capabilities [187]. These designs differ in stream coupling, in the use of text to guide speech generation, and in the choice of speech representation: discrete units, codec tokens, or continuous embeddings. Their main limitation is modality breadth: most remain centered on audio, while video, screen state, tools, memory, and background reasoning are external or absent.

Omni input-output architectures.

Omni input-output systems must coordinate text, visual, and audio information arriving at different rates, extending the temporal problem from speech-centered interaction to images, video, screen context, and spoken response [22,23,24]. Audio is processed in chunks, video as frames or visual tokens, and text as instructions or intermediate representations. A visual event may change when the model should speak, while an acoustic cue may determine whether it should continue or pause. A central design challenge is to align these inputs while keeping speech generation responsive to new evidence [22,25].

The Qwen-Omni family illustrates how this architecture has evolved. Qwen2-VL introduced dynamic-resolution visual processing and multimodal RoPE [49]. Qwen2.5-Omni extended this approach with block-wise audio and visual encoders, time-aligned multimodal positional encoding, and a Thinker–Talker architecture for streaming text and speech generation [22]. The Thinker produces text-oriented hidden states that condition the Talker, allowing speech to reflect the current multimodal state before a complete text response has been emitted. Qwen3-Omni replaces fixed two-second audiovisual blocks with temporal IDs anchored to absolute time, directly aligning audio and video representations and supporting streaming inputs of arbitrary duration [23]. Qwen3.5-Omni retains the Thinker–Talker structure while introducing Hybrid MoE backbones for long audio-video sequences and ARIA, which adaptively interleaves text and speech units during streaming generation [24].

Other systems use different interfaces between multimodal perception, language modeling, and speech generation. LLaMA-Omni and LLaMA-Omni2 connect speech encoders and LLM backbones to streaming speech decoders [50,51], while VITA, AnyGPT, and MiniMind-O explore broader or smaller-scale routes to multimodal input and spoken output [52,54,55]. These systems vary in modality coverage, scale, and how speech decoders receive the model’s current representations. Talker modules, hidden-state handoffs, codec-token paths, and streaming speech decoders provide different degrees of access to the live model state, and their interfaces affect how quickly new visual or acoustic information can influence generated speech, with corresponding costs in alignment, latency, and cancellation. Recent work also begins to extend this output path beyond text and speech: Ex-Omni-2D generates coordinated text, personalized speech, and reference-conditioned streaming video, using speech units as a shared acoustic–temporal interface between the dialogue and visual-generation pathways [188]. These choices form part of the broader fusion and coupling problem discussed next.

Dual-model interaction systems.

Dual-model interaction systems separate immediate interaction from slower computation [25,60]. A foreground model processes ongoing input and controls speech, pauses, acknowledgments, and interruptions, keeping the user-facing state coherent. A background model handles longer reasoning, retrieval, browsing, tool use, planning, or memory operations without blocking the live exchange. Thinking Machines Lab [60] provides a direct formulation of this pattern. Its time-aware interaction model processes continuous audio, video, and text as micro-turns, while an asynchronous background model handles longer-horizon reasoning and tool use. The foreground remains active as this work proceeds and decides when returned results can enter the conversation.

DuplexOmni provides a recent research implementation of the same separation. Its interaction layer processes streaming audio and video while generating text and speech in real time, whereas a replaceable reasoning component returns reasoning and tool-use results asynchronously [25]. Other systems use specialized background components. MoshiRAG combines a compact full-duplex speech model with selective asynchronous retrieval, allowing response onset to precede the arrival of external evidence [26]. Stream RAG predicts tool queries during ongoing user speech and later fuses retrieved text into spoken responses [27]. STITCH interleaves unspoken reasoning chunks with spoken chunks so that reasoning can continue during speech playback [57], while EgoMem adds asynchronous retrieval and memory management to real-time omnimodal dialogue [58]. StreamArena introduces StreamMind, a streaming architecture that separates latency-critical interaction from persistent multimodal recall and longer-horizon processing over hour-scale video [189]. These systems do not all use two independently scaled models, but they share the same temporal separation between immediate interaction and slower computation.

The defining challenge is the interface between the live and background processes, as illustrated in Figure 6. The foreground must decide when to delegate, what context to send, and whether returned results should be inserted, revised, delayed, or discarded as the live state changes, and must validate them against the current interaction state before incorporating them. Starting audio quickly does not necessarily reduce the time until the user receives a useful answer, so explicit policies for insertion, validation, cancellation, and recovery are required.

Modality Fusion and Coupling Approaches.

Fusion describes how modality representations are combined, whereas coupling describes how strongly and continuously streams influence one another’s content and timing. These design choices can be combined in different architectures. A central question is how much temporal and modality-specific structure remains visible across architectural boundaries.

Encoder-based fusion.

Encoder-based fusion converts speech, audio, images, or video into learned features before projecting them into an LLM or multimodal backbone. Pretrained encoders such as Whisper, CLIP, and SigLIP provide strong semantic representations [18,190,191]. LLaMA-Omni, the Mini-Omni series, and Qwen2.5-Omni follow this pattern [22,37,50,53]. Its main advantage is stable and sample-efficient training. The encoder determines which temporal details are retained and how frequently the model state can update, from once per turn to once per chunk or frame: turn-level features suppress within-turn timing, whereas chunk- or stream-level features preserve more local evidence at greater alignment and memory cost. Separate modality encoders can also process audio and video at mismatched rates. OmniEncoder jointly encodes audio and video at 25 frames per second [192].

Discrete-token and codec-token fusion.

Discrete-token fusion converts continuous signals into token sequences or parallel codebook streams. Neural codecs such as SoundStream and EnCodec provide this interface for speech [77,78]; Moshi uses codec tokens for parallel user and model speech streams [21], while AnyGPT extends discrete-token modeling across several modalities [52]. A common token interface makes autoregressive perception and generation easier to combine, but high token rates and multiple codebooks increase decoding and memory costs as well as first-audio latency. Time-aligned user and model streams can represent silence, overlap, interruption, and yielding within the same representation [15,21]. Codec-token language modeling is not the only route to overlapping conversational audio: Alignment-Free Text-Audiobox instead uses a latent diffusion formulation with flow matching and learns text–speech alignment through cross attention rather than forced alignment, supporting long-form full-duplex dialogue synthesis with turn-taking, backchannels, and emotional dynamics [193].

Early or encoder-free fusion.

Early fusion uses lightweight projections of lower-level inputs and jointly trains the backbone to learn their perceptual representations, preserving local acoustic structure, visual change, and fine timing. Native multimodal scaling and encoder-free vision-language studies suggest that this approach can remain competitive when supported by sufficient data and joint training [194,195]. Thinking Machines Lab’s interaction model provides a real-time example by feeding dMel audio features and patch-level image features through lightweight embeddings into a jointly trained transformer [60], allowing incoming observations to update the model state more directly, while requiring more extensive joint training.

Intermediate fusion and expert routing.

Intermediate fusion can operate on encoder features or discrete tokens, retaining separate computation for each modality while allowing information exchange through shared attention, cross attention, adapters, or shared experts. JavisDiT illustrates this pattern in joint audiovisual generation: audio and video tokens interact through shared self-attention before passing through separate FFNs for each modality [196]. For speech and text, MoST combines dedicated experts with shared experts to support information transfer across modalities [197]. DeepOmni assigns experts by their routing statistics: for each MoE layer it measures the fraction of speech tokens and of text tokens routed to each expert, designates as speech experts those with high speech load and low text load, and keeps the remainder as text experts, alongside a small set of shared experts that both modalities use [198]. Cross-modal exchange occurs through these shared experts and through the attention layers, which all tokens pass through regardless of modality. Training proceeds in stages: the router is frozen while speech and text experts are trained separately on their own modality, then unfrozen for joint training on cross-modal data, which limits the loss of language capability during speech adaptation. These mechanisms preserve modality specialization and support interaction within the backbone, but introduce choices about which layers exchange information, routing balance, and temporal synchronization; related routing mechanisms have also been applied to social evidence in omni models [199].

Coupling strength and speech-output paths.

Loose coupling transfers completed segments, such as ASR text, visual summaries, or tool results. Block-level coupling updates the state as chunks arrive. Shared-temporal-grid coupling aligns multiple active streams that arrive at different rates, such as audio chunks every few tens of milliseconds and video frames at a lower rate, on one time grid, preserving overlap and within-turn events. Qwen3-Omni aligns audio and video through temporal IDs anchored to absolute time [23], while Thinking Machines uses interleaved 200-ms input-output micro-turns [60]. ROMA instead groups frequent audio chunks and less frequent video observations into synchronized multimodal units and adds a speak head that predicts response timing separately from linguistic content [200]. These mechanisms increase temporal control but also increase synchronization, memory, and scheduling costs.

Speech decoders can condition on generated text, hidden states, or codec tokens. Text-to-TTS paths provide modularity but pass through a completed or partial text boundary, whereas hidden-state handoffs, Talker modules, codec-token paths, and streaming speech decoders connect speech more directly to the active model state. Qwen3.5-Omni’s ARIA further adapts the interleaving rate between text and speech units during streaming generation [24]. More frequent updates can let new evidence influence speech sooner, while requiring explicit handling of already generated or queued audio after interruption.

Dual-model coupling.

Dual-model coupling defines how the foreground sends context to background processes and incorporates returned results. Context leaves the foreground through a request or context package, while reasoning, retrieval, tool, or memory results return through prompt updates, tool-result streams, or other conditioning inputs. Because the live conversation may change before a result returns, the interface must support insertion, validation, cancellation, and revalidation [25,60].

These choices affect temporal detail, modality specialization, training requirements, decoding cost, routing stability, and coordination. Table 3 compares their interfaces, timing, benefits, and risks.

4.2. Data

Training data determine which interaction behaviors and representations a model can learn. General speech and audiovisual corpora provide abundant supervision for perception and generation [201,202,203,204], but native interaction requires temporal supervision such as separate speaker tracks, overlap and silence, event timestamps, response windows, interruption outcomes, and asynchronous execution traces. Figure 7 summarizes this hierarchy, while Table 4 reports representative datasets and training-data pipelines.

Speaker-separated dialogue data.

Speaker-separated dialogue provides the most direct supervision for turn-taking, overlap, interruption, backchannels, and conversational silence. Fisher is a widely used example, with more than 16,000 conversations and roughly 2,000 hours of two-channel telephone audio [205]. This dataset has been used by dGSLM, Moshi, and SyncLLM to learn multistream conversational timing [15,21,48]. Switchboard and CallHome provide related telephone conversations [206,207], while the Open-source Full-Duplex Dataset, DialoSpeech, and HumDial add human-recorded or large-scale dual-track dialogue, interaction annotations, and interruption and overlap data [208,209,210]. DuplexChat increases the available data by automatically extracting and restoring two-speaker tracks from public podcasts, reporting approximately 282,634 hours of English and 132,723 hours of Japanese dialogue [211]. Despite this increase in scale, the category retains important limitations. Classic corpora often contain narrowband, two-person telephone conversations, primarily in English, while automatically recovered tracks are not equivalent to directly recorded multichannel interaction. High-fidelity multilingual and multiparty recordings, together with fine-grained annotations of addressee, interruption intent, backchannel function, and floor negotiation, remain scarce.

Foundational speech and spoken-instruction data.

Single-channel speech supports training for ASR, TTS, audio codecs, and speech representation learning. Representative resources include LibriSpeech [201], Libri-Light [212], LibriHeavy [213], LibriTTS [202], GigaSpeech [215], Common Voice [219], VoxPopuli [220], YODAS [221], Emilia [222], and WenetSpeech [223]. At a larger scale, Moshi reports pretraining its audio stream on approximately seven million hours of mostly English speech transcribed with Whisper [18,21]. These corpora support perception and generation but generally do not preserve separate speaker timelines or explicit turn-control annotations. Spoken-instruction datasets connect these capabilities to assistant behavior. VoiceAssistant-400K [37], InstructS2S-200K [50], SpeechInstruct [142], AnyInstruct-108K [52], and LLaMA-Omni2’s multi-turn speech data [51] are largely produced by generating or rewriting text instructions and rendering them with TTS. These datasets support training for spoken instruction following and conversational response generation, but their clean request–response structure provides little direct supervision for overlap, interruption, or deciding not to speak.

Temporally aligned audiovisual data.

Omni models require audio and visual streams whose temporal relationship is preserved. Ego4D [203] supplies egocentric video with audio and annotations, HowTo100M [204] provides narrated instructional video at web scale, and AVSpeech [231] and VoxCeleb [232,233] associate visible speakers with speech. These resources support audiovisual grounding but often rely on clip-level captions or weak temporal alignment. Qwen2.5-Omni uses temporally interleaved audio-video training for synchronized multimodal processing [22], while ROMA curates streaming data for reactive and proactive audiovisual interaction [200]. Nevertheless, these model-specific mixtures do not resolve the broader data gap. Native interaction requires event-level supervision of what occurred, when it occurred, and whether it should trigger, delay, or suppress a response, while large-scale natural audiovisual conversations with such annotations remain scarce.

Synthetic interaction traces.

Synthetic data supplies interaction patterns that are difficult to collect naturally at scale. A common pipeline generates or rewrites dialogue with a text model and renders it as speaker-separated speech with controlled silence, overlap, backchannels, speaker identity, and noise. Moshi reports more than 20,000 hours of synthetic multistream speech [21], while SyncLLM progressively introduces pauses, overlapping speech, and natural Fisher conversations during training [48]. OmniFlatten and LLaMA-Omni similarly synthesize multi-turn spoken dialogues through LLM rewriting and TTS [50,51,155].

DuplexOmni extends this approach with a Writer–Director pipeline for continuous multimodal interaction data [25]. The scarcity of synchronized conversational data is compounded by the fact that appropriate turn-taking behavior is scenario-dependent. DuplexGen calibrates synthetic human–AI dialogues using a small amount of human turn-taking preference supervision, producing task-specific timing patterns rather than one universal interaction distribution [238]. The resulting overlap and interruption patterns differ between cooperative and competitive settings, showing why training data should include the context in which a timing behavior is appropriate. More generally, synthetic traces provide scale and control over interaction patterns, but may not reproduce natural disfluency, prosodic coordination, interruption intent, or human timing distributions. Systems such as Moshi and SyncLLM therefore continue to mix synthetic traces with genuine speaker-separated recordings when available [21,48].

Interaction-policy and safety data.

Interaction-policy data supervises when the model should speak, wait, acknowledge, yield, or stop. This layer is smaller and less standardized than the preceding ones. Existing resources include ranked speech candidates for quality and pause control [22], preference pairs for audio-conditioned text responses [239], preference pairs for speech generation [240,241,244], and interaction-specific segments or trajectories for optimizing turn-taking, backchanneling, and interruption [242,243].

Silence is also an explicit positive training target. Moshi, for example, continuously predicts both speech and silence [21]. Training examples should include appropriate silence, brief acknowledgment, yielding after interruption, and abandoning an outdated response, as well as cases where a response must be stopped or corrected when unsafe intent becomes apparent. Safety data must additionally cover streaming refusal, bystander speech, and privacy-sensitive audiovisual input. Most interaction-policy and streaming-safety datasets remain system-specific or unreleased.

Dual-model traces.

Dual-model systems require execution traces that go beyond ordinary dialogue pairs. A trace must record when the foreground delegates, what context is sent, when background reasoning or tool results return, what has already been spoken, and whether the result is inserted, revised, delayed, or discarded. MoshiRAG and Stream RAG provide examples of such coordination traces for asynchronous retrieval and tool use [26,27], while STITCH and EgoMem extend them to background reasoning and long-term memory [57,58]. The scarcity of public traces limits the reproducibility of training for interruption, cancellation, stale-result detection, and recovery.

Cross-cutting data challenges.

Across these categories, three limitations remain. First, speech datasets are growing, but high-quality multilingual and multiparty interaction remains scarce relative to single-channel speech. Second, timestamp and speaker-attribution errors directly corrupt supervision for silence, overlap, interruption, and audiovisual response timing. Third, current data is dominated by English and a small number of conversational settings, leaving other languages, accents, and social norms underrepresented; data should also better represent bystander speech and how participants coordinate turns.

4.3. Training Objectives

Training objectives specify how models learn multimodal representations, response generation, and interaction decisions. Native real-time omni models must generate coherent text and speech, align evidence across modalities, control when to speak or remain silent, respond to interruption, and coordinate computation that may finish after the conversational state has changed. We group these objectives into four categories: content and cross-modal grounding, temporal control, interaction-policy alignment, and concurrent reasoning and coordination. Safety and balancing between objectives apply across these categories. Figure 8 maps these objective families onto events in a live user–model interaction.

Content and cross-modal grounding.

Autoregressive likelihood remains the core objective, with targets drawn from text tokens, semantic speech units, codec tokens, acoustic tokens, and interaction-control symbols. Multi-codebook speech requires prediction across both time and codebook depth. Moshi factorizes these dimensions with temporal and depth Transformers [21], while Mini-Omni uses delayed parallel codebook prediction [37,91]. Text representations can condition incremental speech generation, as in text planning streams and Thinker–Talker handoffs [21,22,92].

Reconstruction and speech–text alignment losses support this objective without defining the interaction policy themselves. Codec reconstruction losses train waveform fidelity so that predicted speech units are acoustically usable [77,78], while auxiliary CTC losses align acoustic and linguistic representations in systems such as Freeze-Omni, VITA 1.5, and LLaMA-Omni [43,50,245]. These losses establish perception and generation capability; temporal and policy objectives determine how that capability is used during interaction.

Temporal control.

Temporal objectives supervise when the model should speak, continue, wait, acknowledge, yield, or stop. dGSLM separates content and duration prediction, using cross-entropy for speech units and an L1 loss for their durations [15]. SyncLLM predicts fixed-duration chunks, including silent chunks, while Moshi continuously predicts speech or silence on aligned streams [21,48]. These formulations make pauses, overlap, and response onset part of the prediction target.

Other systems supervise interaction state more explicitly. Freeze-Omni uses chunk-level state classification for continued listening, interruption, and response regeneration, while SALMONN-omni combines speech-boundary tokens with a weighted loss over ambiguous intervals during which the model should delay responding [43,186]. For audiovisual streams, ROMA applies a weighted binary cross-entropy objective to a speak head at each multimodal time step, using event windows or segment boundaries as response-timing labels [200]. These objectives explicitly train response timing and interruption handling, moving them from runtime heuristics into supervised model decisions.

Interaction-policy alignment.

Supervised likelihood encourages imitation of observed interaction sequences, but it does not directly optimize whether a pause, acknowledgment, or interruption response is well timed. Preference optimization first addressed speech quality and response behavior. Qwen2.5-Omni applies DPO to speech candidates ranked by word-error and pause-error measures [22], while Qwen2-Audio, Align-SLM, and SpeechAlign use preference signals for factuality, generation quality, or speech alignment [239,240,241,246,247].

Recent methods optimize interaction dynamics directly. ASPIRin uses speech–silence actions and timing-oriented GRPO rewards [242], while Multi-Faceted Interactivity Alignment applies reinforcement learning with axis-specific rewards for pause handling, turn-taking, backchanneling, and user interruption, and adds an LLM-based response-quality reward to prevent semantic degradation [243,244]. DuplexPO further factorizes the reward across turn initiation, acknowledgments, yielding, and regularized participation [248]: the turn-initiation term decays with the delay between the reference and predicted response onset, the acknowledgment term rewards agent speech that falls inside the annotated backchannel interval, the yielding term penalizes speech that continues beyond a tolerance after a user interruption, and the participation term penalizes undesirable interaction patterns. Credit is assigned locally by optimizing only the tokens inside short windows around each annotated event, with the preceding history teacher-forced. These methods use separate rewards for interaction behaviors and for response quality, and DuplexPO additionally keeps the policy close to its supervised initialization through a KL term. Without an objective that preserves content quality, timing optimization can reduce response quality.

Concurrent reasoning and coordination.

Training must also support reasoning during ongoing input or output. FLAIR trains a recursively updated latent reasoning state during user speech through an evidence-lower-bound objective and teacher forcing, avoiding the need for explicit reasoning traces [167]. This provides a single-model route to reasoning during perception.

Dual-model systems expose the coordination decisions directly. DuplexOmni uses temporally annotated control events for thinking requests, waiting, interruption, speech cutoff, silence, and delayed result delivery, and trains these events through likelihood objectives over Writer–Director traces [25]. More generally, supervision for dual-model systems covers background requests, the context sent, when returned results are incorporated, and how outdated work is canceled or revalidated. A background result may no longer match the live intent when it returns, so correctness depends on both its content and its validity at the time of insertion. Standardized objectives for stale-result detection and recovery remain limited. Table 5 summarizes the four objective categories by their training signals and the interaction decisions they supervise.

Safety and objective balancing.

Safety training should evaluate partially observed input and partially generated output, because both remain incomplete while interaction unfolds, including when new evidence requires cancellation or correction. More broadly, objectives for content accuracy, acoustic quality, latency, timing, reasoning, and safety may generate competing gradients. Projected action spaces, factorized rewards, and semantic-quality rewards reduce part of this interference [242,243,248], but current systems rarely report objectives that jointly cover continuous multimodal input, partial output, and asynchronous tool results.

5. Maintaining Omni Intelligence

This section examines the capabilities needed to maintain a coherent understanding as new observations, user corrections, and external results arrive during a session. A completed turn gives a model a stable block of context. A live interaction does not. New events continue to arrive, a user may correct an earlier statement, and retrieved evidence may return after the conversation has already moved forward. The model must therefore maintain a coherent representation of events, evidence, and participants as the interaction changes.

We organize this section around five capabilities. Temporal grounding tracks when events occur and how they relate. Memory and session state preserve events, entities, goals, and corrections across the interaction. Reasoning and tool use operate over partial input and continue while speech is being heard or played. Factual grounding ties a claim to the evidence available when that claim is made. Social and pragmatic understanding identifies speakers, intentions, emotional cues, and conversational roles. These capabilities define the intelligence side of native omni interaction. Section 3 and Section 4 discuss the supporting representations and architectures; Section 6, Section 7, Section 8 and Section 9 address interactive behavior, serving, evaluation, trade-offs, and open problems.

5.1. Temporal Grounding

Event time as an internal model state.

Temporal grounding is the ability to represent when an event occurs and how that event is related to the rest of the interaction [249,250,251,252]. The relevant state includes event boundaries, timestamps, duration, ordering, overlap, and relations such as before, during, and after. These quantities are separate from event identity. A model may recognize that a person picked up a cup, that a door closed, or that a user mentioned a meeting while assigning the observation to the wrong interval. In a completed video, this may appear as a localization error [249,251,253]. In a live session, it changes which evidence is valid for the current response.

Earlier video-language models explicitly predict event boundaries and timestamps. VTimeLLM [249] introduces boundary-aware training for video moments, TimeChat [250] introduces a timestamp-aware frame encoder and instruction tuning for time-sensitive long-video understanding, and Grounded-VideoLLM [251] represents temporal locations with dedicated time tokens. In real-time interaction, temporal grounding must update using only observations available so far, rather than relying on a completed clip: at time t, the model can use events observed up to t and cannot assume that the remainder of the stream is known.

Temporal relations across modalities.

An omni model also needs temporal relations across modalities [254]. A spoken reference can overlap a pointing gesture. A sound may begin before its visual source becomes clear. A tool result may arrive several seconds after the request that initiated it. FineMoLA provides a complementary weakly supervised mechanism for learning fine-grained frame–phrase temporal correspondence from clip-level descriptions [255]. The model must retain each observation’s time interval and link observations that refer to the same event. This gives temporal grounding three levels of structure: localization within a stream, relations across streams, and whether an event is past, ongoing, or not yet observed.

The third level is the main change from turn-based modeling. Turn-based input preserves sequence order, but the model receives the sequence after the turn is complete. A native model must know whether a relevant event has already happened, is still unfolding, or has not yet become observable. This distinction is needed even before a response policy is applied. It determines whether the current state contains enough information to support a reference, a reasoning step, a tool argument, or a factual claim.

Systems with explicit temporal alignment.

Several recent systems expose this requirement directly. SyncLLM inserts synchronization tokens at fixed temporal intervals so that language-model updates advance with real-world time [48]. MiniCPM-o 4.5 uses Omni-Flow to align continuous multimodal input and output along a shared temporal axis [256]. Aligned timestamps allow audio and visual observations to update a shared interaction state during speech generation. ROMA introduces a lightweight speak head that predicts response initiation from streaming multimodal state [200]. The speak decision belongs to interaction control, but its input must encode whether the event that justifies a response has occurred and whether the relevant evidence is complete enough to act on.

Figure 9 illustrates a time-indexed event representation with event boundaries, cross-modal links, and the current observation time. Temporal grounding identifies when each observation occurred and whether it is relevant to the current response. Section 3.3 discusses how timestamps, silence, and overlap enter the model representation. Section 7.1 defines timing and temporal measurements. Section 8 analyzes premature and delayed commitment. The role of temporal grounding here is narrower: it is the model’s internal understanding of when the evidence belongs.

5.2. Memory and Session State

A transcript alone may not capture recurring entities, changing goals, and corrections in a multimodal session: people and objects recur, events create consequences, user goals change, and later corrections can replace earlier assumptions. The model must track these changes together with the source and time of the supporting observations. This requirement is especially important for continuous audio and video, where an observation may be a short sound, a gesture, a visible object state, or a relation between two modalities.

Three semantic layers.

We describe session state with three semantic layers. The event state stores temporally grounded observations. Each event records what occurred, when it occurred, its modality and source, and any confidence or cross-modal links needed by later reasoning [249,251]. The entity state connects observations that refer to the same person, object, location, document, or external resource. It preserves identity across time and updates attributes or relations when new evidence arrives [58]. The versioned task state records the active goal, constraints, unresolved steps, corrections, tool results, and commitments. A correction creates a new task version and identifies which earlier conclusions depend on the superseded state [257,258].

These layers answer different questions. Event state supports references to what happened at a particular time. Entity state supports continuity across repeated observations. Versioned task state supports revision. The distinction matters because the same new input can have different effects on each layer. A user correction may leave the identity of an object unchanged, add a new event, and replace one task constraint. Treating the entire session as an append-only text history hides this structure.

Working, episodic, and long-term memory.

The semantic layers operate over several stores that differ in retention duration. Working memory keeps the recent multimodal context required for local references and immediate reasoning [259]. Episodic memory preserves events and task transitions across the current session [260]. Long-term memory stores selected user facts, preferences, entities, and recurring goals across sessions [58,261]. The retention duration describes how long an item remains available; the semantic layer describes what the item means. A person’s identity may be retained across sessions, while a temporary tool result may be relevant only to the current task.

Each stored item should retain its source, modality, timestamp, and relevant task version. Otherwise, a retrieved item may be semantically relevant and still refer to another user, an earlier time, or an outdated version of the task. Table 6 summarizes the three semantic layers and the update operations they require.

Queryable state over multimodal streams.

Recent systems implement parts of this memory design. ReKV stores video key-value caches outside GPU memory and retrieves relevant segments when a question arrives [262]. Its contribution is primarily an efficient streaming mechanism, which belongs to the system discussion in Section 6, but it also shows why long visual history must remain queryable by temporal segment. InteractiveOmni extends multi-turn interaction to audio and vision, requiring earlier multimodal observations to remain usable later in the dialogue [263]. EgoMem associates information with users using face and voice cues, maintaining facts, preferences, and social relations across sessions in real-time omnimodal streams [58].

From retrievable history to evolving latent state.

Streaming memory need not remain an external store that is repeatedly retrieved into the context window. FlexMem uses visual key–value caches as memory, applying two compression pathways together with task-dependent retrieval for long and streaming video without additional training [264]. LatentStream instead progressively consolidates historical observations into a compact, fixed-length latent memory for subsequent reasoning [265]. Taken together, these approaches store history as key–value caches, externally retrieved records, or compressed latent representations. Some also retain semantic or personalized information across sessions. StreamArena’s StreamMind system illustrates the same separation at the system level, keeping latency-critical interaction distinct from persistent multimodal recall over hour-scale video [189]. Streaming interaction requires memory to remain bounded and revisable while new observations continue to arrive.

Session memory should support retrieval and revision while preserving source information. Section 6, Section 8.2, and Section 9.2 discuss storage, consistency failures, and long-session memory, respectively. The event, entity, and versioned task layers defined here provide the shared vocabulary for those later discussions.

5.3. Reasoning and Tool Use under Real-Time Constraints

Reasoning and tool use share a constraint that does not appear in the same form for a completed prompt. Computation can start while the user is still speaking, continue while the model’s audio is being played, and return after the user has changed the task. Intermediate results are therefore tied to the input prefix and task version that produced them. The model needs to reason causally over available evidence and preserve that association when reasoning invokes retrieval or another external tool.

Reasoning while listening.

An internal text stream can support language processing while speech generation continues. Moshi maintains an inner-monologue text stream alongside the speech streams [21]. Qwen3-Omni adopts a Thinker–Talker architecture in which the Thinker processes multimodal inputs and provides semantic states, while the Talker generates streaming speech [23]. Their architectural details belong to Section 4. Semantic computation can retain its own state during speech generation. Cognitive Supersensing illustrates visual latent reasoning in an offline setting [266].

Other methods reason directly during ongoing input or speech. FLAIR updates a latent reasoning state while the user speaks, preserving causal processing [167]. STITCH interleaves internal reasoning with speech playback, using playback time to prepare later reasoning without inserting a full reasoning delay before speech [57]. Can Speech LLMs Think while Listening studies when a speech model should begin reasoning before an utterance is complete, using a question-completeness signal to control the accuracy-latency trade-off [166]. These methods differ in whether reasoning is explicit text or latent state and whether it runs during listening or speaking. Intermediate reasoning must remain consistent with the input observed so far.

Reasoning can also run asynchronously with perception. Video Streaming Thinking processes incoming clips while playback continues, rather than waiting for perception to terminate before deliberation begins [267]. R3-Streaming allocates additional reasoning to difficult queries after memory compression and response-readiness assessment, through a cascaded remember–respond–reason control process [268]. These designs allocate computation adaptively rather than forcing every input through the same reasoning budget, and they connect directly to the multi-rate view developed in Section 9.1.

Reasoning should retain uncertainty when the user’s request is incomplete, and revise affected conclusions when constraints change, without treating all earlier observations as invalid. This property connects reasoning directly to the versioned task state in Section 5.2. Such reasoning concerns the model’s current understanding; deciding when to speak is a separate interaction-control problem addressed in Section 6.

Tool use before the request is complete.

External tools extend the same causal computation. Stream RAG predicts retrieval queries during ongoing user speech and integrates returned evidence into a later response [27]. Related work on multi-turn agentic scientific literature search uses workflow induction to organize external search across turns [269]. MoshiRAG performs retrieval asynchronously between response onset and the delivery of substantive information [26]. DuplexOmni pairs a low-latency interaction layer with a separate reasoning component that can provide reasoning and tool use in parallel [25]. These systems demonstrate that retrieval and external computation can overlap the spoken interaction.

The intelligence requirement concerns grounding and validity. A tool request should record its arguments, supporting evidence, task version, and launch time. Returned results must be checked against the current task before use. Query prediction and asynchronous execution can reduce idle time, but the returned information is useful only if it still answers the current task. Section 3.5 covers how reasoning, tool, and retrieval streams are represented. Section 8.2 analyzes runtime errors when reasoning or external results become inconsistent with the live state. Section 9.1 treats cancellation, rollback, and multi-rate computation as open design questions.

Figure 10 follows a session in which a correction updates the task version. Events update event and entity state, background results retain their originating version and are checked against the current state before the assistant uses them in a response.

5.4. Factual Grounding and Correctness

A model may begin responding before all evidence needed for an answer is available. It must distinguish direct observations, retrieved information, memory, and inference, and identify which claims each source supports at the current time.

Evidence supporting a response.

Before presenting a factual claim, the model should identify its supporting evidence and verify that it remains relevant to the current task [270,271]. Visual evidence can be indexed by object [272], region [273], track [274], or event interval [251]. Acoustic evidence can be indexed by speaker [232], sound source [275], prosodic segment [276], or time range [235]. Retrieved evidence carries the query, source, return time, and task version. Memory-based evidence points back to the event, entity, or earlier task state from which it was recovered [271,277]. This structure allows the model to answer two questions before presenting substantive content: what supports the claim, and whether that support is valid for the current task state.

The strength of evidence can change while the model is already interacting. A partial request may support an acknowledgment before enough information is available for a factual answer. A visible event may resolve a reference after the user’s speech has begun, and a retrieval result may arrive after an initial spoken phrase. The system can keep an answer provisional and communicate its uncertainty until supporting evidence arrives.

Perceptual and external grounding.

Streaming retrieval provides a clear example of external grounding. Stream RAG and MoshiRAG allow later response content to depend on evidence retrieved while speech continues [26,27]. Qwen3-Omni provides a complementary perceptual example through its dedicated audio captioning model, which is fine-tuned for detailed audio descriptions with reduced hallucination [23]. The sources differ, but in both cases the model should track the evidence supporting each claim and check its relevance to the current task.

This subsection defines the positive grounding capability. Section 7 measures content quality and the timing of useful evidence. Section 8.1 examines preservation of pretrained language capabilities, including the degradation reported by URO-Bench and analyses of the text-speech understanding gap [28,29]; Section 8.2 examines premature claims, loss of grounding, and outdated evidence. Table 7 links these capabilities to evaluation and failure analysis.

5.5. Social and Pragmatic Understanding

A live assistant receives social information that is reduced or absent in a text transcript. Prosody can indicate uncertainty, urgency, affect, irony, or conversational stance. Face, gaze, body orientation, and speaker location help determine who produced an utterance and who is being addressed. The lexical form of a request, the speaker’s role, and the preceding interaction also affect register, politeness, and intent. Social and pragmatic understanding concerns interpreting these cues to inform response decisions.

Paralinguistic state.

Speech models that preserve prosody provide a basis for this competence. SPIRIT-LM interleaves speech and text within a shared token stream, while its expressive variant augments speech units with pitch and style information [147]. USDM uses paralinguistic speech information to support more natural spoken responses [145]. Prosodic cues inform estimates of speaker emotion, confidence, stance, and intent. They can change the interpretation of the same words and can influence later responses or memory updates.

Speaker and addressee.

Multi-party interaction requires separate estimates of speaker identity and addressee. A person can be audible without addressing the assistant. A speaker can refer to another participant while looking at an object. Addressee recognition therefore combines linguistic context with gaze, orientation, and conversational history. Earlier multimodal work showed that gaze and utterance content can be combined for addressee recognition [278]. More recent work shows that the problem remains challenging in multimodal multi-party dialogue: on a triadic-dialogue benchmark, GPT-4o achieves addressee-recognition accuracy only marginally above chance [279].

Visual cues help distinguish speakers in multiparty settings. AV-Dialog combines audio and visual cues to track the target speaker in noisy multi-speaker dialogue [274]. The model can then track who is speaking, who is being addressed, and which visible entities are relevant to the current utterance. The later decision to take the floor, backchannel, or remain silent is an interaction behavior and is evaluated in Section 7.

Role, register, and pragmatic context.

Social interpretation also considers participants’ roles and relationships. The same instruction can carry different authority depending on who speaks, and the same words can convey a request, correction, joke, or refusal depending on prosody and context. Role-conditioned systems such as PersonaPlex demonstrate that duplex behavior can be explicitly conditioned on textual role prompts and speech-based voice prompts [280]. Explicit role prompts differ from inferring roles during an unconstrained interaction, where the relevant role may need to be inferred from multimodal context [278,279].

This subsection focuses on interpreting social cues. Section 7 evaluates response behavior and timing, while Section 8.3 and Section 9.2 discuss privacy, personalization, and multiparty coordination.

5.6. Synthesis: Interdependent Capabilities

The capabilities above operate on shared state. Temporal grounding supplies the event structure used by memory and online reasoning. Memory keeps the entities, corrections, and task versions that later reasoning must recover. Reasoning and tools add inferred and external evidence. Factual grounding determines which parts of that evidence support a claim at the current time. Social understanding assigns observations and requests to the correct participants and supplies role and intent information to the task state.

Continuous interaction requires these capabilities to share a consistent state as new input arrives. A completed prompt can be processed after its evidence and intent are fixed, whereas a native omni model must keep time, state, inference, evidence, and participant attribution consistent while the session continues to change. Section 7 and Section 8 examine their evaluation and failure modes.

6. Realizing Omni Interactivity

Omni interactivity concerns response timing and behavior during an ongoing session. We examine response latency, turn-taking, interruption recovery, proactive assistance, adaptation across turns, and the serving and client mechanisms that support them.

These behaviors build on the representations of time, silence, overlap, and backchannels described in Section 3.3, together with the temporal grounding, session state, reasoning, and tool validity examined in Section 5. They depend on both the interaction policy and its execution across the model, server, and client. Section 7 covers their metrics and benchmarks, Section 8 analyzes the intelligence–interactivity trade-off, and Section 9 develops the open research agenda.

Figure 11 summarizes the resulting interaction loop. Continuous multimodal input updates the shared state, and the interaction policy selects silence, a backchannel, substantive speech, a proactive intervention, or a control action. Playback and user feedback inform the next decision. The model, server, and client must agree on input timing, playback progress, and which outputs have reached the user.

6.1. Latency and Responsiveness

End-to-end latency measures the delay from a relevant input event to output reaching the user [141,281]. It includes capture and chunking, audio or visual encoding, inference, speech synthesis, network transport, buffering, and playback delays. Each stage contributes to the total latency, and a stall at any stage delays the observable response.

Cascaded systems serialize several stages [36,141]. Voice activity detection closes the input segment, automatic speech recognition produces text, the language model generates a response, and text-to-speech converts that response into audio. Streaming components reduce individual delays, although module boundaries still introduce buffering and scheduling gaps. Native omni designs can overlap capture, encoding, prefill, decoding, and synthesis [21,22,43]. Section 4 examines their architectural details, while the latency budget records the resulting critical path.

Interruption latency also matters. An interruption must propagate through the client audio controller, server scheduler, generation state, and queued playback. Fast response onset combined with slow stopping still produces a rigid interaction. Systems should therefore specify timing requirements for both starting and stopping output, including input chunking, response onset, per-token generation, playback continuity, and stop propagation. Section 7 defines the corresponding measurements and reporting protocol.

Smaller audio chunks and more frequent video sampling can improve responsiveness but increase processing cost [107,282]. Larger playback buffers tolerate network variation but delay stopping. Larger batches improve throughput but can increase queueing delay. These dependencies determine the deadlines that the serving stack in Section 6.6 must protect. Response onset and the arrival of task-relevant information can occur at different times, an issue examined as part of the intelligence–interactivity trade-off in Section 8. The system should indicate whether it is waiting, acknowledging a request, seeking clarification, or providing an answer. A brief acknowledgment can precede a complete answer, with its delivery timed using turn-taking predictions. In a deployed dialogue robot, recent work prepares a context-aware prefatory response once user intent becomes predictable, while a separate Voice Activity Projection process determines when to deliver it [283]. This separates content readiness from floor-taking readiness.

6.2. Turn-Taking, Silence, and Backchannels

Turn-taking converts the time-aware representations described in Section 3.3 into an output policy. At each update, the model chooses among continued listening, silence, a backchannel, floor-taking, and yielding. Silence can allow the user to continue, indicate uncertainty, or delay a response until an event occurs. Backchannels acknowledge the user, while substantive speech begins the model’s turn. Yielding releases the floor.

A policy can use acoustic, linguistic, visual, and task-state cues to select an action and its timing [16,33,278,284]. Acoustic cues provide prosody, pause structure, overlap, and speech onset. Linguistic context indicates whether an utterance is complete, while visual context provides gaze, gesture, addressee, and ongoing activity. Task state indicates whether the model has enough information to proceed. Section 5 examines how these signals are interpreted and maintained in the interaction state.

Human conversation constrains how these cues should be mapped to actions. Long gaps can feel unresponsive, premature floor-taking cuts off the user, and excessive backchannels become distracting. A pause may indicate hesitation or turn completion, so the appropriate response depends on linguistic, visual, and social context. Background speech may require silence even when the acoustic channel is active.

Current systems place turn-taking control in different components. Speech-native full-duplex models such as dGSLM and Moshi learn timing through parallel conversational streams [15,21]. Freeze-Omni uses a chunk-level state predictor to control listening and response transitions [43]. SALMONN-omni represents speaking decisions with learned state tokens during generation [186]. BayLing-Duplex learns listen, speak, and stop control within a single autoregressive speech–language model using a small set of control tokens [187]. Analyses of interacting Moshi instances identify internal activations associated with upcoming turn changes [285]. A deployed system can combine learned timing predictions with explicit execution rules shared by the client and scheduler.

Section 3 describes the supporting representations, and Section 7 evaluates response timing, takeover rate, silence, and backchannel quality. The present subsection specifies the action policy that these representations support and the system executes.

6.3. Interruption and Repair

Interruptions can change the user’s request while speech or background work continues. The system must stop pending output, update the context and task, cancel outdated work where possible, and resume consistently.

Table 8 separates this process into linked stages. Detection classifies incoming activity during model speech. When an interruption requires the system to yield, it returns the floor and stops pending playback. Rollback then aligns the model context with content delivered to the user, while cancellation stops retrieval, reasoning, and tools associated with a superseded task version. Revision applies the correction to the versioned state defined in Section 5, allowing resumption to proceed from the corrected interaction state.

To support rollback and consistent resumption, the conversational record should distinguish generated, queued, played, and canceled content, using playback progress to track what has been delivered to the user. ChipChat implements this by tracking vocalized tokens and rolling the context back to the point reached by playback [181]. Training can also shape the conversational behavior surrounding these transitions. Multi-Faceted Interactivity Alignment uses separate rewards for pause handling, turn-taking, backchannels, and interruption, together with a response-quality reward [243]. By combining interaction and content objectives, it connects repair behavior to the alignment objectives discussed in Section 4.3.

Repair must also extend to tool operations, which may already have changed external state when an interruption arrives. Recovery therefore needs to account for what was executed and which results remain valid. The versioned task and evidence states in Section 5 help identify outdated work, while recovery may require cancellation, confirmation, or compensation. These requirements also inform the evaluation in Section 7, which examines whether repair succeeds and how much obsolete content is delivered before recovery.

Coordinating this recovery requires information from the client. Echo cancellation and near-end speech detection help identify incoming user speech, while audio ducking reduces playback and cancellation stops pending output. Playback timestamps then help the server distinguish content already delivered to the user from content that should be removed from the shared conversational record. Section 6.6 places these functions within the complete serving and client path.

6.4. Proactive Interaction

Proactive interaction occurs when the system initiates a contribution without a new explicit request. A visual event, acoustic cue, elapsed interval, tool result, or change in task state can trigger this decision [113,116,286,287,288]. Temporal grounding and event understanding examined in Section 5 determine what happened and when, while the proactive policy determines whether an intervention is appropriate at the current moment. The policy should consider task relevance, urgency, confidence, distraction, privacy, social context, and whether an action can be reversed, together with user permissions and preferences [286,288,289]. A safety warning may justify immediate speech, while a low-confidence suggestion can wait for a pause or appear as a quiet visual cue [290].

Streaming video systems provide early implementations of this policy. VideoLLM-online predicts a response or silence at each frame, MM-Duet allows either participant to contribute during video playback, and systems with separate perception, decision, and reaction components continue observing while selecting a response [113,114,115]. StreamBridge adds proactive activation to models trained for offline video understanding [116]. Proact-VL trains multimodal language models for continuous video perception, autonomous response triggering, and latency-aware commentary generation. Its Live Gaming Benchmark evaluates response timing and content selection in solo commentary, co-commentary, and user guidance [288]. These systems repeatedly decide whether to speak or continue observing as new input arrives, using model predictions to guide response timing.

When background results arrive, the model decides whether to use them immediately, wait, notify the user nonverbally, or discard them. It considers whether the result still matches the task and whether the user is currently speaking. Foreground–background architectures provide the concurrent computation, while the proactive policy controls how its results are presented [26,27,60].

Section 7 evaluates proactive response timing, while Section 8 and Section 9 discuss decisions based on incomplete evidence, persistent interaction, privacy, and safety.

6.5. Adaptation Across Turns

Within-session adaptation changes response behavior based on user feedback. Subsequent turns should reflect preferences about response length, speaking pace, acknowledgment frequency, interruption tolerance, and initiative [238,291]. A corrected answer demonstrates repair, while a sustained change in behavior demonstrates adaptation. The interaction layer can represent these preferences through policy variables such as response length, speech rate, pause duration, backchannel rate, confirmation threshold, and proactivity threshold [238,291]. Each preference update should specify where it applies and when it expires. A task-specific correction can expire with the task, while a session preference can persist until the conversation ends. Section 5 examines the memory and provenance needed to store these updates, and Section 9 considers persistent personalization across sessions.

Interaction-policy adaptation.

Appropriate turn-taking depends on the task and participant roles. A tutor may interrupt proactively, whereas a counseling assistant may need to avoid interruption. Instruct-FD evaluates whether full-duplex systems can follow instructions about when to listen, backchannel, interrupt, or continue speaking, and reports a substantial gap between fluid and controllable full-duplex behavior [291]. DuplexGen shows that preferences over turn-taking vary across scenarios and can be learned from a small amount of preference supervision [238]. Together, these studies motivate interaction policies controlled by instructions and user preferences, with timing patterns learned from conversational data serving as an initial policy.

Preferences can be stored explicitly or inferred from recent interaction history. Explicit state supports inspection, correction, and deletion [292], while context-conditioned control infers preferences from recent feedback [293]. Both approaches can adapt behavior during a session without changing model weights. Section 4.3 examines offline alignment for timing, turn-taking, backchannels, and interruption.

Preference updates should rely on repeated or explicit feedback. A single interruption may correct content, signal urgency, or result from background noise, so it does not necessarily indicate a lasting preference [293,294]. Repeated interruptions during long answers provide stronger evidence for shorter responses, and rejected proactive suggestions support a higher intervention threshold in the same context [293,294]. Repeated corrections about an entity should increase confidence in the updated interpretation [292,294].

Section 7 evaluates adaptation by comparing behavior before and after feedback under matched conditions. Within-session adaptation changes response timing, style, initiative, or confirmation behavior, while Section 9 considers learning, personalization, and consistency across sessions.

6.6. Full-Duplex Serving and Client Coordination

The behaviors above depend on a serving stack for long-lived, bidirectional sessions. Serving systems must coordinate incoming media, playback, state updates, and cancellation [25,43,60]. Audio and video arrive in frequent small chunks, speech decoding may continue as new input arrives, and an interruption can invalidate part of the current context. Tools and background models also return results on independent schedules [25,26,27].

Persistent streaming sessions provide the basic execution unit. Per-session encoder state, KV cache, interaction state, and playback state remain available across chunks and workers. Freeze-Omni separates user-specific caches from model execution so that a stateless worker pool can process active sessions [43]. Thinking Machines Lab appends 200 ms micro-turns to a persistent GPU-resident sequence and optimizes MoE kernels for the small input shapes produced by bidirectional serving [60]. Section 5 defines the semantic content of session state, while this subsection covers its storage, scheduling, and mutation.

Scheduling must meet speech-generation deadlines while processing new input. Incoming chunks produce frequent prefills that compete with ongoing decoding. Chunked prefill divides new context into bounded units, and continuous batching schedules work at iteration granularity [39,295,296]. Separating prefill and decoding across resources can reduce interference between them [297,298]. Playback progress and user activity can also help interaction-aware schedulers prioritize sessions whose audio buffers are nearly empty and prepare for barge-in [281].

Cache management must limit memory use while retaining timestamps, task versions, corrections, and playback or output-status records needed for recovery. Paged allocation reduces fragmentation, while sliding windows, attention sinks, selective eviction, and retrieval-backed caches limit active memory [38,262,299,300]. Semantic relevance alone may not identify every state required for repair.

MoE backbones introduce additional latency at the small batch sizes common in interactive decoding. Expert-weight movement, routing, and cross-device communication can dominate each decoding step [301,302,303,304,305,306]. Expert locality, prefetching, phase-specific parallelism, and kernels optimized for small inputs can reduce this cost [60,305,306]. The system should keep decoding time within its latency target under realistic concurrent workloads.

Time-critical functions can run on the client or edge device, while larger models, retrieval, and tools run in the cloud. Edge functions can include wake-word detection [307], VAD [185], echo cancellation [308], audio compression [77,78], floor control [16,33], and interruption detection [43,185]. UAF consolidates several of these functions by predicting VAD, turn detection, speaker recognition, ASR, question answering, and interruption-related control signals as a streaming autoregressive sequence [185]. Network degradation may require reduced media rates, lower codec bitrates, delayed background work, or simpler local behavior [309,310,311,312].

The client completes the control loop by capturing media, performing echo cancellation, reducing playback volume during an interruption, maintaining a bounded jitter buffer, and sending playback timestamps to the server. It also shows whether the system is listening, processing, speaking, muted, or waiting for confirmation. These signals support turn coordination and user control. Table 9 maps the main system mechanisms to the behavioral failures they prevent.

Omni interactivity requires timely responses, appropriate turn-taking, coordinated recovery, useful proactive assistance, adaptation to feedback, and serving mechanisms that support these behaviors. Section 7 measures these capabilities, Section 8 examines their relation to intelligence, and Section 9 identifies the remaining research problems.

7. Evaluating Intelligence and Interactivity

Real-time evaluation must assess response content together with timing, interruption handling, correction recovery, and consistency across turns. We review evaluation metrics in Section 7.1, existing benchmarks and their coverage in Section 7.2, and limitations of the available protocols in Section 7.3.

7.1. Evaluation Metrics

We organize evaluation into six dimensions: latency, timing quality, content quality, robustness, long-session quality, and repair. Figure 12 summarizes their representative metrics and evaluation settings.

7.1.1. Latency

Latency captures several distinct delays, including the time to begin generating a response, deliver audible speech, and react to an interruption. Each measure depends on its starting event: first-token and first-audio latency may be measured from user-turn completion or from an annotated point when sufficient input has arrived. The following measures distinguish these stages of the interaction:

• First-token latency: elapsed time from declared input event to emission of the first text or semantic token.
• First-audio latency: elapsed time from the declared input event to the first audible output at the client, including synthesis, buffering, and playback delays.
• Interruption-stop latency: elapsed time from the onset of an interrupting user utterance to cessation of model speech at the client. State whether the start event is annotated speech onset or system detection of the interruption.
• Response-after-interruption latency: elapsed time from the interruption event to the start of the subsequent response, using the start and end events specified by the evaluation protocol. Full-Duplex-Bench v1 (FDB-v1) reports this as Latency After Interruption [137], which measures response resumption rather than how quickly the model stops its ongoing speech.
• Turn-taking latency: the gap between the end of the user’s utterance and the onset of the model’s response. FDB-v1 measures this as response latency, “the time (in seconds) between the end of the user’s speech and the start of the model’s response,” computed only on turns where the model actually takes over the floor [137].

Observed latency depends on both the architecture and the conditions under which it runs. In cascaded VAD–ASR–LLM–TTS systems, delays accumulate across components and their buffers, with additional delays from network transmission and client playback. Interpreting reported values therefore requires information about the hardware, workload, number of concurrent sessions, and buffering configuration. FDB-v3 reports wall-clock latency for multi-step tool-use tasks [175], while SimulEval provides latency measures for simultaneous translation [313].

7.1.2. Timing Quality

Timing quality evaluates whether responses and backchannels occur at appropriate moments. TurnGPT predicts turn completion from text [127], while Voice Activity Projection (VAP) uses self-supervised learning to predict future speech activity from audio [16]. Follow-up work evaluates the effects of prosody [314] and fillers or pauses [130]. Skantze surveys turn-taking in spoken dialogue systems [33]. MuVAP extends prediction to multimodal, multiparty settings [284].

FDB-v1 uses Takeover Rate (TOR), the fraction of probes in which the model begins a speaking turn, across pause handling, backchanneling, turn-taking, and interruption [137]. Its preferred direction depends on the scenario. A low takeover rate is desirable when a pause does not yield the user’s turn; a high rate is desirable when the user yields and expects a response. Interruption results must identify who interrupts whom and the intended response behavior rather than assigning one preferred direction to all interruptions. For proactive video responses, ProactiveVideoQA evaluates response timing as well as content using PAUC [118].

7.1.3. Content Quality and Instruction Following

Content quality evaluates correctness, completeness, and instruction following. OmniBench measures static multimodal understanding and does not evaluate streaming interaction or response timing [315]. VoiceBench evaluates instruction following under acoustic and speaker variation [316]. Dynamic-SUPERB and its Phase-2 extension cover classification, regression, and sequence-generation tasks [317,318]. Audio MultiChallenge evaluates instruction retention and recall of audio cues that are absent from transcripts [319]. SpokenWOZ reports task completion and dialogue-state tracking performance in spoken task-oriented dialogue [320].

7.1.4. Interaction Robustness

Robustness evaluates performance under overlapping speech, noise, interruptions, and side conversations. Overlap should be evaluated as a normal conversational condition as well as a source of interference; dGSLM models overlapping speech, laughter, and backchannels in two-channel dialogue [15]. FDB-v1.5 evaluates four overlap scenarios: user interruptions, user backchannels, speech directed to another person, and background speech. It measures response behavior, timing, and prosodic adaptation [138]. FDB-v1 also evaluates interruption handling [137]. VoiceBench varies speakers, acoustic conditions, and content [316]. IRAF addresses interfering speech by adjusting the contribution of incoming audio features; it is a method addressing a robustness failure, rather than a benchmark [321].

7.1.5. Long-Session Quality

Long-session evaluation covers consistency, entity and event tracking, task state, and memory over extended interaction. FDB-v2 uses an automated examiner with staged goals and two pacing conditions, Fast and Slow. Its tasks cover daily interaction, correction, entity tracking, and safety, and its results identify difficulties with overlapping speech, user corrections, and entity tracking [139]. Audio MultiChallenge adapts inference-memory and self-coherence evaluations to spoken multi-turn dialogue [319]. M3-DuplexBench evaluates multilingual dialogue with varying amounts of prior context [322]. SpokenWOZ measures task-state maintenance across spoken turns [320], while StreamingBench tests retention of earlier streamed information [120]. Multi-turn evaluation and continuous hours-long interaction should be distinguished when reporting session length. User simulation offers a complementary approach to long-session evaluation: a simulated user interacts with the system, and performance is assessed from the resulting interaction history [323].

7.1.6. Repair Behavior

Repair evaluation tests whether the model incorporates a correction and stops obsolete output or work. Audio MultiChallenge evaluates mid-utterance and previous-turn edits [319], while FDB-v2 tests correction in streaming multi-turn dialogue [139]. We propose reporting correction-recovery rate, the fraction of correction trials whose final response satisfies the revised request, and stale-content rate, the fraction that continue using the superseded request after a specified recovery allowance. Reports should define the correction event, scoring rubric, recovery allowance, and trial denominator.

7.2. Comparative Benchmark Coverage

We group benchmarks by their primary focus on Omni Intelligence or Omni Interactivity, while comparing both groups using the six dimensions in Section 7.1. These groups describe a benchmark’s primary purpose; individual benchmarks may assess capabilities from both groups.

Figure 13 presents the historical development of the benchmark families. The intelligence branch includes multi-turn dialogue evaluation, static omni-modal and spoken understanding, and streaming audiovisual and temporal understanding. The interactivity branch includes full-duplex spoken interaction, streaming and proactive omni interaction, and reusable tools and measures for evaluating turn-taking. Within each family, resources are ordered by first public release year.

Table 10 compares coverage across evaluation dimensions. Filled symbols indicate direct coverage through a dedicated task or separately reported metric. Partial symbols indicate limited or indirect coverage: a subset of tasks, or a capability affecting an aggregate score without a separate measurement. Empty symbols indicate absent coverage. These marks characterize benchmark

scope rather than model performance.

Public reproducibility is treated separately from capability coverage. Benchmarks with released data, code, a hosted evaluator, or a public leaderboard support independent comparison. Publicly documented benchmarks whose release status cannot yet be independently confirmed are marked with ‡. Vendor and internal evaluations are marked with † and are not co-ranked with public benchmarks.

7.2.1. Omni Intelligence Benchmarks

These benchmarks evaluate multimodal understanding, reasoning, event timing, and retention across turns. We distinguish three benchmark families.

Multi-turn dialogue understanding and consistency.

This family evaluates whether model capabilities persist when relevant information and instructions are distributed across multiple dialogue turns. MultiChallenge measures instruction retention, inference memory, self-coherence, and related failures in text-based dialogue [324]. Audio MultiChallenge transfers these evaluation targets to end-to-end speech and introduces audio-specific challenges involving ambient and paralinguistic cues, as well as voice editing and spoken repair [319]. SpokenWOZ evaluates task-oriented spoken dialogue, including dialogue-state tracking, cross-turn slots, reasoning slots, and task completion [320]. ChildAgentEval compares agent capabilities with age-specific developmental measures without imposing real-time interaction constraints [329]. These benchmarks evaluate content quality and retention across turns, primarily under turn-based protocols without explicit real-time deadlines.

Static multimodal and spoken understanding.

This family measures semantic understanding, reasoning, instruction following, and robustness over static or turn-complete input. OmniBench evaluates integrated image, audio, and text reasoning [315,330]. VoiceBench evaluates knowledge, instruction following, safety, and robustness to variation in speaker, acoustic environment, and spoken content [316]. Dynamic-SUPERB and Dynamic-SUPERB Phase-2 provide broad instruction-conditioned evaluation across speech, music, and environmental sounds, covering classification, regression, and sequence-generation tasks [317,318]. These benchmarks assess baseline capabilities that should be preserved when adding real-time interaction.

Streaming audiovisual and temporal intelligence.

This family evaluates whether a model can understand events as a multimodal stream unfolds and associate observations with their temporal positions. StreamingBench evaluates real-time, omni-source, and contextual understanding at multiple points in a streaming video [120]. Charades-STA provides language-conditioned temporal localization, while Charades supplies temporally annotated activity understanding [325,326]. RepCount and TransRAC provide fine-grained evaluation of repetitive-action counting and temporal structure [327]. Charades-STA, Charades, and RepCount assess event timing and repetitive-action counting. These static datasets assess what happened and when, but do not measure autonomous response timing.

7.2.2. Omni Interactivity Benchmarks

The omni interactivity branch evaluates when and how a model acts during an interaction. It focuses on latency, turn coordination, interruption handling, proactive behavior, and recovery from interaction failures.

Full-duplex spoken interaction.

The Full-Duplex-Bench series evaluates simultaneous listening and speaking. FDB-v1 evaluates pause handling, backchanneling, turn taking, and interruption using controlled behavioral probes [137]. FDB-v1.5 extends the setting to overlapping speech, including user interruptions, listener backchannels, side conversations, and ambient speech [138]. FDB-v2 introduces an automated examiner, Fast and Slow pacing conditions, and multi-turn task families covering daily interaction, correction, entity tracking, and safety [139]. FDB-v3 adds multi-step tool use with real human audio and reports task accuracy, wall-clock latency, and turn-taking behavior [175]. HumDial-FDBench further contributes real dual-channel human conversations containing interruption, overlap, and feedback events [210]. M3-DuplexBench evaluates English and Japanese conversation and question answering with different amounts of dialogue history [322]. Instruct-FD evaluates whether systems follow explicit instructions about listening, acknowledging, interrupting, and continuing speech [291]. Together, these resources provide the strongest existing coverage of latency, interaction timing, and overlap robustness, although their modality coverage remains primarily speech-based.

Streaming and proactive omni interaction.

This family evaluates systems that operate over continuous audiovisual input and must decide whether and when to respond. OmniMMI combines streaming audiovisual understanding with proactive reasoning across multiple interaction tasks [328]. ProactiveVideoQA isolates the response-initiation decision and evaluates whether a model answers at an appropriate point during video playback [118]. Proact-VL evaluates proactive commentary over continuous game video, jointly scoring response timing and content across solo commentary, co-commentary, and user guidance [288]. StreamArena evaluates extended streaming interaction, including current perception, recall of earlier observations, proactive responses, and tool use [189]. OmniInteract combines online audiovisual input with measures of correctness, timing, interruption handling, invalid outputs, and context retention [119]. These benchmarks connect omni intelligence with interaction timing, but do not yet provide a complete standard for bidirectional spoken overlap, semantic repair, tool use, and long-session operation under a shared protocol.

Turn-taking prediction and evaluation.

VAP, TurnGPT, and MuVAP provide reusable tasks and measures for assessing conversational timing within broader evaluations. VAP predicts future voice activity from two-channel audio and supports evaluation of turn shifts and backchannels [16]. TurnGPT predicts turn completion from textual dialogue context [127]. MuVAP extends future-activity projection to multimodal, multiparty settings using audio and visual speaker information [284]. These tasks and measures can be incorporated into broader omni-interaction evaluations.

Vendor and internal evaluations.

TimeSpeak and CueSpeak are reported as internal Thinking Machines evaluations [60]. TimeSpeak examines time-triggered speech initiation, while CueSpeak evaluates responses triggered by semantic cues during ongoing speech. These vendor-reported evaluations are presented separately because the survey does not identify public datasets and protocols supporting independent reproduction.

7.3. Limitations of Existing Benchmarks

Current benchmarks cover different aspects of content quality, timing, and interaction behavior, but evaluating these capabilities together remains a challenge. The following proposals concern joint evaluation under common episodes and resource budgets, including where existing benchmarks provide partial coverage:

7.3.1. Joint Intelligence–Interactivity Evaluation

Existing static benchmarks can measure whether an answer is correct, while full-duplex benchmarks can measure whether a model yields, interrupts, or takes the floor. Recent benchmarks such as OmniMMI, ProactiveVideoQA, and OmniInteract begin to combine streaming understanding with online response decisions [118,119,328], but a joint protocol should evaluate continuous audio–video reasoning, bidirectional spoken overlap, semantic correction, and end-to-end serving latency using the same episodes and resource budget. One way to narrow the gap between offline replay and closed-loop evaluation is to augment recorded episodes with synthetic observations of plausible future states [331]. User simulation could also support closed-loop evaluation by generating user actions in response to the model’s outputs, allowing assessment of how the model adapts to changes in user goals and knowledge [332]. A future protocol should evaluate the same system in at least four conditions: offline content evaluation, causal streaming perception, proactive response generation, and full-duplex interruption. Results should be reported as a Pareto frontier.

Proposed measurements: Evaluate content quality and interactive behavior together under the same episodes and resource budgets, reporting event-level timing and computational cost alongside capability scores. These measurements show how response quality changes under real-time constraints. Report accuracy at specified deadlines and define the input event and output endpoint for response onset and useful-answer time. A useful answer should satisfy a task-specific rubric; report failures to produce one separately. Define accuracy after interruption using the revised request, and replay timestamped episodes without access to future input.

7.3.2. Long-session Continuous Audio and Video

StreamingBench and OmniMMI evaluate bounded video streams and timed probes [120,328]; FDB-v2 and Audio MultiChallenge evaluate multiple turns [139,319]. These protocols are substantially shorter and more controlled than persistent meetings, tutoring sessions, assistive agents, embodied monitoring, and long-running task execution. StreamArena begins to probe hour-scale agentic streaming interaction, but remains video-centric rather than bidirectionally spoken [189]. Long-session evaluation must separate recent-scene perception from long-range memory, cap computation and memory growth, and ensure that the model remains an active participant throughout the session rather than answering only at the end.

Candidate metrics: entity and event retrieval as a function of elapsed time; contradiction rate; task-state recovery after long gaps; accuracy at fixed memory and compute budgets; latency drift over session duration. State the memory budget, compute limit, and session duration for each result.

7.3.3. Multi-party and Addressee-aware Full-duplex Interaction

Most end-to-end interaction benchmarks remain dyadic. MuVAP evaluates multiparty next-speaker prediction but is a predictive primitive rather than an interactive-agent benchmark [284]. HumDial-FDBench provides realistic full-duplex speech but does not establish a broad evaluation suite for three-or-more-speaker coordination [210].

Proposed measurements: Evaluate three or more participants with annotated speaker identity, addressee, and expected responder. Report addressee accuracy, responses to bystanders, speaker-conditioned interruption rates, and recovery from attribution errors. Measure response allocation as each participant’s response rate among eligible addressed turns, accounting for differences in opportunities to respond.

7.3.4. Multilingual and Code-switching Full Duplex

Public full-duplex evaluation remains dominated by English, with limited coverage of other languages and little standardized evaluation of code-switching, low-resource languages, and language-dependent timing. M3-DuplexBench begins to address this by evaluating multi-turn full-duplex dialogue in English and Japanese [322], but broader language coverage and code-switching remain open. Static accent or multilingual understanding does not establish multilingual turn-taking competence.

Proposed measurements: Compare content accuracy and response timing by language. Score code-switch recognition against annotated language-switch events, and score recovery by whether the system follows the revised language instruction within a stated interval. Specify backchannel appropriateness in each language.

7.3.5. Repair, Cancellation, and Tool Use During Continuous Input

Audio MultiChallenge evaluates voice editing, FDB-v2 evaluates correction, and FDB-v3 evaluates disfluent multi-step tool use [139,175,319]. What remains missing is their combination: a user revises the request while the model is speaking, a background tool or retrieval process is already running, and the system must cancel, update, or safely reuse partial work without emitting stale results.

Candidate metrics: correction-recovery rate and stale-content rate as defined above; cancellation success among cancellable operations; unnecessary tool execution and its cost; and elapsed time from the correction to the revised action. State the denominator and recovery deadline for each measure.

7.3.6. Controllable Interaction Policy

Existing benchmarks largely ask whether interaction is natural. They rarely ask whether it is appropriate for a specified role, or whether it can be changed on instruction. Instruct-FD evaluates instruction-conditioned turn management and finds a large gap between fluid and controllable behavior [291], while DuplexGen shows that the target turn-taking policy is itself scenario dependent [238].

Proposed measurements: instruction adherence for listening, acknowledging, interrupting, and continuing speech. Specify when the instruction changes and measure the delay until compliant behavior begins, using a stated observation interval and scenario-specific scoring rubric.

7.3.7. Safety, Privacy, and Deployment Robustness Under Continuous Input

Turn-based safety evaluation does not cover bystander speech, visible private information, environmental leakage, cross-channel prompt injection, packet loss, delayed frames, or safety actions that must proactively interrupt the user. A system may also appear safe only because network delay prevents it from responding in time.

Proposed measurements: define privacy leakage as unauthorized disclosure of specified private information captured in audio or video. Identify prohibited actions, required interventions, and intervention deadlines, then report missed and false interventions, disclosure rates, and errors under overlapping speech, controlled packet loss, and frame delay.

8. The Intelligence–Interactivity Trade-Off

A central question is whether real-time speech and multimodal capabilities can be added while preserving the reasoning, factual knowledge, grounding, and tool use learned during language pretraining. Evaluations examine how speech and interaction objectives affect these capabilities [28,29,333].

The trade-off also concerns reliable use of those capabilities under real-time constraints and changing evidence. Reasoning, retrieval, grounding, and tool execution may remain incomplete after speech begins, while users may interrupt or revise the task [25,26,57]. Figure 14 distinguishes capability preservation during adaptation from response, evidence, and state coordination during runtime inference.

8.1. Capability Preservation under Real-Time Omni Adaptation

Evaluations identify capability losses in some models after speech adaptation. URO-Bench reports that several end-to-end spoken dialogue models lag behind their underlying language-model backbones in instruction following and exhibit catastrophic forgetting [28]. Related analysis attributes the text–speech understanding gap to two mechanisms: forgetting of textual capabilities during speech adaptation and misalignment between speech and text representations [29]. Parameter-importance analysis suggests that speech fine-tuning can alter parameters associated with textual reasoning in different layers [333]. The effect depends on training data, optimization, modality interfaces, and whether the language backbone is updated. These findings do not establish that capability loss is inevitable.

A recurring architectural response is to separate interactive speech generation from the representations and computation used for language reasoning. Moshi retains an intermediate linguistic stream within its speech–text foundation model, keeping semantic processing explicit during real-time dialogue [21]. Freeze-Omni freezes the pretrained LLM backbone and attaches separate speech input and output modules [43]. Qwen3-Omni extends this separation through a Thinker–Talker architecture, in which multimodal understanding and reasoning are separated from streaming speech generation [23]. JoyAI-Talker combines a Thinker–Talker architecture with unified speech–text joint training that is designed to mitigate degradation in language understanding and reasoning performance while retaining performance on textual reasoning, STEM tasks, and logic, adding state-driven duplex control and expressive paralinguistic generation [334]. BayLing-Duplex adds listen, speak, and stop control tokens to a speech-language model and reports retained response quality [187].

These designs aim to preserve pretrained reasoning and factual knowledge during speech adaptation. Reported results are consistent with this principle, but they do not yet isolate the effect of individual design choices. Freeze-Omni reports spoken-input performance close to its frozen backbone’s text-input performance, while Qwen3-Omni reports performance comparable to same-sized single-modal models across text, image, audio, and video tasks [23,43]. In contrast, broader analyses of speech-adapted LLMs identify textual forgetting, cross-modal misalignment, and disruption of text-critical parameters [29,333]. Because these studies use different backbones, training data, model scales, and evaluation settings, controlled comparisons are still needed to determine when backbone freezing, constrained adaptation, or explicit reasoning–speech separation best preserves the capabilities of the original language model.

Interaction-policy optimization can also reduce response quality unless its objectives explicitly preserve semantic content. Multi-Faceted Interactivity Alignment uses separate rewards for pause handling, turn-taking, backchanneling, and user interruption, together with a response-quality reward [243]. Capability losses can therefore occur both when adding speech during adaptation and when optimizing interaction timing without preserving response quality.

8.2. Runtime Trade-Offs

8.2.1. Response Timing, Reasoning, and Grounding

A model can start responding before reasoning, retrieval, or tool execution is complete. We distinguish response-onset latency from useful-answer time, when task-relevant information first reaches the user, using the reporting requirements in Section 7.1. Systems that overlap speech with reasoning, retrieval, or tool execution illustrate why these endpoints differ [25,26,27,57].

Early speech is not necessarily a failure. Brief acknowledgments and backchannels can preserve conversational flow while the system continues to reason, and full-duplex benchmarks distinguish these behaviors from interruptions, side conversations, and ambient speech [138]. The problem arises when early output implies that the answer is already grounded, states unsupported content, or commits to an interpretation that the user later corrects. Full-Duplex-Bench-v2 directly evaluates correction and entity-tracking tasks and reports persistent difficulties under overlapping multi-turn interaction [139]. In tool-augmented settings, a query issued before the user’s request is sufficiently clear may retrieve irrelevant information; recent work formalizes this issue as tool-intent stabilization [27,179]. Systems should distinguish early acknowledgments from factual answers and validate later claims or tool results against the current task.

8.2.2. Evidence and State Coordination

Observations and returned results differ in source, timing, reliability, and relevance. These include user and background speech, video, partial transcripts, retrieval outputs, tool results, and memory. This creates three recurring risks: source confusion, in which incidental input is treated as user-directed evidence; temporal misalignment, in which information is attached to the wrong moment; and stale evidence, in which a result associated with an earlier task state is used after the user’s intent has changed. Static multimodal evaluation shows that irrelevant modalities can distort model decisions, while full-duplex and streaming benchmarks expose additional difficulties in overlap handling and online visual understanding [120,138,335]. Recent omni systems accordingly treat temporal alignment and synchronized multimodal processing as central architectural requirements [200,256]. Reliable interaction therefore requires evidence to remain associated with its source, time, and current task relevance.

Longer sessions require consistent tracking of people, objects, events, goals, corrections, tool calls, and superseded tasks. Long-context and memory benchmarks show that relevant information may be overlooked and that sustained interaction requires temporal reasoning, knowledge updates, and consistency across sessions [261,336,337]. Recent spoken and audio-visual benchmarks explicitly extend evaluation to correction handling, entity tracking, and multimodal multi-turn memory [139,263]. Memory should retain each observation’s source, modality, and time, support consistent entity identification, and allow task updates. Retrieved evidence must be checked for relevance to the current task version [261,262]. Together, these runtime risks show that interactivity depends on coordinating when evidence becomes valid and which version of the interaction it belongs to. Table 11 summarizes these recurring forms of the intelligence–interactivity trade-off.

8.3. Safety, Privacy, and Evaluation Implications

Continuous interaction requires rules for capturing, processing, transmitting, and storing information that may not be addressed to the system, including bystander speech and private visual content. Audio and audio–visual safety benchmarks show that unsafe intent can be expressed through speech and distributed across modality combinations, while selective-hearing evaluation reveals substantial leakage of incidental bystander speech [338,339,340,341]. Permission checks must apply to each information-handling operation and to any action it triggers. These boundaries may require different decisions for the same input: sensitive bystander speech may need to be detected locally for redaction without being transmitted to an external model or written into persistent memory. Privacy risk also extends to the model’s internal state. Analysis of full-duplex systems such as SALM-Duplex and Moshi finds that speaker identity can become easier to infer from hidden states as a conversation progresses. Streaming anonymization can reduce this exposure while adding delay or reducing task performance [289]. Privacy in continuous interaction therefore concerns not only stored media but also what persistent internal representations reveal about the user.

Continuous sensing can improve contextual understanding, but it also requires the system to track where information comes from, whether it remains relevant, and how it may be used, including permission checks. Evaluation therefore needs to assess both whether observations and inferences are used for authorized purposes and whether the system intervenes at appropriate times. These requirements for handling information and actions must also be evaluated.

Existing benchmarks remain fragmented: full-duplex evaluations focus on turn-taking and spoken repair, streaming benchmarks evaluate online perception, long-term memory benchmarks test retrieval and consistency across extended histories, and multimodal safety benchmarks assess jailbreak, refusal, or privacy risks [120,138,139,261,337,338,339]. First-token latency cannot reveal when useful information becomes available; final-answer accuracy cannot reveal whether stale or misattributed evidence was used; and final refusal cannot reveal whether sensitive information was already transmitted or stored. Evaluate response timing, reasoning quality, evidence validity, state consistency, privacy exposure, and tool or memory actions together. Event-level records can show their trade-offs more clearly than a single aggregate score.

9. Design Principles and Open Problems

Native real-time omni models must coordinate perception, reasoning, generation, and action as new information arrives. Remaining challenges concern maintaining these capabilities jointly within latency and resource limits [25,119,256,342]. We organize the research agenda around computation at different time scales; persistent, proactive, and social interaction; and safe, efficient operation under changing deployment conditions. Figure 15 summarizes these directions.

9.1. Multi-Rate Interactive Intelligence

Immediate interaction and extended reasoning have different timing requirements. Turn-taking, interruption handling, and short acknowledgments require prompt responses, whereas retrieval, verification, planning, and tool execution may take longer. A system can coordinate multiple models or internal streams, each with different compute, context, and output constraints [25,26,27,167,342].

Recent streaming systems suggest that this coordination is not a single architectural pattern but a family of mechanisms. Asynchronous parallelism runs foreground interaction and background computation concurrently, as in StreamMind, which separates latency-critical interaction from persistent multimodal recall and longer-horizon processing [189]. Conditional computation invokes deeper reasoning only when it is needed: R3-Streaming compresses memory, assesses response readiness, and routes only difficult queries toward stronger reasoning [268]. Perception–reasoning overlap runs reasoning while new evidence arrives, as in Video Streaming Thinking, which processes incoming clips during video playback [267]. Multi-rate state evolution consolidates memory on a slower timescale than immediate perception, as in LatentStream’s progressive latent memory [265]. Decoupled interaction timing separates content preparation from floor-taking, as in contextual preface generation paired with a Voice Activity Projection onset decision [283]. These mechanisms must share a consistent interaction state even when their inputs come from different task versions.

Background work may begin from an incomplete request and finish after the task changes. Associating each operation with its originating task version allows returned results to be checked against the current request, outdated work to be canceled where possible, and reusable intermediate results to be retained. Speculative retrieval and tool use can reduce delay when the relevant intent stabilizes sufficiently early [178,179]. Recovery also depends on what the operation has done: computation can often be discarded or repeated, whereas an external action that cannot be undone may require a compensating action. Coordinating this background work with ongoing interaction creates additional scheduling demands. Frequent input updates, persistent state, asynchronous results, and cancellation must be handled alongside decoding. Disaggregated execution [298], latency-aware serving [39], and interaction-aware scheduling [281] provide mechanisms for allocating computation. When omni systems generate visual output, component-level acceleration can complement scheduling: rectified-flow auto-coding and optimized flow inference reduce generation cost [343,344], although integrating these methods with real-time interaction scheduling remains open. Efficient scheduling must also account for the current task: even a quickly computed result may no longer be useful if it arrives after it is needed, relies on outdated evidence, or has been invalidated by a correction.

9.2. Persistent, Proactive, and Social Interaction

Persistent memory should distinguish current and superseded information, retain unresolved tasks, and record the evidence supporting stored conclusions. Streaming attention, compression, sparse prefill, and cache retrieval reduce computational growth [262,299,345,346]. Memory must also separate observations from inferences, preserve source and time, and support bounded storage, revision, and deletion [261,337].

Proactive responses should use current observations, task requirements, and user instructions. The system must decide whether an event warrants a response, whether the user is available, and whether waiting would be more helpful. Recent work has begun to model proactive response from streaming sensory context and to evaluate both response content and timing [118,119,256,286]. Proact-VL treats response triggering as part of the interactive policy over an ongoing video stream and evaluates timing and content jointly [288], while Instruct-FD and DuplexGen indicate that the appropriate proactive and turn-taking policy is role- and scenario-dependent and may need to be set by instruction rather than learned as a single fixed prior [238,291]. When intent is uncertain, the model should retain alternative interpretations and defer consequential actions until sufficient evidence is available. Evaluation should distinguish helpful interventions, missed opportunities, unnecessary interruptions, and appropriate decisions to wait.

These decisions become harder in multi-party and personalized settings. Speaker identity, addressee, conversational floor, and authority are related but distinct. A person may be speaking near the system without addressing it, and the person addressing the system may not be authorized to trigger an action. Existing full-duplex data and turn-taking models only begin to capture overlapping speakers, repair, and floor competition [210,284]. Personalization should distinguish temporary behavior from lasting preferences and allow users to inspect, correct, or delete stored information. Systems also need explicit boundaries between shared and private participant context. Systems such as Ex-Omni-2D generate synchronized visual output alongside personalized speech [188], introducing additional questions about attention and social timing. The central challenge is to maintain one coherent interaction while preserving the distinct roles, permissions, and histories of its participants.

9.3. Safe and Efficient Operation in Open-World Settings

Continuous perception requires safety checks throughout the interaction, including for incoming audio and video. Audio and video may contain bystander speech, private screens, or environmental details that are available to the model but not permissible to use. Systems must enforce permissions for capture, processing, transmission, storage, and action. Selective-hearing results show that attending to the intended speaker does not by itself prevent leakage from surrounding speech [339]. Analogous leakage arises from internal state: speaker identity can remain recoverable from the hidden states of full-duplex models and grow more linkable across the first conversational turns, and streaming anonymization trades latency and utility to reduce it [289]. A specific open problem is preserving useful context without retaining unnecessary biometric, identity, or other sensitive attributes. Permission checks also apply to visual evidence, retrieved records, and tool outputs. Safety must also govern action timing. A provisional interpretation may be sufficient to start local computation, but not to send a message, disclose information, or control a device. SafeDreamer uses a learned world model to predict rewards and safety costs for candidate action sequences and incorporates cost constraints into action selection [347]. Greater uncertainty or more consequential actions may require additional confirmation or oversight, including monitoring for high-impact operations [348].

Reliable systems need explicit fallback behavior when network, compute, sensors, or tools degrade. Real-time behavior depends on device memory, network delay, server load, sensor quality, and the availability of external tools. When these conditions deteriorate, the system may need to reduce visual sampling, postpone background reasoning, route computation between edge and cloud, or fall back to a more limited interaction mode. Edge-capable omni models and interaction-aware serving show that such choices can be incorporated into system design [256,281], but fallback behavior should communicate missing information or delays and preserve interaction state. Dropping a modality should lower confidence where that evidence matters, delaying a tool should not be hidden behind an unsupported answer, and a connectivity failure should not erase the interaction state. The open problem is to maintain causal input processing, accurate uncertainty reporting, interruption handling, and action safeguards when full capability is temporarily unavailable.

Training data and evaluations should include these failure conditions. Natural interaction data should include overlap, hesitation, correction, cancellation, silence, and cases in which a proactive response or tool call should be withheld [209,210,319]. Evaluations should replay timestamped streams without access to future input and compare systems under controlled resource and network conditions. They should record when intent becomes clear, when useful information is delivered, which evidence is used, and how tool and memory state change. The protocol should define how intent clarity and useful information are annotated. Recent benchmarks increasingly cover disfluent tool use and online audio-visual interaction [119,175], but they do not yet combine long-session state, multi-party authority, privacy boundaries, and deployment stress in one setting. The resulting systems should be compared on a joint frontier of answer quality, informative latency, repair, cost, and safety rather than reduced to a single aggregate score.

9.4. Outlook

Both integrated and modular architectures can support computation at different time scales. Beyond these architectures, an emerging direction is to pair generative omni models with compact, JEV-like decision modules for streaming and latency-critical control. PlayJev provides an early example by mapping raw visual observations and candidate actions directly to real-time action probabilities, although extending this approach from GUI games to open-ended multimodal interaction remains an open problem [349].

The key requirements are consistent task versions, revisable intermediate reasoning, memory with traceable sources, appropriate decisions to wait, and actions that remain valid after interruptions. Progress should be measured by preserved reasoning and grounding during continuous input, consistent state across sessions and participants, and adaptive computation with accurate uncertainty reporting and privacy protection. Evaluate usefulness, time to a useful answer, recovery from errors, resource cost, and safe action together. The goal is a system that remains useful and correct as evidence and user intent change during its response.

10. Conclusions

Real-time interaction changes the conditions under which multimodal intelligence must operate. A model must reason from incomplete input, decide when to respond, and remain receptive to evidence that may change its interpretation or invalidate an ongoing action. This survey examined how these demands connect multimodal representations, model architectures, training objectives, persistent state, and serving systems. Together, they motivate Interactive Omni Intelligence: the ability to sustain understanding, reasoning, and generation as the world and the conversation continue to unfold. Our conceptual framework comprises two coupled components: Omni Intelligence and Omni Interactivity. The Intelligence–Interactivity Trade-off characterizes how the demands of timely, continuous interaction constrain the multimodal capabilities a system can effectively deploy. Together, the framework, design analysis, and evaluation agenda provide a roadmap for native real-time Interactive Omni Intelligence.