Variational Interpretable Framework for Multimodal Instruction Execution

1. Introduction

The capability of artificial agents to comprehend and execute natural language instructions in complex environments has remained a fundamental challenge in AI, gaining increasing relevance with advancements in robotics and embodied intelligence [4]. This task, often referred to as instruction following, demands an intricate fusion of linguistic comprehension and action-oriented planning. While recent developments in imitation learning (IL) and reinforcement learning (RL) have enabled agents to perform such tasks in visually and spatially grounded simulations, such as BabyAI [5] and Room-to-Room (R2R) [2], the high dependency on large-scale paired datasets continues to restrict scalability.

In particular, neural models that map linguistic inputs to navigation actions require tens or hundreds of thousands of annotated instruction-trajectory pairs, even in synthetic gridworlds [5]. This requirement is even more pronounced in real-world scenarios, where data collection is costlier and more error-prone. Thus, enabling agents to learn effectively from partially labeled or unlabeled data becomes crucial to improving data efficiency.

To address this issue, semi-supervised learning approaches have been introduced. One well-known framework is the speaker-follower model [7], which generates synthetic instructions for unpaired trajectories via a trained "speaker" model. The synthetic pairs are then used to train a "follower" policy, reducing the dependence on labeled data. Subsequent refinements of this approach, such as data augmentation [22] and confidence-based filtering [26], have shown promise. Nonetheless, these techniques are inherently limited, as they rely on a speaker trained only from the scarce paired data, which may not generalize well to diverse unseen instructions or environments.

To overcome these limitations, we propose a fundamentally different strategy rooted in probabilistic generative modeling. Our proposed solution, IntraMIX, reinterprets instruction following as a semi-supervised multimodal generation problem. Specifically, we extend the framework of multimodal variational autoencoders (M-VAEs) [25] to sequential settings, thereby enabling the learning of shared representations that capture the underlying semantics of both language and trajectory sequences.

Unlike conventional M-VAEs which are designed for static data modalities (e.g., images and captions), IntraMIX adopts a temporal generative formulation suitable for sequence-to-sequence tasks. A critical innovation is our introduction of a bottleneck attention mechanism, which aligns the variable-length sequences in both modalities by projecting them into a common latent space with fixed-length temporal representations. This structure facilitates compatibility with the attention mechanism [18], widely employed in prior work for modeling language-grounded planning [2,5].

Beyond pure generative modeling, IntraMIX is designed to be flexibly integrated with the speaker-follower architecture. Leveraging the bidirectionality of our generative formulation, we repurpose IntraMIX as both a trajectory generator (follower) and an instruction generator (speaker). This dual functionality allows for a closed-loop learning paradigm, in which unpaired data from either modality can be reconstructed, translated, or leveraged for regularization. To prevent overfitting and distributional mismatch when training with unpaired data, we introduce a novel regularization term in the loss function that minimizes the divergence between the latent distributions of paired and unpaired inputs. This cross-modal regularization ensures that unpaired trajectories and instructions are aligned in the latent space, promoting robustness and generalization across modalities.

We conduct extensive experiments in BabyAI and R2R environments to evaluate the efficacy of IntraMIX. The results demonstrate clear improvements over both purely supervised baselines and previous semi-supervised models. When used as a standalone follower model, IntraMIX outperforms traditional IL-trained agents by margins of 5.1% on BabyAI and 3.7% on R2R in task completion rate. When integrated with the speaker-follower pipeline, it further enhances instruction-following accuracy by an additional 2.5% to 4.9%. These results validate the dual utility of IntraMIX: as a generative learner for semi-supervised settings and as an enhancement module for existing pipelines.

-

We propose IntraMIX, a novel generative framework for instruction-following agents that unifies multimodal sequence modeling with attention-aware representation learning.

-

Our architecture introduces a bottleneck attention mechanism to handle variable-length sequence alignment, which is crucial for language-to-trajectory tasks.

-

We demonstrate that IntraMIX can be seamlessly integrated into the speaker-follower paradigm, enhancing both components through latent alignment and a novel regularization objective.

-

Extensive empirical results on two established benchmarks show that our model achieves competitive performance and outperforms prior semi-supervised approaches under limited supervision.

2. Related Work

Instruction-following tasks under semi-supervised settings have garnered extensive research attention, particularly with the advent of neural agents capable of interpreting natural language commands and executing them in grounded environments. A central paradigm in this domain is the speaker-follower model, which forms the foundational framework for leveraging unpaired trajectory data to mitigate the scarcity of annotated pairs. Pioneering works such as [8,12,22,26] introduced various refinements and extensions to this architecture, enabling more robust utilization of synthetic or unpaired trajectories. Notably, [26] and [8] presented sophisticated strategies for collecting unpaired data with improved semantic fidelity and task relevance, thereby enhancing the downstream policy learning. Complementarily, [12] introduced a discriminator-based selection mechanism to filter out noisy or semantically inconsistent synthetic instructions generated by speakers. Such mechanisms effectively reduce the propagation of erroneous signals during training, particularly in large-scale or multi-instruction scenarios.

Beyond supervised imitation learning, variants of the speaker-follower model have also been adapted to reinforcement learning (RL) settings. For instance, [6] proposed a data augmentation scheme aligned with the spirit of the speaker-follower model, wherein high-reward trajectories are reused with synthesized instructions to improve sample efficiency. However, a fundamental limitation persists across these models: the speaker component is often trained exclusively on the limited set of paired data, constraining its generalizability and rendering it brittle in out-of-distribution or long-horizon tasks.

In contrast, our proposed approach, IntraMIX, departs from the conventional reliance on deterministic back-translation and instead adopts a probabilistic generative modeling framework to unlock richer cross-modal interactions. While IntraMIX is architecturally distinct, it is inherently complementary to the speaker-follower framework. By enabling the learning of a shared latent representation space through semi-supervised training, IntraMIX directly augments the capabilities of both the speaker and follower components. This synergy permits mutual reinforcement between modalities, yielding more accurate synthetic data and more robust policies.

The conceptual underpinning of IntraMIX also draws connections to unsupervised machine translation literature, particularly the works of [3] and [16], which proposed effective frameworks for exploiting unaligned bilingual corpora. These methods typically employ two core strategies: (i) back-translation—generating a sentence in one language from its counterpart in another—and (ii) shared latent reconstruction—learning a joint representation that enables reconstruction of either modality. While the former is conceptually similar to the speaker-follower approach, the latter is more aligned with the generative mechanism adopted in IntraMIX. Specifically, the shared latent variable $\mathcal{V}$ in our formulation (see Eq. 4) corresponds to the cross-modal embedding used to jointly represent and reconstruct both language and trajectory data. However, unlike prior works that use static embeddings, IntraMIX introduces temporal structure via its attention-enabled sequence encoder, thus enhancing contextual coherence across modalities.

The novelty of IntraMIX also lies in its architectural innovations. For example, while [17] proposed a bottleneck-style module for extracting salient subsets from unordered sets, IntraMIX reinterprets this idea within a sequential attention context. Rather than filtering input features, our bottleneck attention serves as a cross-modality mediator, harmonizing variable-length linguistic and behavioral sequences into a temporally structured latent space. This not only facilitates sequence-to-sequence transformations but also aligns with the attention paradigms popularized by transformer models in vision-language navigation [11].

Moreover, our framework relates to earlier work on variational sequence modeling. Kočisky et al. [14], for instance, proposed a latent-variable model for structured prediction tasks, employing recurrent networks to generate latent sequences. While effective, this method introduces additional autoregressive components that limit parallelizability and are susceptible to vanishing gradients. IntraMIX circumvents these issues by relying exclusively on attention-based modules for latent variable inference, enabling parallel computation while maintaining expressive power and stable training dynamics.

Additionally, recent advances in large-scale pretrained language models such as BERT [28] have inspired applications of transfer learning to instruction following. These include leveraging textual embeddings for trajectory grounding [11] or using pretrained transformers as instruction encoders. Despite their success, these methods often overlook the bidirectional generative potential between language and action, which lies at the core of IntraMIX’s architecture. Furthermore, many data augmentation strategies used in prior studies, including environment dropout [22] and reinforcement-driven auxiliary losses [24], can be viewed as orthogonal improvements that are complementary to the generative backbone introduced in our work.

In summary, IntraMIX is positioned at the intersection of several research streams, including generative modeling, semi-supervised learning, instruction-grounded navigation, and sequence-to-sequence representation learning. Its capacity to unify these perspectives under a coherent probabilistic framework distinguishes it from prior approaches and opens up new avenues for multimodal agent training in low-resource or partially annotated environments.

3. Preliminary

3.1. Task Definition and Semi-Supervised Setup

We consider the task of visually grounded instruction following, in which an agent is required to interpret a natural language instruction and generate a corresponding action sequence that fulfills the task objectives. Let the instruction be represented as a variable-length sequence $y_i=[y_{i,1},y_{i,2},\cdots ,y_{i,L_i}]$, where $L_i$ denotes the length of the instruction in episode i. The instruction typically conveys goal-directed behaviors, such as “Turn right at the hallway and go to the red sofa”.

At each discrete timestep t, the agent receives an observation $o_{i,t}$ and produces an action $a_{i,t}$ based on the accumulated sequence of past observations and the language instruction. Formally, the policy function is defined as:

$$\pi :(y_i,o_{i,1:t})\mapsto a_{i,t},\forall t\in [1,T_i],$$

where $T_i$ is the trajectory length in episode i.

We adopt a semi-supervised learning paradigm, following prior frameworks such as [7], where both labeled and unlabeled data are leveraged for training. The labeled (paired) dataset is defined as:

$$D_p={\left\{({\tau }_i,y_i)\right\}}_{i=1}^M,$$

where ${\tau }_i=(o_i,a_i)=([o_{i,1},\cdots ,o_{i,T_i}],[a_{i,1},\cdots ,a_{i,T_i}])$ is the interaction trajectory comprising both environment observations and the agent’s actions. The unlabeled (unpaired) dataset consists only of trajectory data:

$$D_u={\left\{{\tau }_j\right\}}_{j=1}^N,\mathrm{where}{\tau }_j=(o_j,a_j),$$

with M and N representing the number of paired and unpaired samples, respectively.

Throughout our formulation, we assume that the instruction lengths $L_i$ and trajectory lengths $T_i$ vary across episodes. For notational convenience, we drop the subscript i when the context is unambiguous. This setup presents a realistic and challenging scenario where annotated instructions are sparse, motivating the need for semi-supervised or generative learning mechanisms like IntraMIX to bridge the supervision gap.

3.2. Canonical Sequence-Based Follower Architecture

To establish a foundational comparison, we briefly revisit the standard attention-based sequence-to-sequence (seq2seq) follower architecture, which serves as the backbone for many instruction-following agents, including those used in environments like Room-to-Room (R2R) [2] and BabyAI [5]. Our proposed IntraMIX model adopts and extends this architectural baseline with significant improvements in generative inference and multimodal alignment.

The follower architecture consists of three primary components: an instruction encoder $f_{\mathrm{enc}}$, a trajectory decoder $f_{\mathrm{dec}}$, and an attention-based alignment module $f_{\mathrm{att}}$ that bridges them. Concretely, given an instruction $y=[y_1,\cdots ,y_L]$, the encoder maps it to a sequence of contextual hidden representations:

$$[h_1,h_2,\cdots ,h_L]=f_{\mathrm{enc}}\left(y\right),$$

typically using an LSTM or Transformer encoder.

During interaction, at each timestep t, the decoder produces a hidden state based on prior actions and observations:

$$h_t^{\prime }=f_{\mathrm{dec}}(a_{1:t-1},o_{1:t}),$$

which summarizes the agent’s partial trajectory and accumulated environment knowledge.

An attention mechanism then computes a context vector by aligning the decoder’s current state with the instruction embeddings:

$$c_t=f_{\mathrm{att}}(h_t^{\prime },[h_1,\cdots ,h_L]),$$

where the attention output $c_t$ captures the most relevant parts of the instruction at timestep t.

Finally, the action at timestep t is predicted via an action predictor module $f_{\mathrm{act}}$, which integrates the decoder and context vectors:

$$a_t=f_{\mathrm{act}}(h_t^{\prime },c_t).$$

This architecture facilitates effective sequence grounding and supports long-horizon goal execution. However, it relies heavily on paired data for training and lacks mechanisms to incorporate unpaired trajectories or leverage latent semantic structures—limitations that our proposed IntraMIX model is designed to overcome.

3.3. Data Augmentation with Speaker-Follower Paradigm

The speaker-follower framework is a widely adopted strategy for semi-supervised instruction following. It introduces an auxiliary module—the speaker—which is responsible for generating natural language descriptions from observed trajectories. The core idea is to use this component to synthetically annotate unpaired trajectories, thereby expanding the effective training data for the follower.

The speaker is architecturally similar to the follower: it follows an attention-based seq2seq model, but operates in reverse. Given a trajectory $\tau =(o,a)$, it encodes the sequence into latent representations and autoregressively decodes them into a textual instruction $\widehat{y}=[{\widehat{y}}_1,\cdots ,{\widehat{y}}_L]$. More formally:

$${\widehat{y}}_t\sim p_{\mathrm{spk}}\left({\widehat{y}}_t\right|{\widehat{y}}_{1:t-1},\tau ),$$

where $p_{\mathrm{spk}}$ denotes the speaker’s conditional language model, trained using teacher-forced supervision on paired samples $(\tau ,y)$.

Once trained, the speaker is applied to each unpaired trajectory ${\tau }_j\in D_u$ to produce a pseudo-instruction ${\widehat{y}}_j$. This creates an auxiliary paired dataset:

$${\widehat{D}}_p={\left\{({\tau }_j,{\widehat{y}}_j)\right\}}_{j=1}^N,$$

which is then used to train or fine-tune the follower model using standard supervised objectives.

Despite its empirical success, this framework suffers from several inherent limitations. First, the speaker is trained only on the small paired dataset $D_p$, which limits its generalization capability in unseen environments or trajectory styles. Second, the generation process is unidirectional and deterministic, lacking uncertainty quantification or latent modeling of cross-modal semantics. Third, the synthetic instructions may exhibit semantic drift or compositional inconsistencies, introducing noise into the training process.

In this work, our proposed IntraMIX model provides a principled generative alternative to the speaker, capable of modeling uncertainty via latent variables and supporting reconstruction from either modality. Moreover, IntraMIX can act as both a speaker and follower simultaneously, with shared latent grounding. In Section 4.1, we detail how IntraMIX not only subsumes the speaker-follower paradigm but also improves its effectiveness through probabilistic cross-modal alignment and latent-space regularization.

4. IntraMIX: A Generative Probabilistic Perspective

4.1. Overview

In this section, we introduce the underlying probabilistic formulation of our proposed framework, IntraMIX, which stands for Interpretable Multimodal Instruction eXecutor. The core idea is to model instruction following as a multimodal generative task that captures the joint distribution of language instructions, trajectories, and environmental observations through a latent variable framework. Our approach builds upon the foundations of Multimodal Variational Autoencoders (M-VAE) [25], extending them to sequential decision-making settings with attention and autoregressive decoding.

We begin by defining the generative process over a latent representation z, a language instruction y, and an action sequence a given a sequence of environment observations o. The generative model is structured as:

$$p_{\theta }(z,y,a|o)=p_{\theta }\left(z\right)·p_{\theta }\left(y\right|z)·p_{\theta }\left(a\right|z,o),$$

(1)

where $p_{\theta }\left(z\right)$ denotes the prior over latent task embeddings, $p_{\theta }\left(y\right|z)$ represents the language generation process, and $p_{\theta }\left(a\right|z,o)$ models the trajectory conditioned on both the environment context and the latent semantics.

Analogous to standard M-VAE configurations, this factorization allows us to interpret the generation of paired modalities—language and action—from a shared semantic factor z. That is, given $x_{\left(1\right)}=a|o$ and $x_{\left(2\right)}=y$, this model can be viewed as a structured extension of the canonical multimodal generation process.

However, as the true posterior distribution $p_{\theta }\left(z\right|x_{\left(1\right)},x_{\left(2\right)})$ is intractable, we introduce a variational approximation:

$$q_{\varphi }\left(z\right|y,a,o)={\displaystyle \frac{1}{2}}{q}_{\varphi }\left(z\right|y)+{\displaystyle \frac{1}{2}}{q}_{\varphi }\left(z\right|a,o),$$

(2)

following the mixture-of-experts (MoE) strategy shown by [20] to be particularly effective for cross-modal latent alignment. This balanced fusion enforces a symmetrical contribution from each modality, helping to mitigate mode collapse and ensure mutual consistency.

To learn the model, we maximize the evidence lower bound (ELBO) of the marginal log-likelihood under both paired and unpaired settings. For paired data $D_p$, the ELBO becomes:

$$\begin{array}{c}\mathcal{J}={\displaystyle \frac{1}{2}}{\mathbb{E}}_{q_{\varphi }\left(z\right|y)}\left[log{\displaystyle \frac{p_{\theta }\left(z\right)p_{\theta }\left(y\right|z)p_{\theta }\left(a\right|z,o)}{q_{\varphi }\left(z\right|y,a,o)}}\right]\\ \hfill +{\displaystyle \frac{1}{2}}{\mathbb{E}}_{q_{\varphi }\left(z\right|a,o)}\left[log{\displaystyle \frac{p_{\theta }\left(z\right)p_{\theta }\left(y\right|z)p_{\theta }\left(a\right|z,o)}{q_{\varphi }\left(z\right|y,a,o)}}\right].\end{array}$$

(3)

In contrast, for unpaired trajectory-only data $D_u$, we derive a reduced ELBO over the marginal $p_{\theta }\left(a\right|o)$ as:

$$\mathcal{V}={\mathbb{E}}_{q_{\varphi }\left(z\right|a,o)}[log{p}_{\theta }\left(a\right|z,o)]+D_{\mathrm{KL}}[q_{\varphi }\left(z\right|a,o)\Vert p_{\theta }\left(z\right)].$$

(4)

This component allows the model to learn from unannotated sequences by reconstructing plausible actions based solely on contextual observations and inferred task embeddings.

Combining both settings, we formulate the overall training objective as a weighted combination:

$$max{\mathbb{E}}_{\{\tau ,y\}\in D_p}\left[\overline{\mathcal{J}}\right]+\gamma {\mathbb{E}}_{\tau \in D_u}\left[\mathcal{V}\right],$$

(5)

where $\gamma$ modulates the influence of unpaired supervision. This dual-objective framework enables IntraMIX to benefit from large quantities of unpaired data while maintaining alignment with high-quality paired annotations.

To operationalize action generation, we incorporate a timestep variable t and reformulate the trajectory decoder in an autoregressive manner:

$$p_{\theta }\left(a\right|z,o)=\prod _{t=1}^T{p}_{\theta }\left(a_t\right|z,o_{1:t},a_{1:t-1}).$$

(6)

This allows the agent to produce actions sequentially, in alignment with how agents interact in real-world environments.

Similarly, the language decoder is also defined as:

$$p_{\theta }\left(y\right|z)=\prod _{t=1}^L{p}_{\theta }\left(y_t\right|z,y_{1:t-1}).$$

(7)

The inference processes for follower and speaker usage are then defined respectively as:

$$a_t\sim p_{\theta }\left(a_t\right|z,o_{1:t},a_{1:t-1}),z\sim q_{\varphi }\left(z\right|y),$$

(8)

$$y_t\sim p_{\theta }\left(y_t\right|z,y_{1:t-1}),z\sim q_{\varphi }\left(z\right|a,o).$$

(9)

We refer to the two usage modes of IntraMIX as IntraMIX-Follower (direct rollout) and IntraMIX-SpeakerFollower (integration into the speaker-follower framework).

4.2. Cross-Modal Compression via Bottleneck Attention

To support alignment across modalities with potentially differing sequence lengths, we introduce a novel Bottleneck Attention Module, designed to extract fixed-length latent sequences from variable-length modality streams. Unlike naive approaches that produce sequence-level latent variables from modality-specific encoders and risk mismatch in temporal alignment, our mechanism projects variable-length hidden states into a common K-dimensional sequence via trainable token queries $e=[e_1,...,e_K]$.

Let $h=[h_1,...,h_L]$ denote the encoded hidden states (e.g., of a language instruction). The module performs multi-head attention with the bottleneck tokens e as queries, and h as both keys and values:

$$z=\mathrm{Attention}(e,h,h).$$

Each resulting $z_i$ is modeled as a Gaussian latent variable with its own mean and variance, parameterized via the attention output. These variables form the latent sequence used downstream:

$$q_{\varphi }\left(z\right|y)=\prod _{i=1}^K\mathcal{N}({\mu }_i,{\sigma }_i^2).$$

This structure ensures consistent dimensionality across modalities and improves compatibility with attention-based decoding mechanisms. Importantly, the bottleneck attention is modular and can be seamlessly integrated into any Transformer-style encoder or decoder without requiring architectural overhauls.

4.3. Domain Alignment with Latent Distribution Regularization

Despite careful design, domain mismatch between paired and unpaired data poses a risk to generalization. Specifically, encodings of trajectories from $D_p$ and $D_u$ may drift apart due to their involvement in different loss terms. To mitigate this, we introduce Domain Distance Regularization, a penalty that encourages alignment of the latent distributions over z induced by the two domains.

Let:

$$\rho ={\mathbb{E}}_{\tau \in D_p}\left[q_{\varphi }\left(z\right|\tau )\right],\upsilon ={\mathbb{E}}_{{\tau }^{\prime }\in D_u}\left[q_{\varphi }\left(z\right|{\tau }^{\prime })\right].$$

We define a regularization penalty $D(\rho ,\upsilon )$, which measures the divergence between these distributions. While any divergence metric can be used, we adopt the Sliced Wasserstein Distance (SWD) [15] for its efficiency and empirical robustness.

The final objective becomes:

$$max{\mathbb{E}}_{\{\tau ,y\}\in D_p}\left[\overline{\mathcal{J}}\right]+\gamma {\mathbb{E}}_{\tau \in D_u}\left[\mathcal{V}\right]-\alpha D(\rho ,\upsilon ),$$

(10)

with $\alpha$ as the domain alignment strength coefficient.

Through this combination of generative modeling, bottleneck attention, and domain alignment regularization, IntraMIX provides a principled, extensible foundation for instruction-following in multimodal semi-supervised environments.

5. Experiments

5.1. Benchmarks and Setup Overview

To systematically assess the effectiveness of our proposed model IntraMIX, we conducted experiments across two widely-used instruction-following environments: BabyAI [5] and Room-to-Room (R2R) [2]. These environments offer complementary characteristics—BabyAI focuses on discrete, symbolic reasoning in a gridworld, while R2R emphasizes grounded vision-language understanding in photorealistic 3D environments.

Within BabyAI, we evaluate our model across four tasks: GoToSeq, GoToSeqLocal, BossLevel, and BossLocal. The latter two (GoToSeqLocal and BossLocal) are introduced in our study as simplified but semantically aligned variants of their original counterparts, specifically tailored to evaluate generalization with limited input complexity. In GoTo-style tasks, the agent must reach a target object described in natural language, whereas Boss-level tasks require completing multiple sub-instructions, including object manipulation (e.g., “pick up the red key”).

Performance on BabyAI is primarily evaluated using the mean Success Rate (SR), which measures whether the agent successfully completes all subgoals in the instruction. Additionally, to evaluate the quality of the speaker (language generation from trajectories), we adopt the BLEU-4 metric, commonly used in text generation tasks.

In the R2R dataset, the agent is required to navigate a photorealistic environment based on natural language instructions describing the intended path. The dataset contains 7,189 human-annotated trajectory-instruction pairs, with an additional 178,000 unpaired trajectories provided for semi-supervised learning [7]. R2R uses three evaluation environments—validation seen, validation unseen, and test. Performance is measured using:

• Success Rate (SR): Whether the agent stops within 3 meters of the goal.
• Oracle Success Rate (OSR): Whether any position along the path is within 3 meters of the goal.
• Navigation Error (NE): The final distance from the target location (lower is better).

Further experimental configurations, including architectural specifics, optimization hyperparameters, and validation strategies, are detailed in Appendix D.

5.2. Ablation: Impact of Architectural Design

Evaluating Bottleneck Attention We first analyze the architectural contributions of IntraMIX by isolating the effects of attention mechanisms. As shown in Table 1, a conventional seq2seq model without attention underperforms significantly in complex environments such as BossLevel, highlighting its limitations in handling long-horizon or compositional instructions. By incorporating standard attention, performance improves drastically, particularly in semantically dense tasks.

Our bottleneck attention module achieves competitive performance, even surpassing attention baselines in GoToSeqLocal. Notably, with $K=16$, IntraMIX nearly matches the attention-based seq2seq model across all tasks, while offering the added benefit of latent interpretability and alignment. This suggests that bottleneck attention serves as an effective substitute for standard attention in settings where symbolic reasoning and latent sequence alignment are essential.

5.3. Ablation: Training Objective Decomposition

Effect of Loss Components To assess the contribution of each loss component in IntraMIX, Table 2 compares three settings: (1) supervised-only, (2) unsupervised generative modeling without regularization, and (3) the full IntraMIX objective (with both $\mathcal{V}$ and domain distance regularization $D(\rho ,\upsilon )$).

Results indicate that incorporating unpaired trajectory data via $\mathcal{V}$ significantly improves both SR and BLEU over the baseline. The addition of the regularization term further boosts speaker performance, mitigating overfitting caused by domain drift. This validates the importance of aligning embedding distributions across paired and unpaired domains, particularly for speaker generalization.

5.4. Comparison: Alternative Semi-Supervised Strategies

IntraMIX as Follower vs. Speaker-Follower As shown in Table 3, IntraMIX consistently outperforms conventional follower and speaker-follower models across both BabyAI and R2R. Notably, IntraMIX used as a speaker-follower leads to the highest success rate, demonstrating the synergistic value of a generative speaker integrated within a semi-supervised framework. These findings confirm that IntraMIX not only functions as a standalone follower but also strengthens traditional augmentation pipelines when used as a generative speaker.

5.5. Benchmarking Against State-of-the-Art Approaches

Performance on R2RTable 4 presents a comprehensive comparison between IntraMIX and multiple state-of-the-art semi-supervised methods on R2R. When used with greedy decoding, IntraMIX consistently surpasses baseline methods in OSR and performs competitively on SR, despite using fewer training annotations. When equipped with pragmatic inference, IntraMIX significantly outperforms all competitors across every metric and split, confirming its robustness and generalization capabilities.

In particular, IntraMIX’s performance gains in unseen environments underscore its ability to learn transferable semantic representations—a result attributed to its generative modeling of task semantics and its latent space regularization.

6. Conclusions

In this paper, we presented IntraMIX, a novel generative modeling framework for semi-supervised learning in sequence-to-sequence multimodal tasks, with a specific focus on instruction-following agents. Building upon the foundation of Multimodal Variational Autoencoders (M-VAE), our method introduces two key innovations: the bottleneck attention module, which enables alignment-aware latent representation learning, and a domain distance regularization term that ensures consistent cross-domain generalization.

Compared with prior M-VAE architectures, IntraMIX has the distinct advantage of supporting attention mechanisms, making it more compatible with complex sequential tasks where information must be selectively attended over time. This architectural benefit was quantitatively validated through the results shown in Table 1, where attention-enabled models significantly outperformed vanilla seq2seq baselines, particularly on semantically demanding tasks like BossLevel.

Moreover, our ablation results in Table 2 highlight the substantial contribution of each component of our method. The use of unpaired trajectories via the variational ELBO term ($\mathcal{V}$) clearly enhances both policy success rate and speaker performance, while the inclusion of domain distance regularization $D(\rho ,\upsilon )$ further improves language generation fidelity by mitigating embedding drift across data domains.

From a practical perspective, IntraMIX demonstrated dual utility. As shown in Table 2 and Table 3, it significantly improves the performance of the follower when used as a standalone model. Simultaneously, when acting as a speaker within the speaker-follower paradigm, it leads to further gains, as evidenced by the improvements in Table 4. This dual-role capability underscores the complementary nature of our generative approach: it strengthens both components of traditional semi-supervised systems and offers greater flexibility in deployment.

In essence, IntraMIX contributes to the field in two critical ways. First, it offers a powerful framework for exploiting unpaired trajectory data, thereby alleviating the heavy reliance on expensive paired annotation. Second, it introduces a modular design compatible with existing architectures and applicable across modalities, thereby supporting integration into broader multimodal learning scenarios.

Looking ahead, several promising directions remain for future research. One natural extension is to leverage unpaired language data by incorporating a generative model over instructions, i.e., replacing Eq. 4 with a symmetric objective over $p_{\theta }(z,y)$. This would allow IntraMIX to benefit from large-scale, unannotated corpora—a critical resource in instruction-heavy domains such as robotics or AR/VR navigation.

Additionally, given the general nature of our model’s formulation, IntraMIX can be readily applied to other tasks involving sequential multimodal generation, including video captioning, text-to-speech synthesis, dialogue agents with memory, or even vision-language pretraining setups. Since our bottleneck attention mechanism is orthogonal to the specific modality design, its integration with recent large multimodal pre-trained transformers presents an exciting avenue for further investigation.

In summary, this work introduces a flexible, principled, and empirically validated generative learning strategy that improves upon existing semi-supervised frameworks for instruction-following. We hope that IntraMIX will serve as a foundation for future explorations at the intersection of multimodal sequence modeling and semi-supervised reasoning.