Multigranular Unified Synthesis Encoder for Fine-grained Multimodal Emotion Understanding

1. Introduction

Emotion recognition from speech remains a core challenge within the broader context of affective computing, particularly for enabling more nuanced and empathetic human-computer dialogue systems. The task, commonly known as Speech Emotion Recognition (SER), focuses on inferring the speaker’s affective states such as joy, anger, or sorrow through vocal expressions [1]. Despite its potential, SER research faces two fundamental bottlenecks. Firstly, obtaining large-scale labeled emotion datasets is notoriously difficult due to the subjectivity of emotional perception and the demanding nature of multi-annotator labeling processes [2]. Secondly, emotional expression naturally spans multiple modalities, and these modalities exhibit subtle, fine-grained interactions that are challenging to capture and model effectively [3].

A prevalent strategy to address the data limitation is to employ transfer learning paradigms, particularly those driven by self-supervised learning (SSL). SSL has emerged as a game-changing approach, achieving impressive results in NLP [4,5,6] and speech domains [7,8,9]. Within the realm of emotion recognition, pioneering efforts by Acheampong et al. [10] and Pepino et al. [11] have independently leveraged text and speech pre-trained models, yielding promising results. Yet, these endeavors restrict themselves to unimodal processing.

Parallel efforts have sought to enhance multimodal integration by exploring varied fusion strategies. Siriwardhana et al. [12] and Zhao et al. [2] compared early and late fusion schemes combining BERT [4] and Wav2Vec [7] for emotion classification, revealing that late fusion generally achieves better outcomes. However, these methods relied solely on aggregated sentence-level embeddings, neglecting the finer-grained elements such as phonemes or word-level cues. Conversely, models devoid of pre-trained backbones utilized diverse architectures [3,13,14] to capture modality interactions. Notably, Yoon et al. [13] introduced an RNN-based approach to model speech and text separately before concatenating them for classification, albeit at the utterance level. Xu et al. [14] proposed an alignment strategy using LSTM to jointly process text and speech fragments. However, such sequential modeling restricted intramodal interaction exploration. Li et al. [3] developed a fine-grained method using temporal alignment with mean-max pooling and cross-modal mechanisms but introduced extra overhead in alignment prediction, complicating its deployment in practical systems.

While SSL frameworks empower the derivation of robust embeddings from massive unlabeled datasets [12], these representations tend to encode holistic sentence-level semantics rather than fine-grained elements like specific words or phonetic nuances. Addressing this gap necessitates innovative approaches that incorporate granular cross-modal interactions without imposing additional labeling burdens on annotators. Transformer TTS [15] exemplifies such fine-grained modeling in text-to-speech synthesis, employing phoneme and mel spectrogram sequences as inputs to generate speech outputs. Drawing parallels from this domain, we hypothesize that analogous architectures can be repurposed for SER, enabling refined modeling of audio-text interactions at the phoneme level, circumventing explicit alignment steps. Nevertheless, relying solely on phoneme-level inputs falls short in capturing the holistic meaning provided by word-level semantics.

Motivated by these challenges, we propose MUSE, a holistic multigranular framework designed to amalgamate utterance-level pre-trained embeddings with fine-grained representations. At its core, MUSE leverages a specialized multilevel Transformer module to instill cross-modal alignment among voice fragments, words, and phonemes, while integrating phoneme embeddings with their corresponding word embeddings through sophisticated strategies. Additionally, a vanilla Transformer encoder [16] is incorporated to reinforce the sequential multimodal representation aggregation. For global semantics, BERT [4] serves as our chosen pre-trained model backbone, facilitating multigranular fusion. Our rigorous experimental validations on the IEMOCAP dataset [17] demonstrate that MUSE consistently surpasses contemporary state-of-the-art methods, with its multigranular design delivering tangible performance uplift.

To summarize, our principal contributions are outlined as follows:

–

We introduce a novel multilevel Transformer-based encoder, MUSE, capable of modeling intricate cross-modal interactions at the phoneme, word, and utterance levels (Section 3.1), fostering enriched multimodal feature representations.

–

We devise an efficient yet effective multigranular fusion paradigm that seamlessly bridges fine-grained and pre-trained utterance-level representations, enhancing emotion understanding fidelity (Section 3.2).

–

We comprehensively evaluate MUSE on the IEMOCAP benchmark, wherein it achieves superior results over existing approaches, validating the efficacy of our proposed multigranular fusion strategy (Section 4).

2. Related Work

The evolution of speech emotion recognition (SER) has witnessed a significant paradigm shift from traditional handcrafted feature-based methods to contemporary deep learning-driven approaches. Early research predominantly relied on classical machine learning algorithms such as the Hidden Markov Model (HMM) [18] and the Gaussian Mixture Model (GMM) [19], where carefully designed low-level acoustic descriptors and statistical high-level features formed the foundation for emotion classification tasks. These conventional methods, while providing initial insights into the field, were inherently limited by their dependence on shallow representations and lacked the capacity to capture complex, hierarchical patterns present in emotional speech data.

To address these constraints, the advent of deep neural networks (DNNs) introduced transformative capabilities in SER. D. Bertero et al. [20] pioneered the utilization of convolutional neural networks (CNNs) to automatically extract abstract feature representations directly from raw spectrogram inputs, facilitating end-to-end learning without the necessity of manual feature engineering. Similarly, A. Satt et al. [21] proposed a hybrid framework combining CNNs and long short-term memory (LSTM) networks, effectively capturing both spatial and temporal emotional cues. This approach enabled the model to learn hierarchical feature structures while simultaneously modeling contextual dependencies inherent in speech sequences.

With the growing realization that human emotional expression is inherently multimodal, recent research has expanded towards integrating both auditory and textual modalities to enrich emotion recognition systems. For example, S. Yoon et al. [13] developed a multimodal SER model employing recurrent neural networks (RNNs) to encode both audio and text streams independently. Their approach involved utilizing the final hidden state of one modality’s encoder as a query in an attention mechanism applied to the other modality’s encoder outputs, thereby attempting to bridge the two modalities during fusion. Nevertheless, their method did not fully exploit bidirectional inter-modality interactions, and the attention mechanism primarily focused on global representations rather than exploring fine-grained token-level or phoneme-level interactions.

In another line of work, H. Xu et al. [14] proposed a model leveraging LSTM-based encoders alongside an attention mechanism to align and integrate audio and text modalities. Although this method introduced a more explicit alignment between modalities, it remained restricted by the sequential nature of LSTM, which processes information linearly and may suffer from information loss over long sequences. Moreover, their approach did not sufficiently explore intramodal dynamics, potentially neglecting important cues within the individual modalities themselves.

Further pushing the boundaries, H. Li et al. [3] introduced a fine-grained multimodal emotion recognition framework by incorporating a temporal alignment mean-max pooling strategy. This method facilitated more granular interactions between audio and text, allowing the model to focus on localized temporal segments where emotional cues are most prominent. In addition, they integrated a cross-modality interaction module to enhance feature fusion. However, their framework required explicitly aligned audio and text inputs, which imposes significant preprocessing overhead and limits scalability in real-world scenarios where such alignment information might not be readily available or reliable.

Despite these advancements, current approaches still face limitations in capturing the subtle nuances of multimodal emotional expression. The primary shortcoming lies in the insufficient modeling of cross-modal and intra-modal interactions at different granularity levels, including utterance, word, and phoneme levels. This motivates the need for more holistic frameworks that can flexibly incorporate fine-grained interactions while leveraging the robustness of pre-trained representations.

Our proposed MUSE framework draws inspiration from these prior works yet overcomes their limitations by introducing a unified multigranular synthesis encoder that explicitly models cross-modal interactions at multiple granularity levels. Through the integration of a hierarchical Transformer-based encoder and a multigranular fusion module, MUSE allows for seamless blending of pre-trained utterance-level embeddings with fine-grained audio-text representations without requiring additional alignment annotations. Furthermore, by employing advanced attention mechanisms that dynamically capture both inter-modality and intra-modality dependencies, MUSE offers a more comprehensive approach for multimodal emotion understanding.

In contrast to earlier models that relied solely on either early or late fusion strategies, our method harmonizes both strategies within a unified learning framework, offering a more adaptive and fine-grained fusion scheme. Moreover, by leveraging recent advances in self-supervised learning (SSL) and Transformer architectures, MUSE capitalizes on large-scale unannotated data to pretrain its encoders, thereby mitigating the reliance on scarce labeled datasets. This allows the framework to generalize more effectively across diverse emotional contexts and speaker variabilities.

In summary, while the field of multimodal emotion recognition has made remarkable progress over the years, our review of the literature highlights persistent gaps in modeling capabilities, particularly in handling fine-grained interactions and reducing the dependency on explicit alignment information. MUSE is designed to address these gaps by synergizing fine-grained feature modeling with global semantic representations, paving the way for more robust and scalable SER systems.

3. Framework

In this section, we provide a comprehensive introduction to our proposed MUSE (Multigranular Unified Synthesis Encoder) framework. First, we detail the novel multilevel Transformer-based module designed for fine-grained multimodal emotion understanding. Subsequently, we elaborate on the overall multi-granularity fusion strategy, which unifies our proposed module with powerful pre-trained language representations.

3.1. Multilevel Transformer Encoder for Fine-grained Interaction

3.1.1. Model Design and Workflow

Inspired by the architecture of Transformer TTS [15], which itself is based on the foundations of Tacotron2 [22] and the Transformer model [16], we present a novel multilevel Transformer module within MUSE. This module is designed to systematically model the intricate cross-modal dependencies between phonemes, words, and acoustic representations.

Figure 1. Overview of Multilevel Transformer.

Figure 1. Overview of Multilevel Transformer.

Initially, the text input undergoes decomposition into both phoneme and word sequences, ensuring the extraction of both micro and macro linguistic cues. These are subsequently passed through a highway network [23] to enhance gradient flow and prevent information bottlenecks. Following this, an encoder pre-net composed of three convolutional layers and a projection layer processes the textual embeddings into a format suitable for sequential encoding. Simultaneously, the mel spectrogram, capturing the speech signal, is processed through a two-layer fully connected network to generate a condensed representation.

To enable rich cross-modal interaction, the processed text and audio features are fed into a Cross-Modal Attention Interaction Module, followed by a Deep Fusion Module, both of which leverage the vanilla Transformer architecture. Finally, a classification head operates on the aggregated representation derived from the dummy mel input vector, predicting the probability distribution over emotion categories.

3.1.2. Sequential Interaction Modeling and Cross-modal Encoding

During both training and inference phases, the system consumes the complete mel spectrogram and textual information, ensuring maximal context utilization. A dummy mel vector $m_{dummy}$ is prepended to the mel spectrogram sequence, yielding $m=(m_{dummy},m_1,m_2,...,m_{T^{\prime }})$. This enables the model to derive a global representation from the sequence, akin to the [CLS] token in BERT-based models.

The sequence-to-sequence nature of TTS tasks can be formalized as:

$$f\left(o_t\right|x_1,...,x_T)=f\left(o_t\right|o_{<t},x)$$

(1)

However, in our scenario, the target is not mel spectrogram prediction but rather emotion classification, formalized as:

$$p=g(x,m)$$

(2)

where $g(x,m)$ denotes the classification function leveraging both modalities.

3.1.3. Hierarchical Phoneme and Word Embedding Strategy

Recognizing the nuanced contribution of phonemes in emotion cues, we apply a convolutional neural network (CNN) to extract fixed-dimensional phoneme embeddings for each word following [24,25]. Specifically:

$$e_{phoneme}^i=\mathrm{MaxPool}\left(\mathrm{CNN}\left({\mathrm{PhonemeSeq}}_i\right)\right)$$

(3)

In parallel, we incorporate word-level semantic embeddings derived from GloVe [26]:

$$e_{word}^i=\mathrm{GloVe}\left(w_i\right)$$

(4)

3.1.4. Fusion of Phoneme and Word Embeddings

We explore two approaches to integrate phoneme and word embeddings:

1) Concatenation Strategy: A naive yet effective method involves direct concatenation:

$$u_i=[e_{phoneme}^i;e_{word}^i]$$

(5)

2) Highway Network-based Fusion: Inspired by [23], the fused vector is passed through a two-layer highway network:

$$Z\left(u\right)=H\left(u\right)·T\left(u\right)+u·(1-T(u\left)\right)$$

(6)

where $H\left(u\right)=\mathrm{ReLU}(W_{h}u+b_h)$ is the transformation function, and $T\left(u\right)=\sigma (W_{t}u+b_t)$ is the transform gate controlling information flow.

3.1.5. Cross-modal Attention and Deep Fusion Modules

Each module adopts a vanilla Transformer block comprising multi-head self-attention, position-wise feed-forward layers, and residual connections [16]. The Cross-Modal Attention module uses encoder-decoder attention to condition the mel features on the textual embeddings, defined as:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(7)

where Q are the query vectors from mel representations, and $K,V$ are key and value vectors from text encodings.

The Deep Fusion Module aggregates these multimodal features through additional self-attention layers, enhancing both intra and inter-modal interactions.

3.1.6. Objective Functions and Multi-task Considerations

While inspired by multitask learning strategies from [28], we empirically observed no gains from joint TTS and SER loss optimization in our setting. Thus, we optimize only the cross-entropy loss for emotion classification:

$$L_{SER}=-\sum _{i=1}^N\sum _{k=1}^K{y}_{i,k}log{p}_{i,k}$$

(8)

We further regularize our model using auxiliary attention alignment loss:

$$L_{align}={\displaystyle \frac{1}{T\times T^{\prime }}}\sum _{t=1}^T\sum _{t^{\prime }=1}^{T^{\prime }}\mathrm{KL}(A_{t,t^{\prime }}\left|\right|{\widehat{A}}_{t,t^{\prime }})$$

(9)

where $A_{t,t^{\prime }}$ is the predicted attention matrix and ${\widehat{A}}_{t,t^{\prime }}$ is the ideal alignment.

3.2. Multigranular Fusion Network

3.2.1. Global and Local Representation Integration

Our multigranular framework leverages both global sentence-level embeddings from pre-trained BERT and the fine-grained multimodal representation from the multilevel Transformer encoder. BERT’s [CLS] token is used as the global semantic summarization of the text sequence:

$$e_{BERT}={\mathrm{BERT}}_{CLS}\left(x\right)$$

(10)

Concurrently, the first vector from the Deep Fusion Module output represents the fine-grained cross-modal embedding:

$$e_{MUSE}={\mathrm{DFM}}_{\left[1\right]}(m,x)$$

(11)

3.2.2. Fusion Strategy and Classification Head

Adopting a late fusion strategy inspired by [12,29], we perform:

$$e_{concat}=[\mathrm{Proj}\left(e_{BERT}\right);\mathrm{Proj}\left(e_{MUSE}\right)]$$

(12)

This concatenated representation is passed through a feed-forward classifier:

$$p=\mathrm{softmax}(W_c{e}_{concat}+b_c)$$

(13)

where $W_c$ and $b_c$ are learnable parameters, and p is the predicted emotion category distribution.

Figure 2. Illustration of Multigranular Fusion Network.

Figure 2. Illustration of Multigranular Fusion Network.

3.2.3. Enhanced Objective Function with Regularization

In addition to $L_{SER}$ and $L_{align}$, we introduce an auxiliary embedding alignment loss ensuring the coherence of $e_{BERT}$ and $e_{MUSE}$:

$$L_{emb\_align}={\parallel e_{BERT}-e_{MUSE}\parallel }_2^2$$

(14)

Our final loss is:

$$L_{total}=L_{SER}+{\lambda }_1{L}_{align}+{\lambda }_2{L}_{emb\_align}$$

(15)

where ${\lambda }_1$ and ${\lambda }_2$ are balancing hyperparameters.

4. Experiments

In this section, we conduct extensive experimental evaluations to validate the effectiveness and robustness of our proposed MUSE framework. We systematically describe the dataset utilized, elaborate the implementation configurations, and present comprehensive performance comparisons with state-of-the-art methods. Additionally, we perform an ablation study to investigate the contribution of different modules within the MUSE architecture. All experimental results are analyzed both quantitatively and qualitatively to offer in-depth insights.

4.1. Dataset and Experimental Settings

To comprehensively assess the performance of our models, we adopt the widely-used Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset [17], a benchmark dataset extensively applied in multimodal emotion recognition research. This dataset comprises approximately 12 hours of audiovisual recordings, including speech, video, and textual transcripts. In this study, we exclusively utilize the speech audio and corresponding text transcripts.

Following the prevalent experimental protocol proposed by prior works such as [13], we consider four core emotion categories: angry (1103 samples), sad (1084 samples), neutral (1708 samples), and happy (merged with excited, totaling 1636 samples), resulting in an overall dataset containing 5531 utterances. For experimental consistency, we apply 5-fold speaker-independent cross-validation with splits configured as 60% for training, 20% for development, and 20% for testing. Each experiment is conducted thrice with distinct random seeds, and the averaged performance is reported to mitigate randomness bias.

4.2. Implementation Details and Training Protocol

All models are implemented using the PyTorch deep learning framework. For acoustic feature extraction, 128-dimensional filterbank features are computed from raw speech waveforms using a window size of 25ms and a hop size of 12ms. For textual modality, we utilize 300-dimensional pre-trained GloVe embeddings [26] to represent words. All Transformer modules within our MUSE framework are configured with a hidden size of 128 and utilize multi-head self-attention with 4 heads.

The models are optimized using the Adam optimizer [30] with an initial learning rate set to $1e^{-5}$ and a mini-batch size of 4. We employ early stopping based on the best validation Weighted Accuracy (WA) over 10 epochs. Final model evaluations are performed using both WA and Unweighted Accuracy (UA) metrics on the held-out test set to provide a comprehensive understanding of performance across class distributions.

4.3. Evaluation of MUSE’s Multilevel Transformer Component

We perform an extensive evaluation of our multilevel transformer module embedded within MUSE on the IEMOCAP dataset. Table 1 reports the comparative results. It is evident that the proposed multilevel transformer model significantly outperforms all previous baselines in both WA and UA. Notably, our approach surpasses the current best method [3], achieving a WA of 0.735 and UA of 0.747, demonstrating the effectiveness of fine-grained cross-modal modeling without relying on costly alignment annotations.

The ablation analysis further reveals that word embeddings contribute more substantially to the recognition task compared to phoneme-only representations, suggesting the importance of semantic context. Furthermore, integrating both word and phoneme embeddings through a highway network achieves superior performance, validating the design choice of using highway networks to enable efficient information fusion. Additionally, the absence of the Deep Fusion module leads to observable performance degradation, emphasizing its crucial role in capturing deep multimodal dependencies.

As illustrated in Table 2, we conduct an in-depth sensitivity analysis on the number of Transformer layers across different modules. Our findings indicate that a configuration with 1-layer text encoder, 1-layer cross-modal attention, and 2-layer deep fusion achieves optimal results, balancing model capacity and overfitting risk. These observations align with the intuition that deeper fusion layers help the model capture complex cross-modal dependencies effectively.

4.4. Assessment of MUSE’s Multigranular Fusion Framework

We further evaluate the full MUSE framework with multigranular fusion. As depicted in Table 3, integrating global utterance-level representations from BERT with fine-grained multimodal features yields a notable performance uplift, achieving a WA of 0.752 and UA of 0.756. This result demonstrates the clear advantage of our multigranular fusion strategy, which synergistically combines complementary information from different granularity levels. The empirical findings confirm that such unified fusion can effectively bridge the gap between local acoustic-textual patterns and global semantic cues, enhancing the robustness and discriminative power of emotion recognition systems.

5. Conclusions and Future Directions

In this work, we introduced MUSE, a novel Multigranular Unified Synthesis Encoder framework designed to advance the state-of-the-art in speech emotion recognition (SER). At the core of MUSE lies an innovative multilevel Transformer-based module, meticulously architected to enable fine-grained and hierarchical interactions across diverse modalities, specifically voice fragments, words, and phonemes. To the best of our knowledge, this constitutes the first endeavor to repurpose and adapt the structural principles of Transformer TTS [15] within the domain of SER. Our approach bridges the gap between coarse-grained sentence-level representations and fine-grained token-level features, facilitating more nuanced emotional understanding. Furthermore, we proposed a carefully designed multi-granularity fusion strategy within MUSE, which effectively amalgamates the fine-grained representations generated by our multilevel Transformer with global utterance-level embeddings derived from powerful pre-trained language models, such as BERT [4]. Unlike existing methods that either focus solely on pre-trained models or require costly alignment annotations, our framework offers a simple yet highly effective fusion mechanism, which significantly enhances the capability of multimodal emotion recognition systems.

Extensive and rigorous experimental evaluations conducted on the benchmark IEMOCAP dataset [17] demonstrate that our proposed MUSE framework consistently outperforms previous state-of-the-art methods across all evaluated metrics, including Weighted Accuracy (WA) and Unweighted Accuracy (UA). In particular, the multilevel Transformer component alone surpasses prior multimodal fusion approaches, while the complete multigranularity fusion framework further elevates performance, indicating the complementary nature of fine-grained and pre-trained representations. Our ablation studies also verify the robustness and critical contributions of each module within MUSE.

We believe that our proposed MUSE framework not only presents an effective solution for SER but also offers broader implications for other multimodal understanding tasks. By demonstrating the feasibility and superiority of integrating fine-grained cross-modal interactions with global semantic embeddings, our approach can serve as a reference design for future works involving other modalities or tasks, such as sentiment analysis, empathy modeling, or multimodal dialogue systems. To promote further research and facilitate reproducibility, we will release our code, pretrained models, and detailed training configurations to the public community. Looking forward, we envisage several promising avenues for extending this work. One natural direction is to incorporate recent advances in self-supervised acoustic models, such as Wav2Vec 2.0 [8], into the MUSE framework. By replacing or complementing the current mel spectrogram representation with more expressive representations from such pre-trained acoustic models, we expect to further enhance the emotion recognition capability, particularly in low-resource and noisy scenarios. Additionally, exploring adaptive granularity fusion strategies, where the fusion weightings are dynamically adjusted based on context or emotional intensity, could further refine our framework’s effectiveness.

Another research direction involves extending MUSE into a broader multi-task learning paradigm, jointly optimizing emotion recognition along with auxiliary tasks such as speaker identification, sentiment detection, or affective reasoning. We hypothesize that such auxiliary tasks could provide beneficial supervision signals to enrich the emotion understanding capability of the model. Finally, applying and validating MUSE on more diverse datasets, including in-the-wild conversational datasets and multi-language emotional corpora, will be critical to assess the generalization potential of our proposed methods. In conclusion, we hope our work serves as a stepping stone toward more general, robust, and explainable multimodal emotion understanding systems, and we encourage the community to further explore and extend the directions outlined in this paper.