Enhanced Multimodal Integration Using TriFusion Networks for Comprehensive Emotion Analysis

1. Introduction

The advent of sophisticated sensors capable of capturing high-fidelity audio and video data [50] has set the stage for breakthroughs in various fields, particularly in passive, non-invasive monitoring systems that could significantly enhance continuous healthcare management for chronic and mental health conditions including diabetes, hypertension, and depression [27]. The integration of these sensors into everyday environments like homes and offices is anticipated to revolutionize how spaces interact with occupants, adapting to and moderating their emotional and psychological states seamlessly and invisibly.

Emotion recognition has emerged as a pivotal area of research within affective computing, driven by the necessity to understand and interpret human emotions in a variety of applications ranging from interactive gaming to psychological analysis [31]. Recent advancements in this field have primarily leveraged deep learning techniques to enhance accuracy and efficiency in detecting emotions from complex datasets. Studies have increasingly focused on multimodal emotion recognition, integrating signals from various sources such as facial expressions, voice intonations, and physiological responses to achieve a holistic understanding of emotional states. Researchers like Kossaifi et al. have demonstrated the effectiveness of neural networks in disentangling these intricate modalities to predict emotions with greater precision [23,40]. Despite progress, challenges remain in handling the subtleties of context-dependent emotional expressions and the inherent subjectivity in emotional data interpretation, which continue to drive innovative solutions in this dynamic field.

The integration of audio and video data analysis is a critical aspect of multimodal emotion recognition, providing a richer context for understanding the nuances of human behavior. Audio-video analysis benefits from the confluence of visual cues, such as facial movements and body language, and auditory signals, like tone and pitch of voice, to form a comprehensive view of an individual’s emotional state. The synchronization of these modalities presents unique challenges, particularly in aligning temporal dynamics and extracting meaningful features that are indicative of emotions. Pioneering work by Trigeorgis et al. on the fusion of audio and video streams through deep learning models exemplifies the advancements in this area, revealing the potential for significantly improved recognition rates over using single modalities [21,48]. These approaches underscore the necessity of developing robust algorithms that can efficiently process and analyze the complex interplay of auditory and visual data to enhance the accuracy and applicability of emotion recognition systems.

The potential of these technologies extends beyond mere convenience, aiming to provide critical support in managing conditions such as autism spectrum disorders, fatigue, and drug addiction through constant monitoring and immediate feedback. The capability to accurately identify and respond to affective states through multimodal emotional analysis is essential in realizing this future [11,21,22]. However, the journey towards effective real-world application is fraught with challenges, including the accurate capture and interpretation of complex spatio-temporal data across diverse populations and environmental conditions [45,46]. Additionally, the creation of expansive, well-annotated multimodal datasets necessary for training robust models remains a costly and labor-intensive endeavor.

To address these challenges, this paper proposes a novel approach to dynamic emotional state analysis using the TriFusion Network, which leverages deep learning to perform intermediate-level fusion of data from auditory, visual, and textual sources [1,3,10]. This method surpasses traditional early and late fusion techniques by optimizing feature extraction, classification, and fusion processes in a cohesive, end-to-end manner. The effectiveness of the TriFusion Network is rigorously validated against contemporary methodologies using the SEWA database, a benchmark in the field of affective computing.

Following this introduction, the paper is structured as follows: Section II provides a detailed review of existing emotional recognition methodologies across different modalities. Section III elaborates on the architecture and theoretical underpinnings of the proposed TriFusion Network. Section IV outlines the experimental setup employed to assess the network’s performance, and Section V discusses the outcomes of these experiments across individual and combined modalities. The paper concludes with a summary of findings and a discussion on future research directions in Section VI.

2. Related Work

Over the last several years, the study of facial expression recognition (FER) has been propelled into the forefront of computational emotion analysis. Numerous methodologies have been developed to identify the seven universally recognized emotions—joy, surprise, anger, fear, disgust, sadness, and neutral—from static facial imagery [6,14,18,23,26,27]. These approaches are generally categorized into two primary techniques: appearance-based and geometric-based methodologies. The recent advent of dynamic facial expression recognition offers a compelling enhancement over static methods, analyzing emotions through a sequence of images or video frames which capture the temporal progression of facial expressions [19]. This dynamic approach facilitates the extraction of both spatial features and their temporal evolution, utilizing shape-based, appearance-based, and motion-based techniques for more nuanced emotion detection.

Shape-based methods, such as the Constrained Local Model (CLM), delineate facial structures using defined anchor points whose movements are tracked to infer emotional states. Appearance-based methods, exemplified by LBP-TOP, analyze textural and intensity patterns across facial images to classify expressions. Motion-based techniques, including free-form deformation models, examine the spatial-temporal dynamics of facial expressions, often necessitating robust facial alignment algorithms for accurate performance. For instance, Guo et al. utilized an atlas construction combined with sparse representation to concurrently harness spatial and temporal data, achieving significantly enhanced recognition accuracy by integrating these dimensions [9].

Emotion recognition has increasingly become a crucial field within human-computer interaction, facilitating advancements that range from customer service bots to therapeutic aids. Significant research has focused on improving recognition algorithms through machine learning models that process complex datasets from facial, vocal, and biometric modalities [72]. The field has seen a particular emphasis on the application of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to capture the subtle dynamics of emotional expressions over time [22]. Recent studies have explored the incorporation of context-aware systems that adjust their processing based on the situational context, aiming to tackle the variability and ambiguity inherent in human emotions [37]. These systems are designed not only to detect basic emotions but also to understand complex affective states and their transitions, challenging the traditional paradigms of emotion recognition with richer, more adaptive models [67,68,69,70,71,72].

In the realm of audio-video analysis for emotion recognition, the synergy between auditory and visual cues has been extensively studied to develop more accurate and reliable systems. This interdisciplinary approach leverages the strengths of each modality to compensate for the limitations of the other, often utilizing advanced signal processing and deep learning techniques [24,45]. the advancement of machine learning techniques, particularly supervised learning, For instance, the integration of facial expression analysis with voice tone analysis allows systems to discern subtleties in emotional expressions that might be ambiguous when only one modality is considered [101]. Researchers have developed frameworks that dynamically align audio and video data streams, extracting temporally correlated features to improve the coherence and accuracy of the emotion detection process [102,104]. These methodologies have been pivotal in advancing real-time emotion recognition systems, enhancing their application in real-world scenarios such as interactive media, surveillance, and telecommunication.

The automatic detection of emotional states through auditory cues has also seen significant advances, particularly in the context of depression and emotion detection. These systems draw parallels in their use of acoustic features to infer psychological states. Research by France et al. demonstrated that variations in formant frequencies could reliably indicate depression and suicidal tendencies [8]. Cummings et al. and Moore et al. achieved considerable success using energy, spectral, and prosodic features to classify depression with accuracies around 70-75% [7,16]. With the increasing prevalence of machine learning, deep neural networks, Long-Short Term Memory networks (LSTMs), and Convolutional Neural Networks (CNNs) have become ubiquitous in enhancing the precision of emotion detection systems [2,3,11,28].

The integration of multimodal data sources has been identified as a particularly potent method for improving the accuracy and reliability of emotion recognition systems. This approach is often implemented at feature, score, or decision levels, with each modality potentially providing complementary information that enhances overall system performance [2,3,11]. Recent studies have explored hierarchical frameworks that adaptively merge input modalities, leveraging varying degrees of certainty from vocal and facial data to detect depression and other emotional states [5,12]. For example, Meng et al. introduced a layered system utilizing Motion History Histogram features, and Nasir et al. employed a multi-resolution model combining audio and video features to diagnose depression more effectively [15,17]. Williamson et al. proposed a system that harnesses speech, prosody, and facial action units to assess depression severity, illustrating the value of multimodal integration [25].

Despite these advancements, several challenges persist in deploying these technologies in real-world scenarios. Often, models are built on limited datasets that may not be fully representative of the population [95,105,106], leading to potential biases and inaccuracies in emotion recognition. The operational variability in how data is captured—using standard equipment in uncontrolled environments—introduces additional complexity. The dynamic nature of human expressions and the contextual factors of recording environments necessitate adaptive models capable of handling intra-class variations and domain shifts. This paper proposes the use of the TriFusion architecture, a sophisticated deep learning model designed to effectively integrate multimodal information for robust emotion recognition. This approach extends beyond conventional feature-level and score-level fusion, implementing a hybrid system that optimizes both features and classifiers for comprehensive multimodal integration.

3. Methodology

This paper introduces the TriFusion architecture, an innovative deep neural network (DNN) framework designed to robustly interpret the complex interplay of behavioral and emotional signals from multimodal sources. Leveraging the nuanced variations in facial expressions, vocal intonations, and textual cues captured in the AVEC SEWA database, TriFusion excels in learning optimal feature representations along with advanced classification and fusion strategies to predict emotional states such as arousal, valence, and liking accurately.

3.1. Feature Modeling

The core strategy of TriFusion involves the simultaneous learning of discriminative feature representations for each modality, accompanied by their respective classification and fusion into a cohesive decision-making framework. Initially, subsets of features from each modality—audio, video, and text—are processed independently through dedicated hidden layers. These layers are tailored to extract the most relevant features for the specific emotional recognition task at hand. Subsequently, the features are amalgamated in the later stages of the network through interconnected fully connected layers that execute both classification and fusion tasks, enabling the system to integrate and interpret multimodal data effectively.

3.1.1. Audio Features

In the audio domain, TriFusion processes 23 distinct acoustic low-level descriptors (LLDs), including energy, spectral components, cepstral features, pitch, voice quality, and micro-prosodic elements. These are sampled every 10ms across short-term frames. For each 6-second segment, a comprehensive feature vector is constructed using a codebook of 1,000 audio words, culminating in a 1,000-dimensional feature vector represented by a histogram of these audio words.

3.1.2. Video Features

Video data is handled by extracting key facial metrics at a frame rate of 20ms. This includes the normalized orientation of the face in degrees and the coordinates of critical facial landmarks—10 points around the eyes and 49 additional facial points. Each type of facial feature is encoded using a unique codebook, generating a histogram that contributes to a composite 3,000-dimensional feature vector for each video segment.

3.1.3. Text Features

Textual information is derived from transcriptions of spoken content, formatted into a bag-of-words model. The model encompasses 521 unique words, focusing solely on unigrams. These text-based features are aggregated over 6-second segments to form a feature set containing 521 distinct elements, providing a textual perspective on the expressed emotions.

3.2. TriFusion Architecture

The TriFusion architecture independently processes each modality—audio, video, and text—through dual-layer fully connected networks designed to capture intra-modality correlations. Following this, a concatenation layer merges these independent outputs into a unified representation, which is then fed into a subsequent fully connected layer that integrates the essence of all modalities. The final output is computed using a single linear neuron that functions as a regression estimator for the overall network, adjusted by a scaling module to align prediction magnitudes with actual label scales. Various scaling techniques such as decimal scaling, min-max normalization, and standard deviation scaling have been evaluated to optimize performance.

The training of the TriFusion model utilizes the Mean Squared Error (MSE) as its loss function, defined as follows:

$$MSE={\displaystyle \frac{1}{m}}\sum _{i=1}^m{(\widehat{y_i}-y_i)}^2$$

(1)

where $\widehat{y}$, y, and m denote the predicted values, actual observed values, and the number of samples, respectively. During the training phase, the `checkpoint’ feature of the Keras API [4] is employed to save and retrieve the most effective model configuration, ensuring each dimension (arousal, valence, liking) is optimally trained for reliable performance.

3.3. System Training

3.3.1. Initialization

The training process of the TriFusion architecture begins with a rigorous initialization phase designed to set optimal conditions for learning. The initialization targets the setup of neural network parameters, specifically the weights and biases, which are crucial for the model’s performance. We employ the He initialization method for weight setup, which is particularly effective for layers using ReLU activation functions by keeping the variance of activations across layers consistent. Each weight matrix W in the network is initialized according to the formula:

$$W=\sqrt{{\displaystyle \frac{2}{n}}}·\mathcal{N}(0,1)$$

where n is the number of inputs to a layer, and $\mathcal{N}(0,1)$ denotes a standard normal distribution. Bias terms are initially set to zero, ensuring a neutral starting point for the first forward pass. This initialization phase is critical as it prevents the gradient vanishing or exploding problems commonly encountered in deep networks, thereby facilitating a stable and efficient gradient descent during the training phase.

3.3.2. Optimization and Backpropagation

Once initialization is complete, the TriFusion model enters the core phase of training, where it learns to minimize a predefined loss function through iterative optimization. We utilize the Adam optimizer, a method well-suited for large-scale and high-dimensional optimization problems. Adam combines the advantages of two other extensions of stochastic gradient descent, namely AdaGrad and RMSProp, specifically designed to handle sparse gradients on noisy problems. The optimizer adjusts the learning rate for each parameter based on estimations of first and second moments of the gradients:

$${\theta }_{t+1}={\theta }_t-{\displaystyle \frac{\eta ·m_t}{\sqrt{v_t}+ϵ}}$$

where $\theta$ represents the parameters, $\eta$ is the step size, $m_t$ and $v_t$ are estimates of the first and second moments of the gradients, respectively, and $ϵ$ is a small scalar added to improve numerical stability. The backpropagation algorithm is applied to compute the gradient of the loss function with respect to each parameter in the network, effectively allowing the optimizer to update all weights and biases in the direction that minimizes the loss.

3.3.3. Loss Function and Regularization

The loss function is pivotal in guiding the training of the neural network. For the TriFusion system, which performs regression tasks, the Mean Squared Error (MSE) is used as the primary loss function, as defined:

$$MSE={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(\widehat{y_i}-y_i)}^2$$

where N is the number of training samples, $\widehat{y_i}$ is the predicted value, and $y_i$ is the actual value for the i-th sample. This choice ensures that the model is penalized based on the square of the difference between predicted and actual values, emphasizing larger errors more significantly, which is suitable for the regression nature of our task. Additionally, to combat overfitting—a common problem in deep learning architectures—L2 regularization is incorporated into the loss function. This regularization term adds a penalty equivalent to the square of the magnitude of coefficients, encouraging the model to maintain smaller weight values:

$$Loss=MSE+\lambda \sum _{w\in W}{w}^2$$

where $\lambda$ is the regularization parameter, and W represents the set of all weights in the network. This regularization method not only helps in reducing overfitting but also promotes a more generalized model that performs well on unseen data.

These paragraphs comprehensively cover the setup, optimization, and regularization processes involved in training the TriFusion model, providing the necessary technical details and mathematical formulations to ensure clarity and depth in understanding the system’s training dynamics.

4. Experimental Settings

4.1. Dataset and Protocol

The RECOLA dataset serves as the primary data source for our experiments, consisting of a training set and a development set. In our study, the development set was further divided into two subsets to refine the evaluation of the TriFusion model. The first subset comprises five subjects selected randomly from the original fourteen, while the remaining nine subjects form a secondary test subset. This division allows for a nuanced assessment of the model’s generalization capabilities across different subsets of data.

For standard Support Vector Regression (SVR) approaches, the conventional protocol was adhered to for constructing unimodal and early fusion models. In contrast, our late fusion approach utilized the initial development subset for optimizing individual unimodal systems, with the fusion function refined using the secondary subset. This stratification addresses the need for precise model tuning under varied conditions.

4.2. Deep Neural Network Configurations

The TriFusion model’s configuration varies according to the emotional dimension being evaluated—arousal, valence, and liking. Each dimension’s specific architecture is meticulously designed to accommodate the idiosyncrasies of the corresponding emotional data. Table 1 details the DNN architecture for each emotional dimension, illustrating the modality-specific layer setups for audio, video, and text, along with the fusion layer’s composition.

The evaluation metric employed is the Concordance Correlation Coefficient (CCC), defined as:

$${\rho }_{y^{\prime }y}={\displaystyle \frac{2*s_{y^{\prime }y}}{s_{y^{\prime }}^2+s_y^2+{(\overline{y^{\prime }}-\overline{y})}^2}}$$

where $y^{\prime }$ and y are the data sets for which the correlation is calculated, $s_{y^{\prime }}^2$ and $s_y^2$ are the variances of these sets, and $\overline{y^{\prime }}$ and $\overline{y}$ are their means.

4.3. Preprocessing Techniques

Adjustments for temporal delays significantly enhance CCC scores. An optimal delay value (d) was experimentally determined for each emotional dimension. For arousal and valence, $d=1.5$ seconds was optimal, while for liking, extending the delay to $d=2.5$ seconds provided the best results.

4.4. Postprocessing Strategies

Postprocessing in the TriFusion model involves scaling the DNN output to enhance prediction accuracy. We employ three scaling methods:

• Min-Max Scaling Normalizes the output within a predefined range, calculated as:

  $${\overrightarrow{y}}_{norm}={\displaystyle \frac{({max}_l-{min}_l)·(\overrightarrow{y}-{min}_p)}{{max}_p-{min}_p}}+\underset{l}{min}$$

  where ${max}_l$ and ${min}_l$ are the maximum and minimum label values, respectively, and ${max}_p$ and ${min}_p$ are the maximum and minimum predicted values.
• Standard-Deviation Ratio Adjusts predictions based on the ratio of standard deviations between predictions and labels, enhancing consistency across data scales:

  $${\overrightarrow{y}}_{norm}={\displaystyle \frac{{\sigma }_p}{{\sigma }_l}}\otimes \overrightarrow{y}$$

  where ${\sigma }_p$ and ${\sigma }_l$ are the standard deviations of predictions and labels, respectively, and ⊗ denotes element-wise multiplication.
• Decimal Scaling Modifies the scale of prediction values to ensure they fall within a normalized range:

  $${\overrightarrow{y}}_{norm}={\displaystyle \frac{\overrightarrow{y}}{{10}^{{min}_p}}}$$

  where ${min}_p$ is the smallest power of ten for which the maximum absolute value of ${\overrightarrow{y}}_{norm}$ is less than one.

The implementation of these postprocessing techniques is critical for aligning the model’s output with actual emotional states, ensuring both accuracy and reliability in the system’s predictions.

5. Experimental Results and Discussion

This section delineates the experimental setup, discusses the methodologies employed, and analyzes the results derived from these experiments.

5.1. SVR-Based Baseline Results

Initial trials were conducted using Support Vector Regression (SVR) to establish a robust baseline for comparison. The parameters, including complexity, epsilon (ranging from 0.0 to 0.0001), and delay (ranging from 0 to 3 seconds), were meticulously optimized on the development set to ensure optimal performance. The results of these trials, as presented in Table 2, provide a comprehensive view of the performance across different modalities. The baseline paper protocol [20] was adhered to, with modifications made only to the early fusion configuration.

A significant observation from these trials is the challenging nature of predicting ’liking’ using audio and video modalities, whereas text data proved more efficacious. Despite the effectiveness of text, its reliance on transcription, which is both costly and time-consuming, poses a practical challenge for real-life applications. However, advancements in speech recognition technologies may potentially mitigate this issue. An intriguing aspect for future exploration is the impact of speech recognition inaccuracies on the precision of emotion detection systems.

Late fusion experiments, utilizing simple linear regression techniques from the sklearn package, indicated varied effectiveness across dimensions. Notably, late fusion excelled for the liking dimension but did not perform as well in others compared to early fusion.

5.2. DNN-Based TriFusion Results

The deployment of the TriFusion DNN architecture was aimed at enhancing the integration of modal inputs more effectively than traditional methods. The early fusion DNN setup was directly contrasted with the SVR models to assess performance variations, with results detailed in Table 3. While the SVR and DNN models performed comparably for arousal, the SVR model showed a 6% improvement over the DNN for valence. Conversely, the DNN model demonstrated significant resilience against less effective modalities, outperforming the SVR by 26% for liking.

Interestingly, the TriFusion model surpasses both DNN and SVR models in predicting arousal and valence, though it still trails behind in the liking prediction when compared to the late fusion approach. This highlights potential areas for model refinement, particularly in managing detrimental modality effects. Ongoing and future investigations will focus on addressing these discrepancies and enhancing model robustness, especially given the initial promising results on the development set.

6. Conclusion and Future Work

6.1. Summary of Contributions

This research introduces the TriFusion architecture, a cutting-edge deep neural network (DNN) designed to enhance emotion recognition by integrating audio, video, and text modalities. This approach innovatively processes each modality through dual fully connected layers before merging them into a unified representation, effectively capturing the nuances of emotional states. Our end-to-end training methodology enables the TriFusion model to surpass previous architectures in terms of Concordance Correlation Coefficient (CCC) performance.

One of the key advancements of the TriFusion model is its ability to handle multimodal data seamlessly, ensuring robust feature extraction and fusion. Preliminary results have demonstrated that our model achieves superior performance metrics, suggesting that the detailed representation learning and fusion strategy are highly effective. However, there remains potential for further enhancement, particularly through the normalization of input features and the application of temporal smoothing techniques to stabilize the regressed outputs.

6.2. Technical Improvements and Optimization

Future developments will focus on refining the input normalization process to accommodate the dynamic range and distribution discrepancies across modalities. This refinement is expected to facilitate more consistent learning and improve the model’s ability to generalize across diverse datasets. Additionally, implementing temporal smoothing on the output predictions will aim to reduce volatility and enhance the temporal coherence of the emotion recognition process, thereby aligning the predictions more closely with the inherent temporal progression of emotional states.

6.3. Expanding Model Capabilities

Looking ahead, the next phase of our research will explore the integration of a recurrent neural network (RNN) layer into the TriFusion architecture. This addition aims to capitalize on the temporal patterns in emotional expressions, potentially boosting accuracy and providing deeper insights into the sequential dynamics of emotions. By leveraging RNNs, we anticipate a significant improvement in the model’s ability to track and predict emotional changes over time, which is crucial for applications requiring continuous emotional monitoring.

Furthermore, we plan to enhance the video component of our model by incorporating features extracted via a convolutional neural network (CNN) that is explicitly trained for emotion-related tasks. This approach is expected to refine the visual feature extraction process, allowing for more precise and contextually relevant information to be captured, which could dramatically improve the model’s performance in real-world scenarios.

6.4. Broader Implications and Future Evaluations

The implications of these advancements extend beyond academic interest, promising significant applications in areas such as interactive media, teletherapy, and human-computer interaction, where understanding and responding to human emotions accurately is crucial. As part of our ongoing work, we will conduct extensive evaluations to assess the practical effectiveness of the TriFusion architecture across various domains and under different operational conditions.

Additionally, to ensure the robustness and applicability of our findings, future studies will involve cross-validation with larger, more diverse datasets and potentially real-time testing environments. These evaluations will help identify any biases or limitations in the current model and guide the development of more adaptive and resilient emotion recognition systems.

6.5. Conclusion

In conclusion, the TriFusion architecture represents a significant step forward in multimodal emotion recognition. By continuously refining and expanding this model, we aim to set new benchmarks in the field and contribute to the development of technologies that can empathetically interact with users across a spectrum of applications. Further research and development will be critical in realizing the full potential of this innovative approach.