Multimodal Information Fusion with Neural Gating

1. Introduction

Representation learning techniques have garnered substantial attention from both researchers and industry practitioners due to their remarkable success in addressing intricate challenges across various domains, including computer vision, speech recognition, and natural language processing [7,39,66]. Despite the natural occurrence of multimodal data in real-world scenarios, where information is often presented through multiple channels such as images, text, and audio, the majority of existing efforts have predominantly focused on unimodal data. Multimodality encapsulates the concept that a single real-world entity can be depicted through different perspectives or data types. For instance, collaborative platforms like Wikipedia provide comprehensive descriptions of notable individuals by integrating textual narratives, visual images, and occasionally audio recordings. Similarly, users on social media platforms often comment on events such as concerts or sporting events using concise phrases accompanied by multimedia attachments, including images, videos, and audio clips. In the medical field, patient records are typically represented by a heterogeneous collection of images, sounds, textual notes, and various physiological signals.

The burgeoning availability of multimodal datasets from diverse sources has propelled the advancement of automated analytical techniques designed to harness the rich potential of such data. These techniques aim to uncover patterns and structures that reveal complex interrelationships among different data modalities [15,18,67]. In recent years, the research community focused on representation learning has increasingly prioritized multimodal tasks. Notable applications include visual question answering [14] and image captioning [32,60,62,74], where innovative methods for integrating various representation learning architectures have been developed to effectively combine information from multiple modalities.

Most contemporary models in multimodal learning emphasize either mapping information from one modality to another or addressing auxiliary tasks to construct a unified representation that encapsulates the information from all involved modalities [4,5,6]. In contrast, our work proposes a novel module specifically designed to integrate multiple information sources in a manner that is directly optimized with respect to the primary objective function of the task at hand. The cornerstone of our proposed module is the concept of gating mechanisms, which selectively determine the contribution of each input modality to the final representation. By employing multiplicative gates, the Fusion Gate Unit (FGU) assigns varying degrees of importance to different features concurrently [1,2], thereby generating a rich and dynamic multimodal representation. This approach obviates the need for manual tuning of feature weights, as the FGU autonomously learns the optimal gating patterns directly from the training data.

One of the key advantages of our gated model is its versatility; the FGU can be seamlessly integrated into various neural network architectures tailored to different tasks. Furthermore, it can be optimized in an end-to-end fashion alongside other components of the network using standard gradient-based optimization techniques, ensuring cohesive and efficient learning across the entire model.

To illustrate the practical applicability of our approach, we investigate the task of movie genre identification based on two distinct modalities: plot summaries and poster images. Genre classification is a fundamental task with wide-ranging applications, including document categorization [33,82], recommendation systems [43], and information retrieval systems. [83,84] Previous predictive models, such as MaxoutMLP_w2v and VGG_transfer (detailed in Section 3), which are grounded in representation learning, demonstrate that even human evaluators might struggle to accurately classify genres without access to both plot and visual information.

The central hypothesis of this research is that integrating Fusion Gate Units into the neural network architecture, as opposed to employing manually engineered multimodal fusion strategies, will enable the model to learn input-dependent gating patterns. These patterns will effectively determine the relative contribution of each modality to the activation of hidden units, thereby enhancing the model’s ability to generate accurate and meaningful predictions.

The remainder of this paper is structured as follows: Section 2 provides a comprehensive review of related literature and discusses the context of previous research in multimodal learning and fusion techniques. Section 3 delineates the methodologies employed as baselines and elaborates on our proposed representation-learning-based model incorporating Fusion Gate Units. Section 4 outlines the experimental setup, including the specifications of the MM-IMDb dataset. Section 5 presents and analyzes the results obtained from the movie genre classification experiments. Finally, Section 6 summarizes the key findings, discusses their implications, and proposes avenues for future research.

2. Related Work

2.1. Multimodal Fusion Research

Extensive reviews [15,18,20,40,86,97] have consolidated various methodologies addressing the challenges of multimodal analysis. A prevailing theme across these studies is the demonstrated advantage of multimodal approaches over their unimodal counterparts in performing automatic analysis tasks. Typically, a multimodal analysis framework processes two or more distinct modalities—such as video, audio, images, and text—that collectively describe a specific concept or entity. In recent developments, there has been a growing consensus on leveraging representation learning models to effectively capture and represent information from these diverse data sources [39,88]. Despite advancements in feature extraction, the optimal strategy for integrating these extracted features remains an active area of research.

The primary objective of multimodal fusion is to construct a unified representation that facilitates more efficient and accurate automatic analysis tasks, such as building classifiers or predictors. A rudimentary yet commonly employed technique involves concatenating features from different modalities to form a single, comprehensive feature vector [35,51,57]. While this approach is straightforward and easy to implement, it often overlooks the intrinsic correlations and interdependencies between the various modalities, potentially limiting the model’s performance.

To address the limitations of simple concatenation, more sophisticated fusion strategies have been proposed, including the use of Restricted Boltzmann Machines (RBMs) and autoencoders. For instance, Ngiam et al. [48], Fei et al. [94] demonstrated the efficacy of concatenating higher-level representations by training two separate RBMs to independently reconstruct audio and video representations. Furthermore, they developed a model capable of reconstructing both modalities from just one, effectively capturing the interplay between them. Notably, their approach was able to emulate the McGurk effect, a perceptual phenomenon illustrating the interaction between auditory and visual stimuli in speech perception. Building on this, Srivastava Salakhutdinov [56], Xu et al. [75] extended the framework by incorporating Deep Boltzmann Machines, modifying both the feature learning and reconstruction phases. They argued that such a strategy could exploit vast amounts of unlabeled data, thereby enhancing performance in tasks like retrieval and annotation.

Similar methodologies have emerged that utilize neural network architectures for multimodal fusion [13,21,34,37,41,45,58,61,102,103]. These approaches typically involve separate input layers for each modality, which are then integrated into a final supervised layer, such as a softmax regression classifier. This modular design allows for flexibility in handling various types of input data while maintaining a unified prediction mechanism.

An alternative fusion paradigm focuses on designing objective or loss functions tailored to the specific target task [11,22,38,45,54,55,64]. These strategies often operate under the assumption that a shared latent space exists, where different modalities can represent the same semantic concepts through appropriate transformations of the raw data. Semantic embedding representations are crafted such that semantically similar concepts are mapped to proximate locations in this latent space [49]. For example, Socher et al. [54] proposed a multimodal approach for zero-shot classification by training a word-based neural network to represent textual information and employing unsupervised feature learning models for image representation. Their fusion technique involved learning a linear mapping to project images into the semantic word space, supplemented by a Bayesian framework to discern whether an image belonged to a seen or unseen class. Similarly, Frome et al. [22] utilized a Convolutional Neural Network (CNN) trained on the ImageNet dataset for image representation and a word-based neural language model [47,105] for text. They achieved fusion by retraining the CNN with text representations as targets, significantly improving scalability from 2 to 20,000 unknown classes in zero-shot learning tasks. Norouzi et al. [49] further refined this approach by constructing a convex combination using classifier-estimated probabilities and semantic embedding vectors for unseen labels, achieving state-of-the-art results. These task-specific fusion models, however, tend to be tightly coupled with their respective tasks, necessitating adaptations when applied to different domains or objectives.

Our proposed model, the Gated Multimodal Unit (FGU), shares conceptual similarities with the Mixture of Experts (MoE) framework [31]. While MoE is predominantly utilized for decision fusion—combining multiple predictors to tackle supervised learning problems [63]—the FGU is designed as an integral component within the representation learning paradigm. Unlike traditional MoE models, the FGU operates independently of the final task, such as classification, regression, or unsupervised learning, provided that the associated cost function remains differentiable. This design allows the FGU to seamlessly integrate into various neural network architectures, enhancing their ability to learn rich, multimodal representations without being confined to specific application domains.

2.2. Movie Genre Classification Application

In the realm of movie genre classification, a diverse array of methodologies has been explored, leveraging various modalities to characterize each film. These modalities include textual features, image-based features, and multimedia elements such as audio and video. One of the pioneering studies in this area was conducted by Huang et al. [27], who classified movie previews into three genres by extracting handcrafted video features and training a decision tree classifier. Their evaluation was based on a relatively small dataset of 44 films, highlighting the nascent stage of research in this domain.

Focusing solely on textual data, Shah et al. [52] tackled single-label genre classification using clustering algorithms applied to movie scripts from a dataset comprising 260 movies. This approach underscored the potential of textual analysis in genre prediction but was limited by its single-label framework. Subsequently, Pais et al. [50] extended this work by incorporating both visual and textual features to classify movies as either drama or non-drama, utilizing a dataset of 107 samples. Their bimodal approach demonstrated improved classification performance over unimodal methods, albeit within a binary genre classification context.

Expanding the scope to include multiple modalities, Hong Hwang [25] investigated the use of Probabilistic Latent Semantic Analysis (PLSA) models to integrate audio, image, and text data for predicting the genre of movie previews. Their study focused on single-label classification across four genres using a dataset of 140 movies sourced from IMDb, highlighting the challenges of multimodal integration in genre prediction tasks.

More recently, Fu et al. [23] employed a combination of handcrafted visual features for poster analysis and bag-of-words representations for synopses. They trained separate Support Vector Machines (SVMs) for each modality and subsequently combined their predictions. Their approach was evaluated on a substantially larger dataset of 2,400 movies, each annotated with a single genre out of four possible categories. This study underscored the scalability of multimodal fusion techniques but remained constrained by its single-label classification framework.

The aforementioned studies predominantly addressed genre classification within a single-label setup. However, the multilabel scenario is arguably more reflective of real-world conditions, as most movies encompass multiple genres (e.g., "The Matrix" (2000) is categorized as both Sci-fi and Action). In this context, Anand [12] explored the efficacy of utilizing keywords and user-generated tags for multilabel genre classification, employing the MovieLens 1M dataset, which includes 1,700 movies. Additionally, Ivasic-Kos et al. [29], Ivasic-Kos et al. [30] conducted multilabel classification using handcrafted features derived from movie posters, working with a dataset of 1,500 samples across six genres. Further extending this line of research, Makita Lenskiy [43], Makita Lenskiy [44] leveraged the movie ratings matrix and genre correlation matrix to predict genres within a smaller subset of the MovieLens dataset, encompassing 18 movie genres.

A common limitation across these studies is the reliance on publicly available MovieLens datasets, which lack a standardized experimental setup, making systematic comparisons between different methodologies challenging. Moreover, the scale of these datasets is relatively modest, with none exceeding 10,000 samples. Addressing these limitations, our work introduces a new dataset derived from the MovieLens 20M dataset, significantly expanding the number of samples and genres. This dataset not only includes genre annotations, poster images, and plot summaries but also encompasses over 50 additional characteristics sourced from the IMDb website. By releasing both the dataset and the accompanying source code, we aim to facilitate the inclusion of more movies and genres in future research, thereby providing a robust benchmark for evaluating and comparing multimodal genre classification models.

Our FGU-based approach stands out in this landscape by offering a flexible and scalable solution for multilabel genre classification. Unlike previous models that often require task-specific adaptations, the FGU can dynamically learn to weigh the contributions of different modalities, enhancing classification performance across a diverse set of genres and dataset scales. This adaptability is particularly beneficial in multilabel scenarios, where the interplay between various genres and their corresponding modalities can be complex and multifaceted.

3. Methods

This study introduces a sophisticated neural network-based framework tailored for the multilabel classification of multimodal datasets. Central to this framework is the innovative Gated Multimodal Unit (FGU), a novel hidden unit designed to autonomously determine the influence of each modality on the unit’s activation through the use of gating mechanisms. Detailed elaboration of the FGU architecture is provided in SubSection 3.1.

Recognizing that statistical properties often vary significantly across different modalities [56], it is essential to adopt distinct representation strategies that align with the inherent characteristics of each data type. This research explores a variety of techniques for representing textual and visual data. For textual information, we evaluated multiple approaches, including word2vec models, n-gram models, and Recurrent Neural Network (RNN) models, as discussed comprehensively in SubSection 3.2. Conversely, for visual data, we assessed two different Convolutional Neural Network (CNN) architectures, detailed in SubSection 3.3, to effectively capture and represent visual features.

3.1. Gated Multimodal Unit for Multimodal Fusion

Multimodal learning is intrinsically linked to the concept of data fusion, which seeks to integrate multiple information sources into a cohesive representation that encapsulates more comprehensive information than any individual source could provide [18]. Data fusion can be categorized broadly into two primary strategies: feature fusion and decision fusion. Feature fusion, often referred to as early fusion, involves selecting and combining subsets of features from different modalities or creating new feature combinations that better represent the underlying information necessary to address a specific problem. This approach aims to create a unified feature space that captures the complementary strengths of each modality. On the other hand, decision fusion, or late fusion, entails combining the outputs or decisions from separate systems or classifiers, typically through methods such as averaging, voting, or more sophisticated Bayesian frameworks, to reach a consensus decision.

In this research, we propose the Gated Multimodal Unit (FGU), a novel model that amalgamates principles from both feature and decision fusion methodologies through the use of gated neural networks. The FGU draws inspiration from the gating mechanisms found in recurrent architectures like Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM) networks, which are renowned for their ability to control information flow within neural networks. The primary function of the FGU is to serve as an internal component within a larger neural network architecture, facilitating the creation of intermediate representations by effectively combining data from multiple modalities.

Each modality is represented by a feature vector, denoted as $x_i$, where i indexes the different modalities. These feature vectors are processed through neurons equipped with hyperbolic tangent (tanh) activation functions, which encode modality-specific internal representations. For every input modality $x_i$, there exists an associated gate neuron, represented by sigmoid ($\sigma$) activation functions. These gate neurons play a crucial role in regulating the contribution of each modality’s features to the overall output of the FGU. Specifically, when a new data sample is introduced to the network, each gate neuron evaluates the collective feature vectors from all modalities to determine the extent to which its corresponding modality should influence the internal encoding of that particular sample.

The mathematical formulation governing the FGU is as follows:

$$\begin{array}{cc}\hfill h_v& =tanh\left(W_v·x_v\right)\hfill \\ \hfill h_t& =tanh\left(W_t·x_t\right)\hfill \\ \hfill z& =\sigma \left(W_z·\left[x_v,x_t\right]\right)\hfill \\ \hfill h& =z\odot h_v+\left(1-z\right)\odot h_t\hfill \\ \hfill \Theta & =\left\{W_v,W_t,W_z\right\}\hfill \end{array}$$

Here, $h_v$ and $h_t$ represent the internal representations derived from the visual and textual modalities, respectively. The gate z is computed by applying a sigmoid function to a linear combination of the concatenated input modalities, effectively determining the weighting between $h_v$ and $h_t$. The symbol ⊙ denotes element-wise multiplication, ensuring that each feature is appropriately scaled based on the gate’s output. The parameter set $\Theta$ comprises the weight matrices $W_v$, $W_t$, and $W_z$, which are learnable parameters optimized during the training process.

One of the significant advantages of the FGU is its differentiable nature, which allows seamless integration with other neural network components. This compatibility ensures that the entire network, including the FGU, can be trained end-to-end using standard gradient-based optimization algorithms, such as stochastic gradient descent (SGD). By enabling the network to learn optimal gating patterns directly from the training data, the FGU obviates the need for manual tuning of feature weights, thereby enhancing the model’s adaptability and performance across diverse multimodal tasks.

3.2. Text Representation

Effective text representation is pivotal for classification tasks leveraging machine learning techniques. Traditional text representation methods, such as n-gram models and bag-of-words approaches, rely on counting the frequency of word occurrences or sequences of characters. While these methods are straightforward and computationally efficient, they often fail to capture the nuanced relationships between words and their contextual usage within the text. This limitation hampers the model’s ability to understand semantic and syntactic nuances essential for accurate classification.

To address these shortcomings, more advanced representation techniques have been developed. Notably, Bengio et al. [16] introduced a neural network-based language model (NNLM) capable of learning distributed representations of words that encapsulate contextual information. Building upon this foundation, the word2vec model [46] emerged as a simplified yet powerful unsupervised learning algorithm that generates vector representations for words based on their contextual surroundings. The word2vec model leverages large corpora of unlabeled text to learn these embeddings, capturing both semantic and syntactic relationships through vector arithmetic operations.

In this study, we evaluated three distinct text representation strategies:

n-gram 

Inspired by the methodology proposed by Kanaris Stamatatos [33], we employed the n-gram approach for text representation. Despite its simplicity, the n-gram model serves as a robust baseline, effectively capturing local word dependencies and sequences within the text data.

Word2Vec 

Word2Vec is an unsupervised learning framework that generates dense vector representations for words by analyzing their contextual co-occurrences [46]. These vector embeddings are capable of capturing complex semantic and syntactic relationships, enabling the model to perform operations such as word analogies through vector arithmetic. In our approach, each movie is represented by the average of the word vectors corresponding to the words in its plot outline. This averaging process leverages the additive compositionality property of word2vec, where the combined representation retains meaningful semantic information. By averaging rather than summing the vectors, we mitigate the risk of excessively large input values, thereby maintaining numerical stability during subsequent neural network processing.

Recurrent Neural Network 

For a more context-aware representation, we explored the use of Recurrent Neural Networks (RNNs) to model the sequential nature of textual data. Specifically, we investigated two variants:

• RNN_w2v: This variant employs transfer learning by utilizing pre-trained word2vec vectors as input embeddings. The RNN processes the sequence of word vectors, capturing temporal dependencies and contextual information to generate a comprehensive representation of the plot outline.
• RNN_end2end: In contrast, this variant learns word embeddings from scratch in an end-to-end manner, allowing the RNN to optimize the embeddings jointly with the classification task. This approach enables the model to tailor the embeddings specifically to the genre classification objective, potentially enhancing performance by capturing task-relevant features.

Each of these text representation methods offers unique advantages, and their comparative effectiveness is evaluated within our experimental framework to determine the most suitable approach for multimodal genre classification.

3.3. Visual Representation

In the domain of computer vision, Convolutional Neural Networks (CNNs) have established themselves as the cornerstone for visual representation learning. CNNs excel at automatically extracting hierarchical feature representations from raw image data, making them indispensable for a wide array of vision-related tasks. A key characteristic of CNNs is their ability to leverage large-scale datasets to learn transferable features that generalize well across different domains, a property that is extensively utilized in transfer learning paradigms.

For visual representation in this study, we explored two primary strategies:

VGG Transfer 

This approach leverages the VGG Network [53], a deep CNN architecture renowned for its performance on the ImageNet dataset. By utilizing the pre-trained VGG model as a feature extractor, we extract the activations from the last hidden layer as the visual representation for each movie poster. This transfer learning strategy capitalizes on the rich feature representations learned from extensive image data, enabling effective utilization of visual information without the need for training a network from scratch.

End-to-End CNN 

In contrast to the transfer learning approach, the end-to-end CNN strategy involves training a custom CNN architecture from the ground up, tailored specifically to our dataset and classification task. Our architecture comprises five convolutional layers designed to progressively extract higher-level features from the input images, followed by a Multi-Layer Perceptron (MLP) that serves as the classifier. This end-to-end training allows the CNN to learn feature representations that are highly specialized and optimized for the task of genre classification based on poster imagery.

By evaluating both transfer learning and end-to-end training methodologies, we aim to ascertain the most effective approach for capturing and representing visual features pertinent to movie genre classification.

3.4. Classification Model

Following the extraction and representation of multimodal data, the next critical step involves mapping these feature vectors to their corresponding genre labels. To achieve this, we explored two distinct classification methodologies: Logistic Regression and a more complex Neural Network architecture.

Logistic Regression 

As a baseline classification approach, we employed Logistic Regression, a well-established statistical method for binary and multiclass classification tasks. Logistic Regression models the probability of each genre label given the input feature vectors, making it a straightforward yet effective choice for multilabel classification scenarios.

Neural Network Architecture 

To harness the expressive power of deep learning, we implemented a Multilayer Perceptron (MLP) with two fully connected layers, incorporating the Maxout activation function. The Maxout activation function, defined as:

$$h_i\left(\mathbf{s}\right)=\underset{j\in \left[1,k\right]}{max}{z}_{i,j}$$

(1)

where $\mathbf{s}\in {\mathbb{R}}^n$ is the input vector, $z_{i,j}={\mathbf{s}}^T{\mathbf{W}}_{\cdots ij}+{\mathbf{b}}_{ij}$ represents the output of the j-th linear transformation for the i-th hidden unit, and $\mathbf{W}\in {\mathbb{R}}^{d\times m\times k}$ and $\mathbf{b}\in {\mathbb{R}}^{m\times k}$ are the learnable parameters, offers several advantages. Maxout networks have been demonstrated to act as universal approximators with as few as two hidden units, providing the capability to model complex, non-linear functions [24]. Additionally, the Maxout activation function mitigates the issue of unit saturation, allowing for more stable and efficient training dynamics. By incorporating Maxout into the MLP, we aim to enhance the model’s capacity to capture intricate patterns and relationships within the multimodal feature space, thereby improving classification performance.

Both classification models were rigorously evaluated to determine their efficacy in mapping the rich, multimodal feature representations to accurate genre predictions. The comparative analysis between Logistic Regression and the Maxout-based MLP provides insights into the trade-offs between simplicity and expressiveness in the context of multilabel genre classification.

4. Experiments

4.1. Dataset

In this study, we introduce the Multimodal IMDb (MM-IMDb)1 dataset, which will be made accessible to the public. The MM-IMDb dataset was meticulously constructed by leveraging IMDb identifiers sourced from the Movielens 20M dataset2, encompassing ratings for approximately $27,000$ films. Utilizing the IMDbPY library3, we systematically excluded any movies that lacked corresponding poster images, ensuring that each entry in the final dataset is complete with essential visual and textual information. Consequently, the MM-IMDb dataset comprises $25,959$ movies, each annotated with detailed plot summaries, poster images, genre classifications, and an extensive collection of 50 additional metadata fields. These metadata include attributes such as the release year, language, writer, director, aspect ratio, and more, providing a rich context for each film.

It is important to note that films can be classified under multiple genres simultaneously. The co-occurrence matrix of genre tags illustrates the frequency with which different genres appear together, highlighting the interconnected nature of film classifications. The distribution of movie poster dimensions and the lengths of plot summaries are also characterized, with poster sizes varying and plot lengths averaging $92.5$ words. The longest plot summary in the dataset contains $1,431$ words, and on average, each movie is associated with $2.48$ genres. The primary task defined in this work is the prediction of movie genres based on plot summaries and poster images. However, the rich metadata available opens avenues for additional tasks, such as predicting movie ratings and enabling content-based retrieval systems.

4.2. Experimental Configuration

The MM-IMDb dataset was divided into three distinct subsets to facilitate model training and evaluation. Specifically, the training set comprises $15,552$ samples, the development set includes $2,608$ samples, and the test set consists of $7,799$ samples. This distribution ensures that the model is trained on a substantial portion of the data while retaining adequate samples for validation and testing. The distribution of genre tags across these subsets is detailed in Table 1. To maintain a balanced representation of genres, the dataset was stratified such that the training, development, and test sets contain $60\%$, $10\%$, and $30\%$ of samples for each genre, respectively.

Evaluation Metrics

Multilabel classification introduces complexities beyond those found in traditional multiclass classification, particularly in the realm of performance evaluation. Various metrics can yield significantly different insights into model performance [42]. In this context, we report four distinct averages of the f-score ($f_1$):

• Sample-based Average ($f_1^{sample}$): This metric calculates the f-score for each individual sample and subsequently averages these scores across all samples.
• Micro-average ($f_1^{micro}$): This approach aggregates the contributions of all classes to compute the f-score globally by considering all true positives, false positives, and false negatives.
• Macro-average ($f_1^{macro}$): Here, the f-score is computed independently for each genre and then averaged, treating all genres equally regardless of their prevalence.
• Weighted Macro-average ($f_1^{weighted}$): Similar to the macro-average, but each genre’s f-score is weighted by the number of true instances it has, providing a balance between rare and common genres.

The precise mathematical formulations for these metrics are as follows [42]:

$$\begin{array}{ccc}\hfill f_1^{sample}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{2\times \left|{\widehat{y}}_i\cap y_i\right|}{\left|{\widehat{y}}_i\right|+\left|y_i\right|}}& f_1^{macro}={\displaystyle \frac{1}{Q}}\sum _{j=1}^Q{\displaystyle \frac{2\times {\mathrm{p}}_j\times {\mathrm{r}}_j}{{\mathrm{p}}_j+{\mathrm{r}}_j}}& f_1^{weighted}={\displaystyle \frac{1}{Q^2}}\sum _{j=1}^Q{Q}_j{\displaystyle \frac{2\times {\mathrm{p}}_j\times {\mathrm{r}}_j}{{\mathrm{p}}_j+{\mathrm{r}}_j}}\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\mathrm{p}}^{micro}={\displaystyle \frac{{\sum }_{j=1}^{Q}t{p}_j}{{\sum }_{j=1}^{Q}t{p}_j+{\sum }_{j=1}^{Q}f{p}_j}}& {\mathrm{r}}^{micro}={\displaystyle \frac{{\sum }_{j=1}^{Q}t{p}_j}{{\sum }_{j=1}^{Q}t{p}_j+{\sum }_{j=1}^{Q}f{n}_j}}& f_1^{micro}={\displaystyle \frac{2\times {\mathrm{p}}^{micro}\times {\mathrm{r}}^{micro}}{{\mathrm{p}}^{micro}+{\mathrm{r}}^{micro}}}\hfill \end{array}$$

where:

• N denotes the total number of samples.
• Q represents the total number of genres.
• $Q_j$ is the count of true instances for the j-th genre.
• ${\mathrm{p}}_j$ and ${\mathrm{r}}_j$ are the precision and recall for the j-th genre, respectively.
• ${\widehat{y}}_i$ and $y_i$ are the predicted and true binary label vectors for the i-th sample.
• $t{p}_j$, $f{p}_j$, and $f{n}_j$ correspond to the true positives, false positives, and false negatives for the j-th genre.

Textual Feature Representation

For the textual data, we utilized the pre-trained Google Word2vec embeddings4 to represent words in a dense vector space. By intersecting the Word2vec vocabulary with the words present in the MM-IMDb plot summaries, we established a final vocabulary comprising $41,612$ unique terms. Prior to embedding, all text was converted to lowercase, and no additional preprocessing steps were applied, preserving the original linguistic structures. To assess the impact of network depth on model performance, we also evaluated a simplified architecture featuring a single fully connected layer.

To benchmark the effectiveness of our textual representation, we conducted evaluations on two publicly available datasets: the 7genre dataset, which contains $1,400$ web pages classified into 7 distinct genres, and the ki-04 dataset, encompassing $1,239$ samples across 8 genres. Our model was compared against state-of-the-art results presented by Kanaris Stamatatos [33], who employed character n-grams combined with structured HTML tag information for genre prediction in web pages.

Visual Feature Representation

For visual data, our initial approach involved utilizing the VGG network as a feature extractor, referred to as VGG_Transfer. This method leverages the deep features learned by the VGG architecture, which are then used for downstream tasks without further modification. The second approach involves processing raw images directly through a Convolutional Neural Network (CNN). Given the variability in image sizes, all poster images were standardized by scaling and cropping them to a resolution of $160\times 256$ pixels while maintaining the original aspect ratio. The CNN architecture comprises five convolutional layers with filter sizes of $5\times 5$, $3\times 3$, $3\times 3$, $3\times 3$, and $3\times 3$, respectively, each followed by a pooling layer with a $2\times 2$ window. Each convolutional layer contains 16 hidden units. The convolutional layers are subsequently connected to the MaxoutMLP classifier, which aggregates the extracted features for genre prediction.

Multimodal Feature Integration

To effectively combine textual and visual modalities, we explored four distinct fusion strategies as baselines:

Average Probability (Late Fusion) 

This strategy involves averaging the probabilities output by the best-performing model for each modality and then applying a threshold to determine the final genre predictions.

Concatenation 

Building on findings from prior research [35,51,57], we concatenated the feature representations from both modalities and fed the combined vector into the MaxoutMLP architecture for classification.

Linear Sum 

Inspired by the approach of Vinyals et al. [60], this method applies a linear transformation to each modality’s representation to align their dimensionalities before summing them. The resultant vector is then processed by the MaxoutMLP classifier.

Mixture of Experts (MoE) 

We adapted the MoE model [31] for multilabel classification by exploring two gating mechanisms: tied gating, where a single gate influences all logistic outputs, and untied gating, where each logistic output has its own dedicated gate. Both logistic regression and MaxoutMLP were evaluated as expert models within this framework.

Neural Network Training Procedures

All neural network models were trained using the Batch Normalization technique [28], which normalizes the inputs of each layer across the mini-batch, ensuring that each hidden unit maintains a zero mean and unit variance. This approach facilitates faster training and more stable convergence. We employed Stochastic Gradient Descent (SGD) with the ADAM optimization algorithm [36] to update the network weights. To mitigate overfitting, dropout regularization and max-norm constraints were applied.

The hyperparameters explored during training included hidden layer sizes ($\{64,128,256,512\}$), learning rates ($[{10}^{-3},{10}^{-1}]$), dropout rates ($[0.3,0.7]$), max-norm values ($[5,20]$), and weight initialization ranges ($[{10}^{-3},{10}^{-1}]$). We trained 25 models with randomly initialized hyperparameters, selecting the best-performing model based on validation set performance. This random search strategy is advantageous over grid search, particularly for training deep models, as it more efficiently explores the hyperparameter space [17]. All implementations were carried out using the Blocks framework [59]5.

During training, Batch Normalization significantly accelerated the training process and improved convergence rates, reducing the model’s sensitivity to hyperparameter settings such as learning rates and initialization ranges. Additionally, the incorporation of dropout and max-norm regularization techniques contributed to enhanced generalization performance on the test set.

5. Experimental Results

5.1. Performance on Synthetic Data

To assess the capability of our model to discern the modality that provides the most informative features for classification, we devised a synthetic task based on a generative model. This model defines a binary target variable C and two input feature vectors $x_v$ and $x_t$, each residing in ${\mathbb{R}}^2$. A latent binary variable M determines which modality—visual or textual—contains the relevant information for classifying the sample.

The generative process is defined as follows:

$$\begin{array}{cc}\hfill C& \sim \mathrm{Bernoulli}\left(p_C\right)\hfill \\ \hfill M& \sim \mathrm{Bernoulli}\left(p_M\right)\hfill \\ \hfill y_v& \sim \mathcal{N}\left({\gamma }_v^C\right)\hfill \\ \hfill {\widehat{y}}_v& \sim \mathcal{N}\left({\widehat{\gamma }}_v\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill x_v& =M{y}_v+(1-M){\widehat{y}}_v\hfill \\ \hfill y_t& \sim \mathcal{N}\left({\gamma }_t^C\right)\hfill \\ \hfill {\widehat{y}}_t& \sim \mathcal{N}\left({\widehat{\gamma }}_t\right)\hfill \\ \hfill x_t& =M{\widehat{y}}_t+(1-M)y_t\hfill \end{array}$$

In this setup:

• C is the binary class label.
• M decides which modality holds the class-informative features.
• $y_v$ and $y_t$ are class-dependent features drawn from Gaussian distributions centered at ${\gamma }_v^C$ and ${\gamma }_t^C$, respectively.
• ${\widehat{y}}_v$ and ${\widehat{y}}_t$ are noise features drawn from Gaussian distributions centered at ${\widehat{\gamma }}_v$ and ${\widehat{\gamma }}_t$, respectively.
• The input features $x_v$ and $x_t$ are composites of either informative or noise features based on the value of M.

We trained a single FGU model with a sigmoid activation function applied to the hidden state h, optimizing it using binary cross-entropy loss. For each experiment, we generated 200 samples per class, and conducted $1,000$ independent trials with varying random seeds. The FGU model outperformed a logistic regression classifier in 370 instances and matched its performance in the remaining trials. This demonstrates the FGU’s ability to effectively learn the latent variable M that dictates the relevance of each modality for classification tasks.

Moreover, we investigated the relationship between the gate activations z and the latent variable M. The analysis revealed a perfect correlation of 1, indicating that the FGU successfully inferred the modality relevance solely from the input features $x_v$ and $x_t$.

To visualize how the gate activations z correspond to different regions of the feature space, we projected z back onto the input feature dimensions in a synthetic experiment where $x_v,x_t\in \mathbb{R}$. The left plot illustrates the regions where $z>0.5$ (indicating a preference for the visual modality) and $z\le 0.5$ (indicating a preference for the textual modality). The right plot shows the model’s predictions, confirming that the gating mechanism effectively isolates noise and maintains decision boundaries consistent with modality relevance.

5.2. Genre Classification Performance

Prior to integrating our textual representation into the multimodal framework, we validated its efficacy on separate classification tasks using two public datasets. The MaxoutMLP_w2v model achieved state-of-the-art performance on the ki-04 dataset and improved the f-score on the 7Genre dataset from $0.841$ to $0.854$ compared to the baseline presented by Kanaris Stamatatos [33]. It is noteworthy that the baseline method utilized additional HTML structural information from web pages, whereas our representation relies solely on textual data, highlighting its robustness and effectiveness.

Table 2 presents a comprehensive overview of the classification performance across different modalities and representations. For the textual modality, the highest performance was achieved using the MaxoutMLP_w2v representation, indicating the effectiveness of Word2vec embeddings combined with a multilayer perceptron classifier. The performance trends were consistent across all evaluation metrics. In contrast, models trained from scratch, such as the RNN-based architectures, exhibited inferior performance, likely due to insufficient data to capture meaningful word relationships. This observation underscores the importance of leveraging pre-trained embeddings for tasks with limited labeled data.

Regarding the visual modality, models utilizing pre-trained networks like VGG_Transfer outperformed those trained end-to-end, suggesting that the dataset size is insufficient for learning complex visual features from scratch. This aligns with the general understanding that deep visual models require large amounts of data to generalize effectively.

5.3. Detailed Genre-wise Analysis

To gain deeper insights into model performance, we conducted a genre-wise evaluation, as detailed in Table 3. The table compares the macro f-scores for single-modality approaches (textual and visual) against the multimodal FGU approach across various genres.

The FGU-based multimodal approach consistently outperformed single-modality models across the majority of genres, demonstrating the efficacy of integrating textual and visual information. Notably, in genres such as Animation and Family, the visual modality alone sometimes surpassed the textual representation, underscoring the significance of visual features in these categories. Conversely, genres like Biography and War showed substantial improvements when both modalities were combined, highlighting the complementary nature of textual and visual data.

5.4. Analysis of Modality Influence

To further understand the contribution of each modality to the FGU model’s predictions, we analyzed the gate activations (z) across test samples. Specifically, we examined the proportion of samples for which the model predominantly relied on the textual modality ($z\le 0.5$) versus the visual modality ($z>0.5$) when assigning genre labels, which illustrates the percentage distribution of gate activations corresponding to each genre.

As anticipated, the textual modality generally plays a more significant role in genre prediction. However, certain genres like Animation and Family exhibit a higher reliance on visual features, aligning with their inherent visual characteristics. This observation is consistent with the genre-wise performance results, where the visual modality demonstrated superior performance in these categories. The gating mechanism effectively identifies and leverages the most informative modality for each genre, enhancing overall classification performance.

5.5. Qualitative Analysis of Predictions

To provide a qualitative perspective on the FGU model’s performance, we examined specific test examples where the model exhibited notable improvements in genre prediction. Table 4 showcases instances where the FGU successfully utilized the most accurate modality, thereby eliminating false positives and enhancing prediction accuracy. These examples highlight the model’s ability to integrate and prioritize information from both textual and visual sources effectively.

These examples demonstrate how the FGU model effectively integrates textual and visual information to produce accurate genre classifications. In cases where the textual modality alone may introduce false positives, the incorporation of visual data allows the model to refine its predictions, ensuring that only the most relevant genres are assigned. This qualitative analysis underscores the practical benefits of multimodal integration in complex classification tasks.

6. Conclusions and Future Directions

In this study, we introduced an innovative approach for learning fusion transformations from multiple data modalities. Drawing inspiration from the mechanisms employed by recurrent neural networks to regulate information flow, our proposed model leverages multiplicative gating mechanisms. The Gated Multimodal Unit (FGU) is designed to accept two or more distinct input sources and autonomously learn the extent to which each modality influences the activation of the unit. Through a series of synthetic experiments, the FGU demonstrated its capability to uncover and utilize hidden latent variables effectively. Furthermore, when applied to real-world scenarios, the FGU consistently outperformed models that relied solely on single modalities, highlighting the advantages of multimodal integration.

A notable attribute of the FGU is its differentiable nature, which allows it to be seamlessly integrated into a wide array of neural network architectures. This compatibility ensures that the FGU can be trained using standard gradient-based optimization techniques, facilitating its adoption in diverse applications. The flexibility and robustness of the FGU make it a valuable component for enhancing the performance of complex neural networks dealing with heterogeneous data sources.

In addition to the methodological advancements, this work also contributes to the research community by releasing a comprehensive new dataset. This dataset encompasses approximately $27,000$ movie plots, accompanying images, and a wealth of metadata, making it the largest of its kind for the task of movie genre classification based on multimodal information. To our knowledge, this is the first publicly available dataset that amalgamates such a substantial volume of textual and visual data for this specific application, providing a valuable resource for future research and development in the field.

Looking ahead, our future work aims to explore deeper and more sophisticated architectures of FGU layers. By stacking multiple FGU layers, we anticipate capturing more intricate interactions between different modalities, thereby further enhancing the model’s ability to learn complex fusion transformations. Additionally, we plan to integrate attention mechanisms into the FGU framework. Attention mechanisms have shown great promise in allowing models to focus on the most relevant parts of the input data, and their incorporation could enable the FGU to dynamically prioritize information from different modalities based on the context of each input instance.

Another promising direction for future research involves delving deeper into the interpretability of the features learned by the FGU. Understanding how the FGU weighs and combines different modalities can provide valuable insights into the decision-making processes of multimodal models. This transparency is crucial for applications where explainability is essential, such as in recommendation systems or automated content moderation.

Moreover, we intend to extend the application of the FGU beyond movie genre classification to other domains that inherently involve multimodal data. Potential areas of application include healthcare, where combining medical images with patient records can lead to more accurate diagnoses, and autonomous driving, where integrating visual data from cameras with sensor information can enhance vehicle perception systems.

In summary, the FGU presents a robust and flexible framework for multimodal fusion, demonstrating superior performance over single-modality models in both synthetic and real-world settings. The release of the MM-IMDb dataset further empowers researchers to explore and expand upon our work. By pursuing the outlined future directions, we aim to advance the capabilities of multimodal learning systems, fostering the development of more intelligent and context-aware applications.