BiMLI: Bidirectional Link–Modality Interaction for Link Prediction in Attribute-Based Multimodal Knowledge Graphs

1. Introduction

Knowledge graphs (KGs) organize entities and their relations in the form of factual triples, providing structured knowledge for applications such as recommender systems, natural language understanding, and knowledge augmentation for large language models (LLMs) [1,2,3,4,5,6]. For example, existing studies have integrated KGs with LLMs to improve the factual consistency and trustworthiness of conversational AI systems; other studies have employed LLMs to extract entities, relations, and temporal information from heterogeneous unstructured data, thereby automatically constructing domain-specific temporal knowledge graphs [7,8]. However, owing to the limited coverage of data sources and the costs associated with knowledge extraction, verification, and maintenance, real-world KGs are typically incomplete. Knowledge graph completion (KGC) aims to predict missing entities or relations from observed facts. A related survey systematically reviews translation-based, semantic-matching, and neural-network-based knowledge graph embedding methods, and compares their representation capabilities, computational efficiency, and use of auxiliary information [9]. Methods such as TransE, DistMult, ComplEx, RotatE, GC-OTE, and TuckER primarily learn entity and relation representations from graph structure [10,11,12,13,14,15] and have achieved strong performance on link prediction. However, the learning signals of these structural models are mainly derived from observed triples, preventing them from directly exploiting the images and textual descriptions associated with entities. When structural evidence is insufficient to distinguish among candidate entities, visual and textual information can provide complementary semantic cues.

Attribute-based multimodal knowledge graphs associate entities with images and textual descriptions as attributes, enabling link prediction models to jointly exploit graph structure and multimodal semantic information [16,17]. Early multimodal knowledge graph completion (MMKGC) methods primarily investigated visual feature representation, cross-modal mapping, and joint embedding [18,19,20,21,22], whereas recent studies have further explored interactive fusion, modality adaptation, fine-grained representation, mixture-of-experts learning, and non-Euclidean modeling [23,24,25,26,27,28]. Beyond these studies centered on multimodal entity embeddings, MLKGC further combines LLMs with multimodal modules that process images and audio, thereby providing additional contextual information for missing-fact inference [29]. This direction focuses on enhancing candidate-fact inference through LLMs and external multimodal inputs, and therefore differs from the internal representation learning problem in attribute-based multimodal knowledge graphs considered in this work. Overall, studies centered on entity attribute representation indicate that visual and textual attributes can provide complementary semantics for structural representations. In particular, modality-adaptive methods further suggest that the contributions of different modalities are not fixed but may vary with the entity and relational context [23,27]. Therefore, multimodal attributes should not be treated merely as auxiliary information to be fused statically; their relevance should also be assessed in relation to the local facts surrounding each entity.

To improve the contextual specificity of multimodal representation learning, several studies have further incorporated relational semantics, local links, or subgraph structures. In this paper, we define a local link as a one-hop factual triple directly connected to a central entity in the training graph. LAFA employs link information to guide visual feature fusion and neighborhood aggregation, whereas TSAM, SAFER, and C2RS enhance multimodal representations from the perspectives of structural awareness, subgraph modeling, and relational semantics, respectively [30,31,32,33]. Despite their different mechanisms, these methods generally exploit relational semantics, local links, or subgraph structures to guide the selection, fusion, and aggregation of multimodal representations. In link-aware methods in particular, information primarily flows from links to modalities; that is, the link context is used to determine which modality provides more relevant evidence. By contrast, how to condition on multimodal representations to perform fine-grained comparison and weighting of the different local links surrounding a central entity remains insufficiently explored. These two problems involve different conditioning variables and selection spaces: the former selects semantic evidence in the modality space, whereas the latter selects topological evidence in the local-link space. Consequently, one cannot naturally replace the other through a single unidirectional selection operation. The central question addressed in this work is therefore not merely how to fuse multiple modalities, but how to jointly model link-conditioned modality selection and modality-conditioned link selection, thereby capturing the bidirectional conditional dependencies between multimodal semantics and local topology.

Figure 1 illustrates this problem using the football player Paul Konchesky as an example. The entity is associated with visual and textual attributes and is connected to clubs, countries, and cities through relations such as playsFor, isCitizenOf, and wasBornIn. Its textual description contains semantic information about his professional career, clubs, and nationality; its visual attributes depict the person and football-related scenes; and its local links provide topological facts with explicit relational meanings. For a local link of type playsFor, football-related visual content and club information in the textual description may provide more direct semantic evidence. For isCitizenOf or wasBornIn, textual information concerning nationality or birthplace, together with the structural context, may be more useful. Conversely, given club-related semantics in the textual representation, the playsFor link connecting the entity to a club usually provides more relevant topological support than a link concerning birthplace. This example demonstrates that local links can guide modality-evidence selection, while modality semantics can, in turn, participate in local-link selection.

Figure 2 further decomposes this interaction into two directions. In the Link→Modality direction, each local link serves as the condition for selecting relevant evidence from the structural, visual, and textual representations, producing a modality-enhanced representation for that link. In the Modality→Link direction, each modality representation serves as the condition for comparing and weighting the local links surrounding the central entity, producing a link-enhanced representation for that modality. The two directions are not symmetric operations that become functionally equivalent when their inputs are exchanged. Link→Modality performs selection along the modality dimension and answers the question, “Which modalities does a link require?” Modality→Link performs selection along the local-link dimension and answers, “Which links does a modality require?” The two directions therefore have distinct conditional semantics and selection spaces and cannot be naturally replaced by a single unidirectional attention operation.

Figure 3 presents empirical observations of the different contributions made by the two interaction directions. On MKG-W, retaining only Link→Modality yields an MRR of 42.9% and a Hits@1 score of 35.5%. Retaining only Modality→Link produces corresponding scores of 37.6% and 30.1%. When the two directions are combined, the complete BiMLI model achieves 43.5% and 36.1%, respectively. The same overall trend is observed on DB15K. The performance of Link→Modality is closer to that of the complete model, indicating that link-conditioned modality selection is the primary source of performance gains. Adding Modality→Link on top of this component yields further improvements, showing that modality-conditioned link selection provides complementary information. Thus, the two interaction directions are not symmetric modules with equal contributions; instead, they exhibit asymmetric complementarity in semantic-evidence selection and local-topology selection.

Based on the above analysis, we propose BiMLI (Bidirectional Link–Modality Interaction), a bidirectional link–modality interaction framework for link prediction in Attribute-Based Multimodal Knowledge Graphs. BiMLI first encodes the structural, visual, and textual information of entities and represents the local one-hop facts connected to each entity in the training graph. Link-conditioned modality interaction then uses each local link as a query to select relevant evidence from different modalities, producing a modality-enhanced link representation. Modality-conditioned link interaction uses each modality representation as a query to dynamically weight local links, producing a link-enhanced modality representation. Interaction-aware feature fusion and neighborhood aggregation subsequently integrate the representations produced in both directions, and the aggregated evidence is used to update entity representations. The model additionally employs a cross-modal contrastive constraint to facilitate the joint optimization of different modalities [34,35] and finally uses a RotatE decoder to compute scores for candidate triples [12].

The main contributions of this work are summarized as follows:

• We investigate the interaction between multimodal attributes and local links from the perspective of conditional selection. We explicitly distinguish between link-conditioned modality selection and modality-conditioned link selection and show that the two directions are functionally asymmetric yet complementary.
• We propose BiMLI, a bidirectional link–modality interaction framework that jointly models multimodal semantic evidence and local topological evidence through link-conditioned modality interaction, modality-conditioned link interaction, interaction-aware feature fusion, and neighborhood aggregation.
• We conduct extensive link prediction experiments on DB15K and MKG-W. BiMLI achieves MRR scores of 42.35% and 43.55%, corresponding to relative improvements of 6.43% and 11.98% over the strongest respective baselines. Ablation studies and fine-grained analyses further demonstrate the distinct roles and asymmetric complementarity of the two interaction directions.

2. Related Work

2.1. Knowledge Graph Completion

Knowledge graph completion aims to predict missing entities or relations based on observed triples. Translation-based methods, represented by TransE, characterize the geometric transformation between head and tail entities using relation vectors [15]. Semantic matching methods, such as DistMult, ComplEx, and TuckER, model entity–relation interactions through bilinear operations or tensor factorization [10,11,14]. RotatE and GC-OTE capture relation patterns such as symmetry, antisymmetry, inversion, and composition through rotations in complex space and orthogonal transformations, respectively [12,13]. These methods lay the foundation for link prediction; however, their learning signals are derived primarily from observed structured triples, preventing them from directly exploiting the visual and textual attributes associated with entities. Building on these approaches, multimodal knowledge graph completion incorporates visual and textual semantics to provide complementary information for entity representation and candidate ranking when structural evidence is insufficient.

2.2. Multimodal Knowledge Graph Completion

This work focuses on attribute-based multimodal knowledge graph completion, in which images and textual descriptions serve as entity attributes for link prediction. Early methods primarily investigated the joint modeling of visual features and structural embeddings. IKRL and MMKRL incorporated entity images into knowledge representation learning [18,19], while RSME, VBKGC, and OTKGE further explored relation-sensitive modality gating, pretrained multimodal representations, and optimal-transport-based cross-modal alignment, respectively [20,21,22]. These methods demonstrated the complementary value of visual and textual attributes for structural representations, primarily focusing on entity-level multimodal information aggregation, cross-modal interaction, and joint representation learning. Although methods such as RSME adapt modality information according to the relational context, the local one-hop facts surrounding a central entity are generally not explicitly represented as a set of links over which conditional selection can be performed.

Beyond the aforementioned methods centered on entity embeddings and multimodal representation learning, recent studies have also explored LLM-driven multimodal knowledge graph completion. MLKGC constructs query-related candidate triple sets and combines the semantic reasoning capabilities of LLMs with multimodal features, including images and audio, to provide additional contextual information for missing-fact prediction and long-tail entities [29]. This method primarily focuses on LLM-driven candidate-fact inference and external multimodal augmentation, without explicitly modeling the bidirectional conditional selection between the set of local links and the structural, visual, and textual representations in attribute-based multimodal knowledge graphs. Within the representation learning paradigm for attribute-based multimodal knowledge graphs, recent studies have further investigated the granularity and adaptability of multimodal interactions. IMF integrates structural, visual, and textual representations through an interactive fusion mechanism [24]. AdaMF dynamically adjusts fusion weights according to the modality information of each entity [27]. MyGO enhances entity semantic modeling using fine-grained modality tokens [28]. NativE employs relation-guided adaptive fusion to address modality diversity and distributional imbalance [23]. MoMoK learns complementary representations through multiple modality-specific knowledge experts [25], while MCKGC jointly models multimodal semantics and graph structure in a mixed-curvature space [26]. These methods mainly investigate how to encode, align, and fuse different modalities, demonstrating that modality contributions may vary across entities or relational contexts. However, their adaptive selection generally operates over modalities, tokens, experts, or representation spaces. The effects of local facts are typically encoded implicitly through structural representations or training objectives, rather than by explicitly weighting a set of local links conditioned on each modality. From the perspective of this work, these methods have not explicitly distinguished between two problems involving different selection spaces: selecting, along the modality dimension, the evidence required by a local link, and selecting, along the local-link dimension, the structural support required by a particular modality.

2.3. Structure- and Relation-Aware Multimodal Learning

To improve the adaptability of multimodal representations to relational contexts and local structures, several MMKGC methods have further incorporated relation, link, or subgraph information. LAFA selects visual features according to link contexts and combines them with neighborhood aggregation to learn entity representations [30]. TSAM strengthens the guidance provided by graph structure through fine-grained modality-aware fusion and structure-aware constraints [31]. SAFER integrates local structures using subgraph-aware adaptive fusion and hierarchical relation modeling [32]. C2RS jointly employs cross-modal consistency learning and relational semantic encoding to enhance entity and relation representations [33]. In addition, DM-MKGC improves multimodal fusion through dynamic prompt learning and multi-granularity aggregation [36]. Another line of research mainly optimizes multimodal representations from the perspective of alignment and consistency. For example, KG-MRI employs triple-level contrastive learning to integrate multimodal attributes with knowledge graph embeddings [37], whereas ROAD improves cross-modal representations through intra-modal and inter-modal consistency constraints [38]. These methods focus on structure-guided fusion, relation modeling, multi-granularity aggregation, or cross-modal alignment, and therefore do not share exactly the same modeling objective.

Relation-aware and structure-aware methods typically use relation representations, global graph structure, or local subgraphs to adapt multimodal representations. In methods such as LAFA that directly exploit local links, the explicit conditioning direction primarily proceeds from links to modalities; that is, relevant modality information is selected and fused according to the link context. By contrast, at the level of explicit conditional selection, relatively little attention has been paid to conditioning on each modality representation to compare individual local links surrounding a central entity and assign them normalized weights. Existing methods can implicitly exchange modality and structural information through representation fusion and joint training, but they generally do not separately parameterize a modality-conditioned selection distribution over the local-link dimension. Consequently, link-conditioned modality selection and modality-conditioned link selection have yet to be explicitly distinguished and jointly modeled.

Motivated by this distinction, BiMLI treats entity-level structural, visual, and textual representations and link-level local one-hop facts as two sets of evidence that can condition on each other. Link→Modality conditions on each local link to select relevant multi-view evidence along the modality dimension, whereas Modality→Link conditions on each modality representation to select relevant local structural evidence along the local-link dimension. The two directions involve different conditioning variables, normalization spaces, and representation objectives. They therefore constitute functionally distinct yet complementary selection processes rather than symmetric operations with equal contributions. BiMLI integrates the representations produced in both directions through interaction-aware feature fusion and neighborhood aggregation. Unlike methods whose primary objective is cross-modal alignment, the contrastive constraint in BiMLI serves only as an auxiliary regularizer; its core objective is to jointly model link-conditioned modality selection and modality-conditioned link selection.

3. Problem Definition

In this section, we provide the necessary background and formally define the task of MMKGC.

A knowledge graph (KG) is denoted by $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$, where $\mathcal{E}$ and $\mathcal{R}$ denote the sets of entities (nodes) and relations (edges), respectively. Let $N^e=\left|\mathcal{E}\right|$ and $N^r=\left|\mathcal{R}\right|$ denote the numbers of entities and relations. The triple set of the knowledge graph is defined as

$$\mathcal{T}=\left\{(h,r,t)\mid h,t\in \mathcal{E},r\in \mathcal{R}\right\},$$

(1)

where $r\in \mathcal{R}$ represents the relation between the head entity h and the tail entity t. Let $N^{\mathrm{tr}}=\left|\mathcal{T}\right|$ denote the number of triples.

An attribute-based multimodal knowledge graph (MMKG) augments a conventional knowledge graph with textual and visual information. It can be formally represented as

$$\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T},\mathcal{V},\mathcal{D}),$$

(2)

where $\mathcal{V}$ and $\mathcal{D}$ denote the collections of images and textual descriptions associated with the entities in $\mathcal{E}$, respectively. In this work, we treat graph structure s as an independent modality. Accordingly, the representation of modality k is denoted by ${\mathbf{Z}}^k$, where

$$k\in \mathcal{K}=\{s,v,t\},$$

(3)

corresponding to the structural, visual, and textual modalities, respectively.

Link prediction in attribute-based multimodal knowledge graphs aims to predict the tail entity $(h,r,*)$ or the head entity $(*,r,t)$ of an incomplete triple by incorporating multiple types of entity-associated modality data, such as textual descriptions and images, into conventional structure-based knowledge graph representations. To simplify the task, we convert head-entity prediction into tail-entity prediction by introducing the inverse triple $(t,r^{-1},h)$. The model therefore focuses on tail-entity prediction, where $r^{-1}$ denotes the inverse relation of r.

4. Materials and Methods

We propose BiMLI, a Bidirectional Link–Modality Interaction framework designed to jointly exploit entity-level multimodal attributes and local structural information for link prediction in attribute-based multimodal knowledge graphs. The central idea is to treat entity-level structural, visual, and textual representations and link-level local one-hop facts as two sets of mutually conditioning evidence, over which information is selected along the modality and local-link dimensions, respectively. Given an attribute-based multimodal knowledge graph $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T},\mathcal{V},\mathcal{D})$, BiMLI first encodes the structural, visual, and textual information of entities, together with their local links in the training graph. Link→Modality then uses each local link as a query to select relevant evidence from the three modalities, producing a modality-enhanced link representation. Conversely, Modality→Link uses each modality representation as a query to dynamically weight the local links, producing a link-enhanced modality representation. Because the two directions perform selection along the modality and link dimensions, respectively, they constitute functionally distinct yet complementary interaction processes. The model further derives the final entity representations through interaction-aware feature fusion and neighborhood aggregation and employs a RotatE decoder for link prediction. A cross-modal contrastive loss is introduced as an auxiliary constraint to preserve the semantic consistency of the same entity across the structural, visual, and textual representation spaces. The overall framework is illustrated in Figure 4.

4.1. Multimodal and Link Encoding

BiMLI employs GAT  [39], VGG16  [40], and BERT  [41] to extract the structural, visual, and textual features of each entity, respectively, and maps them into a unified representation space through modality-specific projection layers. For a central entity $h_i$, the model retrieves the one-hop factual triples directly connected to it from the training graph. The representations of the central entity, relation, and neighboring entity are then concatenated and projected to obtain a local-link representation. Local links are constructed exclusively from training triples, while target links from the validation and test sets are excluded from the encoding process to prevent target information leakage.

Multimodal encoding. Let $\mathcal{K}=\{s,v,t\}$ denote the set of structural, visual, and textual modalities. We employ a GAT pretrained on the training graph  [39], a VGG16 pretrained on ImageNet  [40], and a pretrained BERT  [41] to extract features for the three modalities:

$$\begin{array}{cc}\hfill {\mathbf{Z}}^s& =GAT\left(g\right)\in {\mathbb{R}}^{N^e\times d_s},\hfill \\ \hfill {\mathbf{Z}}^v& =VGG16\left(\mathcal{V}\right)\in {\mathbb{R}}^{N^e\times d_v},\hfill \\ \hfill {\mathbf{Z}}^t& =BERT\left(\mathcal{D}\right)\in {\mathbb{R}}^{N^e\times d_t},\hfill \end{array}$$

(4)

where $d_s=256$, $d_v=4096$, and $d_t=768$ denote the dimensions of the structural, visual, and textual embeddings, respectively. All three pretrained encoders remain frozen during the training of BiMLI. To obtain a unified feature dimension, we apply a modality-specific linear projection to each modality $k\in \mathcal{K}$:

$${\overline{\mathbf{Z}}}^k={\mathbf{Z}}^k{\mathbf{W}}_k,{\mathbf{W}}_k\in {\mathbb{R}}^{d_k\times d_h}.$$

(5)

After projection, the modality representation of entity $h_i$ is denoted by ${\overline{\mathbf{z}}}_i^k\in {\mathbb{R}}^{d_h}$.

Link encoding. We initialize the entity and relation embedding matrices as $\mathbf{E}\in {\mathbb{R}}^{N^e\times d_e}$ and $\mathbf{R}\in {\mathbb{R}}^{2N^r\times d_e}$, respectively. The number of relations is doubled because inverse relations are introduced. For a central entity $h_i$, its neighborhood and local-link sets are defined as

$${\mathcal{N}}_i=\left\{t_j\in \mathcal{E}\mid (h_i,r_j,t_j)\in \mathcal{T},r_j\in \mathcal{R}\right\},$$

(6)

and

$${\mathcal{L}}_i=\left\{l_{ij}=(h_i,r_j,t_j)\in \mathcal{T}\mid t_j\in {\mathcal{N}}_i,r_j\in \mathcal{R}\right\},$$

(7)

respectively. The initial representation of a local link $l_{ij}$ is obtained by concatenating its three constituent elements:

$${\mathbf{l}}_{ij}=[{\mathbf{h}}_i\parallel {\mathbf{r}}_j\parallel {\mathbf{t}}_j]\in {\mathbb{R}}^{3d_e}.$$

(8)

The entity, relation, and local-link representations are then projected into a unified hidden space:

$$\overline{\mathbf{E}}=\mathbf{E}{\mathbf{W}}_e,\overline{\mathbf{R}}=\mathbf{R}{\mathbf{W}}_r,{\overline{\mathbf{l}}}_{ij}={\mathbf{l}}_{ij}{\mathbf{W}}_l,$$

(9)

where ${\mathbf{W}}_e\in {\mathbb{R}}^{d_e\times d_h}$, ${\mathbf{W}}_r\in {\mathbb{R}}^{d_e\times d_h}$, and ${\mathbf{W}}_l\in {\mathbb{R}}^{3d_e\times d_h}$. Equations (8) and (9) use only training triples. Target links from the validation and test sets are not included in ${\mathcal{L}}_i$, thereby preventing target information leakage.

4.2. Bidirectional Interaction of Modalities and Links

The multimodal representations of an entity and its local links exhibit many-to-many conditional dependencies. On the one hand, different local links may require different structural, visual, and textual evidence. For example, club-related links may rely more heavily on career information or football-related visual cues. On the other hand, a given modality is not equally relevant to all local links surrounding the central entity. For instance, birthplace semantics in the textual modality are more appropriately supported by geographic relations. Therefore, using only local links to select modalities cannot explicitly distinguish which local links are required by a given modality. Conversely, using only modalities to select links cannot organize the corresponding multimodal evidence for each individual link.

Motivated by this observation, BiMLI introduces two conditionally opposite yet complementary interaction processes. Link→Modality performs selection along the modality dimension to identify relevant multi-view evidence for each local link, whereas Modality→Link performs selection along the local-link dimension to identify relevant local structural support for each modality. Here, “bidirectional” refers to the existence of two opposite conditioning directions; it does not imply that they are inverse operations or contribute equally. Because the two directions differ in their query inputs, normalization spaces, and output granularities, they constitute functionally asymmetric yet complementary representation learning processes.

To capture correlations across different representation subspaces, both interaction directions employ multi-head scaled dot-product attention. Let H denote the number of attention heads and d the unified representation dimension. The dimension of each attention head is therefore

$$d_{head}={\displaystyle \frac{d}{H}}.$$

(10)

In the following, the superscript $\left(p\right)$ denotes the p-th attention head.

4.2.1. Link-Conditioned Modality Interaction

Different local links correspond to different relational semantics and therefore should not indiscriminately fuse all modality information associated with an entity. Link→Modality conditions on each local link to determine how much evidence the structural, visual, and textual representations provide for that link, thereby generating a multimodal representation tailored to its specific link context. This direction addresses the following question: given a local link, which modality evidence is more relevant?

Given a local link $l_{ij}$ of the central entity $h_i$, we use the link representation as the query and the three modality representations $\{{\overline{\mathbf{z}}}_i^s,{\overline{\mathbf{z}}}_i^v,{\overline{\mathbf{z}}}_i^t\}$ as both keys and values. After partitioning the link and modality representations into H subspaces, the relevance score between local link $l_{ij}$ and modality k under the p-th attention head is computed as

$$s_{ij,k}^{\left(p\right)}={\displaystyle \frac{{\left({\overline{\mathbf{l}}}_{ij}^{\left(p\right)}\right)}^{\top }{\overline{\mathbf{z}}}_i^{k,\left(p\right)}}{\sqrt{d_{head}}}}.$$

(11)

For each local link, the softmax operation is performed over the modality set $\mathcal{K}$:

$${\alpha }_{ij,k}^{\left(p\right)}={\displaystyle \frac{exp\left(s_{ij,k}^{\left(p\right)}\right)}{{\sum }_{n\in \mathcal{K}}exp\left(s_{ij,n}^{\left(p\right)}\right)}},\sum _{k\in \mathcal{K}}{\alpha }_{ij,k}^{\left(p\right)}=1.$$

(12)

The p-th attention head then aggregates the corresponding value vectors according to the modality weights:

$${\mathbf{o}}_{ij}^{\left(p\right)}=\sum _{k\in \mathcal{K}}{\alpha }_{ij,k}^{\left(p\right)}{\overline{\mathbf{z}}}_i^{k,\left(p\right)}.$$

(13)

Finally, the outputs of all attention heads are concatenated and passed through an output projection layer to obtain the modality-enhanced link representation:

$${\tilde{\mathbf{z}}}_{ij}=\varphi \left({\mathbf{W}}_O^{LM}\left[{\mathbf{o}}_{ij}^{\left(1\right)}\parallel {\mathbf{o}}_{ij}^{\left(2\right)}\parallel \dots \parallel {\mathbf{o}}_{ij}^{\left(H\right)}\right]\right),$$

(14)

where ${\mathbf{W}}_O^{LM}$ is the output projection matrix and $\varphi (·)$ denotes the activation or normalization operation adopted in the implementation. The multi-head mechanism enables different attention heads to capture distinct semantic subspaces within the structural, visual, and textual representations. The resulting representation ${\tilde{\mathbf{z}}}_{ij}$ preserves the modality-selection result associated with each local link and is subsequently used for link-level fusion and neighborhood aggregation. This process can be interpreted as “a link retrieving multimodal evidence.”

4.2.2. Modality-Conditioned Link Interaction

Link→Modality can determine which modalities are required by a given link, but it cannot identify the local facts from which a particular modality should obtain structural support. A central entity may be surrounded by multiple relations and neighboring entities, whose relevance to a given modality can vary substantially. Modality→Link therefore conditions on each modality representation to compare and weight the local links. This direction addresses the following question: given a modality representation, which local links can provide relevant structural evidence?

For modality k of entity $h_i$, the model uses ${\overline{\mathbf{z}}}_i^k$ as the query and the local-link set ${\mathcal{L}}_i$ as the keys and values. Under the p-th attention head, the relevance between modality k and local link $l_{ij}$ is computed as

$$r_{ij,k}^{\left(p\right)}={\displaystyle \frac{{\left({\overline{\mathbf{z}}}_i^{k,\left(p\right)}\right)}^{\top }{\overline{\mathbf{l}}}_{ij}^{\left(p\right)}}{\sqrt{d_{head}}}}.$$

(15)

For each entity, modality, and attention head, the softmax operation is performed over the local-link set of that entity:

$${\beta }_{ij,k}^{\left(p\right)}={\displaystyle \frac{exp\left(r_{ij,k}^{\left(p\right)}\right)}{{\sum }_{l_{in}\in {\mathcal{L}}_i}exp\left(r_{in,k}^{\left(p\right)}\right)}},\sum _{l_{ij}\in {\mathcal{L}}_i}{\beta }_{ij,k}^{\left(p\right)}=1.$$

(16)

In the implementation, head_indices identifies the central entity to which each local link belongs, and scatter_softmax performs normalization separately within the local-link set of each entity. This allows entities with different neighborhood sizes to be processed within the same batch. The p-th attention head then computes a weighted sum of the local-link representations:

$${\mathbf{u}}_{i,k}^{\left(p\right)}=\sum _{l_{ij}\in {\mathcal{L}}_i}{\beta }_{ij,k}^{\left(p\right)}{\overline{\mathbf{l}}}_{ij}^{\left(p\right)}.$$

(17)

The outputs of all attention heads are concatenated and linearly transformed to produce the link-enhanced modality representation:

$${\tilde{\mathbf{z}}}_i^k=\varphi \left({\mathbf{W}}_O^{ML}\left[{\mathbf{u}}_{i,k}^{\left(1\right)}\parallel {\mathbf{u}}_{i,k}^{\left(2\right)}\parallel \dots \parallel {\mathbf{u}}_{i,k}^{\left(H\right)}\right]\right),$$

(18)

where ${\tilde{\mathbf{z}}}_i^k$ denotes the modality representation reorganized according to the local structure. This process can be interpreted as “a modality retrieving local structural support.”

For an entity with an empty neighborhood, i.e., ${\mathcal{N}}_i=⌀$, no local links are available for selection. In this case, the model skips local-link aggregation and retains the current modality representation:

$${\tilde{\mathbf{z}}}_i^k={\overline{\mathbf{z}}}_i^k.$$

(19)

If modality k is itself missing for entity i, then ${\overline{\mathbf{z}}}_i^k$ is obtained by imputing the mean representation of that modality over the remaining entities. This strategy avoids applying softmax to an empty set and prevents the model from constructing nonexistent local links for the entity.

4.3. Interaction-Aware Fusion and Neighborhood Aggregation

After bidirectional link–modality interaction, BiMLI generates the final representation of each entity through a “link-level fusion followed by entity-level aggregation” procedure. For a local link $l_{ij}$ of the central entity $h_i$, the model first fuses the base representation ${\overline{\mathbf{h}}}_i$ of the central entity, the modality-enhanced link representation ${\tilde{\mathbf{z}}}_{ij}$ produced by Link→Modality, and the three link-enhanced modality representations ${\tilde{\mathbf{z}}}_i^k$ produced by Modality→Link. The interaction representation of the central entity with respect to this local link is given by

$${\widehat{\mathbf{e}}}_{ij}=\sigma \left({\overline{\mathbf{h}}}_i+{\tilde{\mathbf{z}}}_{ij}+{\tilde{\mathbf{z}}}_i^s+{\tilde{\mathbf{z}}}_i^v+{\tilde{\mathbf{z}}}_i^t\right).$$

(20)

The resulting representation ${\widehat{\mathbf{e}}}_{ij}$ simultaneously contains information about the central entity, link-specific modality evidence, and modality-specific local structural evidence, thereby integrating information from different sources at the link level.

Based on these representations, the model further employs neighborhood attention to aggregate information from different local links and their neighboring entities into the central entity. Specifically, we define

$${\mathbf{q}}_i={\overline{\mathbf{h}}}_i\mathbf{Q},{\mathbf{k}}_j={\overline{\mathbf{t}}}_j\mathbf{K},{\mathbf{v}}_{ij}=\left[{\widehat{\mathbf{e}}}_{ij}\parallel {\widehat{\mathbf{e}}}_{ji}\right]\mathbf{V},$$

(21)

where ${\widehat{\mathbf{e}}}_{ji}$ denotes the interaction representation of neighboring entity $t_j$ on the corresponding inverse link, and ${\mathbf{v}}_{ij}$ jointly encodes the interaction information at both ends of the local link. The relevance score between the central entity and its neighbor is computed as

$${\eta }_{i,j}={\displaystyle \frac{{\mathbf{q}}_i{\mathbf{k}}_j^{\top }}{\sqrt{d_h}}}.$$

(22)

The normalized neighborhood attention weight is then given by

$${\zeta }_{i,j}=softmax\left({\eta }_{i,j}\right)={\displaystyle \frac{exp\left({\eta }_{i,j}\right)}{{\sum }_{n\in {\mathcal{N}}_i}exp\left({\eta }_{i,n}\right)}}.$$

(23)

Finally, the information carried by different local links is aggregated according to the neighborhood attention weights:

$$\overrightarrow{{\mathbf{e}}_i}=\sigma \left(\sum _{n\in {\mathcal{N}}_i}{\zeta }_{i,n}{\mathbf{v}}_{in}\right).$$

(24)

BiMLI therefore first integrates the modality information of the central entity with the results of bidirectional interaction within each local link and then selectively aggregates different neighbors and local links at the entity level. In this way, the central entity’s own information, multimodal attributes, local links, and neighborhood information are jointly encoded into the final entity representation $\overrightarrow{{\mathbf{e}}_i}$, which is subsequently used for link prediction.

If an entity has no neighbors in the training graph, i.e., ${\mathcal{N}}_i=⌀$, the model adopts the fallback representation

$$\overrightarrow{{\mathbf{e}}_i}={\overline{\mathbf{h}}}_i.$$

(25)

4.4. Cross-Modal Contrastive Regularization

Structural, visual, and textual information is extracted using different encoders. Even after being linearly projected into representation spaces of the same dimensionality, the feature distributions of different modalities may remain inconsistent. Bidirectional link–modality interaction relies on representation similarity to estimate the conditional relevance between links and modalities. If the representations of the same entity across different modalities lack basic semantic consistency, modality discrepancies may interfere with the subsequent relevance estimation. BiMLI therefore introduces cross-modal contrastive regularization. While preserving modality-specific information, it encourages representations of the same entity in different modalities to maintain entity-level semantic consistency within the shared space.

Let ${\overline{\mathbf{z}}}_i^k\in {\mathbb{R}}^{d_h}$ denote the projected representation of entity $e_i$ under modality k. For any two distinct modalities $k_1,k_2\in \mathcal{K}$, we use ${\overline{\mathbf{z}}}_i^{k_1}$ as the query, regard ${\overline{\mathbf{z}}}_i^{k_2}$ of the same entity as the positive sample, and treat the modality-$k_2$ representations of the other entities in the batch as negative samples. The directed contrastive loss from $k_1$ to $k_2$ is defined as

$${\mathcal{L}}_{k_1\to k_2}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}log{\displaystyle \frac{exp\left(sim\left({\overline{\mathbf{z}}}_i^{k_1},{\overline{\mathbf{z}}}_i^{k_2}\right)/\tau \right)}{{\sum }_{j=1}^{N}exp\left(sim\left({\overline{\mathbf{z}}}_i^{k_1},{\overline{\mathbf{z}}}_j^{k_2}\right)/\tau \right)}},$$

(26)

where N denotes the batch size, $sim(·,·)$ denotes cosine similarity, and $\tau$ is the temperature parameter.

Although cosine similarity is symmetric, $k_1\to k_2$ and $k_2\to k_1$ are normalized over candidate sets from different modalities and are therefore not equivalent. To simultaneously constrain the bidirectional correspondences among the three modalities, we average the losses over all six ordered modality pairs:

$${\mathcal{L}}_{\mathrm{cl}}={\displaystyle \frac{1}{6}}\left({\mathcal{L}}_{s\to v}+{\mathcal{L}}_{v\to s}+{\mathcal{L}}_{s\to t}+{\mathcal{L}}_{t\to s}+{\mathcal{L}}_{v\to t}+{\mathcal{L}}_{t\to v}\right).$$

(27)

4.5. Decoder

We employ a RotatE-based decoder for link prediction  [12]. Given the fused representation ${\mathbf{e}}_i\in {\mathbb{R}}^{d_h}$ of a head entity, we first apply batch normalization and dropout and then divide the resulting representation into its real and imaginary components:

$${\mathbf{e}}_i^{\prime }=Dropout\left(BN\left({\mathbf{e}}_i\right)\right)=\left[{\mathbf{e}}_{i,\mathrm{Re}}\parallel {\mathbf{e}}_{i,\mathrm{Im}}\right].$$

(28)

The first $d_h/2$ dimensions of the relation embedding ${\mathbf{r}}_j$ are used as the phase vector ${\theta }_j$. RotatE performs an element-wise rotation of the head entity using a unit-modulus complex relation:

$${\mathbf{c}}_{ij}=\left({\mathbf{e}}_{i,\mathrm{Re}}+\mathrm{i}{\mathbf{e}}_{i,\mathrm{Im}}\right)\odot \left(cos{\theta }_j+\mathrm{i}sin{\theta }_j\right),$$

(29)

where $\mathrm{i}$ denotes the imaginary unit and ⊙ denotes element-wise multiplication. The real and imaginary parts of the rotated result are concatenated as ${\mathbf{e}}_{ij}^{\prime }=[Re\left({\mathbf{c}}_{ij}\right)\parallel Im\left({\mathbf{c}}_{ij}\right)]$. This representation is multiplied by the candidate entity matrix $\mathbf{E}\in {\mathbb{R}}^{N^e\times d_h}$:

$$\Psi (h_i,r_j)={\mathbf{e}}_{ij}^{\prime }{\mathbf{E}}^{\top }.$$

(30)

Here, $\Psi (h_i,r_j)\in {\mathbb{R}}^{N^e}$ contains the unnormalized scores of all candidate tail entities. The model is trained using one-to-many cross-entropy, while head-entity prediction is converted into tail-entity prediction through inverse relations.

4.6. Training Objective

For a training query $(h_i,r_j,?)$, the RotatE decoder produces scores for all candidate tail entities, $\Psi (h_i,r_j)\in {\mathbb{R}}^{N^e}$. The predicted probability of candidate entity $t_s$ is

$$p_{ijs}=\sigma \left(\Psi (h_i,r_j,t_s)\right),$$

(31)

where $\sigma (·)$ denotes the sigmoid function. Let $y_{ijs}\in \{0,1\}$ denote the label of candidate triple $(h_i,r_j,t_s)$. For the same query, all known correct tail entities are assigned a label of 1. The one-to-many binary cross-entropy loss for link prediction is defined as

$${\mathcal{L}}_{\mathrm{cross}}=-{\displaystyle \frac{1}{\left|\mathcal{B}\right|N^e}}\sum _{(h_i,r_j)\in \mathcal{B}}{\displaystyle \sum _{s=1}^{N^e}}\left[y_{ijs}log{p}_{ijs}+(1-y_{ijs})log(1-p_{ijs})\right],$$

(32)

where $\mathcal{B}$ denotes the training batch and $N^e$ is the total number of candidate entities. In practice, we use BCEWithLogitsLoss to operate directly on the logits produced by the decoder. Therefore, an additional sigmoid operation is not required during the forward pass.

Combining the bidirectional cross-modal contrastive loss ${\mathcal{L}}_{\mathrm{cl}}$, the final training objective is defined as

$$\mathcal{L}=\beta {\mathcal{L}}_{\mathrm{cross}}+\alpha {\mathcal{L}}_{\mathrm{cl}},$$

(33)

where $\alpha$ and $\beta$ are weighting coefficients that balance the cross-modal alignment objective and the link prediction objective. By introducing inverse relations, head-entity prediction is likewise converted into tail-entity prediction and optimized using the same loss function.

The training procedure begins by extracting structural, visual, and textual features using pretrained encoders. Local-link sets are constructed exclusively from the training triples. For each training batch, the model sequentially performs Link→Modality and Modality→Link interactions to obtain modality-enhanced link representations and link-enhanced modality representations, respectively. Feature fusion and neighborhood aggregation are then applied to generate the final entity representations, after which the RotatE decoder simultaneously computes scores for all candidate entities. The model is trained end-to-end using the weighted sum of the link prediction loss and the cross-modal contrastive loss. The complete training procedure is summarized in Algorithm 1.

4.7. Complexity Analysis

We analyze the computational complexity of BiMLI for a single training batch. Let $N^e$, $N^r$, and $N^{\mathrm{tr}}$ denote the numbers of entities, relations, and triples in the multimodal knowledge graph, respectively. Let $d_e$ be the initial embedding dimension of entities and relations; $d_s$, $d_v$, and $d_t$ be the dimensions of the structural, visual, and textual embeddings, respectively; and $d_h$ be the hidden embedding dimension of the model. For a given entity $h_i$, let $N^{e_i}$ denote the number of its tail entities. The modality set is denoted by $\mathcal{K}=\{s,v,t\}$, and the number of modalities is denoted by $K^n=\left|\mathcal{K}\right|$.

Because inverse relations are introduced, the numbers of relations and triples are doubled. We denote the resulting quantities by $N^{2r}=2N^r$ and $N^{2\mathrm{tr}}=2N^{\mathrm{tr}}$, respectively. Let B denote the training batch size. The numbers of entities, relations, and triples in a batch are denoted by $N_b^e$, $N_b^r$, and $N_b^{\mathrm{tr}}$, respectively, with other batch-level variables defined analogously using the subscript b. We have

$${\displaystyle \sum _{i=1}^{N_b^e}}{N}_b^{e_i}=N_b^{\mathrm{tr}},N_b^{2\mathrm{tr}}=2N_b^{\mathrm{tr}}.$$

(34)

Because the features extracted by GAT, VGG16, and BERT are precomputed and remain frozen during BiMLI training, the following analysis excludes the offline computational overhead of the pretrained encoders.

4.7.1. Module-Wise Complexity

Multimodal encoding. The multimodal encoding module only projects pretrained features from different modalities into a unified representation space. Its time complexity is

$$\mathcal{O}\left(N_b^e\sum _{k\in \mathcal{K}}{d}_k{d}_h\right),$$

(35)

and the space complexity of its output features is

$$\mathcal{O}\left(K^n{N}_b^e{d}_h\right).$$

(36)

link encoding. Local-link encoding concatenates the representations of the head entity, relation, and tail entity and projects the resulting vector into a $d_h$-dimensional space. Its time and space complexities are therefore

$$\mathcal{O}\left(N_b^{2\mathrm{tr}}{d}_e{d}_h\right)\mathrm{and}\mathcal{O}\left(N_b^{2\mathrm{tr}}{d}_h\right),$$

(37)

respectively, where the constant factor of 3 introduced by concatenating the three triple elements is omitted from the asymptotic complexity.

Bidirectional link–modality interaction. In the Link→Modality direction, each local link computes attention over the $K^n$ modalities. In the Modality→Link direction, each modality computes attention over the local-link set of its corresponding entity. The total time complexity of bidirectional interaction is therefore

$$\mathcal{O}\left(K^n{d}_h{\displaystyle \sum _{i=1}^{N_b^e}}{N}_b^{e_i}\right)=\mathcal{O}\left(K^n{N}_b^{\mathrm{tr}}{d}_h\right).$$

(38)

Both interaction directions scale linearly with the total number of local links $N_b^{\mathrm{tr}}$ and do not require pairwise comparisons among different local links. Their primary space complexity is

$$\mathcal{O}\left(K^n{N}_b^{\mathrm{tr}}+\left(N_b^{\mathrm{tr}}+K^n{N}_b^e\right)d_h\right),$$

(39)

where the first term accounts for the attention weights and the second term stores the representations produced by the two interaction directions.

Interaction-aware fusion and neighborhood aggregation. Interaction-aware fusion and neighborhood aggregation primarily consist of local-link representation fusion, linear projection, and neighborhood-attention aggregation. Their time complexity is

$$\mathcal{O}\left(N_b^{2\mathrm{tr}}{d}_h+N_b^{\mathrm{tr}}{d}_h^2\right)=\mathcal{O}\left(N_b^{\mathrm{tr}}{d}_h^2\right),$$

(40)

and their space complexity is

$$\mathcal{O}\left(N_b^{\mathrm{tr}}{d}_h\right).$$

(41)

Cross-modal contrastive regularization. Cross-modal contrastive regularization requires computing pairwise similarities among the entities in a batch. For $K^n$ modalities, there are $K^n(K^n-1)$ ordered modality pairs. Its time complexity is therefore

$$\mathcal{O}\left(K^n(K^n-1){\left(N_b^e\right)}^2{d}_h\right),$$

(42)

and its space complexity is

$$\mathcal{O}\left(K^n(K^n-1){\left(N_b^e\right)}^2\right).$$

(43)

Because $K^n=3$ is fixed in this work, these two terms can be simplified to $\mathcal{O}\left({\left(N_b^e\right)}^2{d}_h\right)$ and $\mathcal{O}\left({\left(N_b^e\right)}^2\right)$, respectively. Decoder. The RotatE-based decoder matches each query against all $N^e$ candidate entities. Its time complexity is

$$\mathcal{O}\left(N_b^{2\mathrm{tr}}{N}^e{d}_h\right),$$

(44)

while the space complexity of the candidate-entity score matrix is

$$\mathcal{O}\left(N_b^{2\mathrm{tr}}{N}^e\right).$$

(45)

The computational complexity of the binary cross-entropy loss is $\mathcal{O}\left(N_b^{2\mathrm{tr}}{N}^e\right)$, which is subsumed by the complexity of the decoder.

4.7.2. Overall Complexity

Combining the complexities of the above modules, the time complexity of BiMLI for a single training batch is

$$\begin{array}{cc}\hfill T=\mathcal{O}(& N_b^e\sum _{k\in \mathcal{K}}{d}_k{d}_h+N_b^{2\mathrm{tr}}{d}_e{d}_h+K^n{N}_b^{\mathrm{tr}}{d}_h+N_b^{\mathrm{tr}}{d}_h^2\hfill \\ \hfill & +K^n(K^n-1){\left(N_b^e\right)}^2{d}_h+N_b^{2\mathrm{tr}}{N}^e{d}_h).\hfill \end{array}$$

(46)

Using $K^n=3$ and $N_b^{2\mathrm{tr}}=2N_b^{\mathrm{tr}}$ and omitting constant factors, the time complexity can be further simplified as

$$\begin{array}{cc}\hfill T=\mathcal{O}(& N_b^e{d}_h\sum _{k\in \mathcal{K}}{d}_k+N_b^{\mathrm{tr}}{d}_e{d}_h+N_b^{\mathrm{tr}}{d}_h^2\hfill \\ \hfill & +{\left(N_b^e\right)}^2{d}_h+N_b^{\mathrm{tr}}{N}^e{d}_h).\hfill \end{array}$$

(47)

When batch inputs, intermediate representations, similarity matrices, and candidate-entity score matrices are all considered, the overall space complexity is

$$\begin{array}{cc}\hfill S=\mathcal{O}(& N_b^e\sum _{k\in \mathcal{K}}{d}_k+N_b^e{d}_h+N_b^{\mathrm{tr}}(d_e+d_h)\hfill \\ \hfill & +{\left(N_b^e\right)}^2+N_b^{\mathrm{tr}}{N}^e).\hfill \end{array}$$

(48)

When the number of modalities and the representation dimensions are treated as constants, the dominant time and space complexities can be summarized as

$$T=\mathcal{O}\left({\left(N_b^e\right)}^2+N_b^{\mathrm{tr}}{N}^e\right),S=\mathcal{O}\left({\left(N_b^e\right)}^2+N_b^{\mathrm{tr}}{N}^e\right).$$

(49)

Here, ${\left(N_b^e\right)}^2$ arises from cross-modal contrastive learning within the batch, whereas $N_b^{\mathrm{tr}}{N}^e$ results from one-to-many decoding over all candidate entities. In contrast, the bidirectional link–modality interaction in BiMLI scales linearly with the number of local links $N_b^{\mathrm{tr}}$ and does not introduce quadratic overhead with respect to the number of local links.

In the implementation, we use stack, scatter_softmax, and scatter_add to process modality features and variable-sized neighborhoods in a batched manner. This optimization reduces the runtime overhead caused by explicit loops and repeated data organization without changing the asymptotic time or space complexity of the model.

5. Results and Discussion

5.1. Experiment Settings

5.1.1. Datasets

To evaluate the effectiveness of BiMLI, we conduct experiments on two publicly available benchmark datasets for multimodal knowledge graph completion: DB15K [42] and MKG-W [43]. DB15K is derived from DBpedia [44], while MKG-W is a subset of Wikidata [45]. Both datasets provide visual and textual attributes for entities. Their statistics are summarized in Table 1. Here, #Image and #Text denote the numbers of entities with available visual and textual attributes, respectively, rather than the total numbers of image or text files.

5.1.2. Evaluation Metrics

Following existing MMKGC studies, we evaluate link prediction performance under the filtered setting using mean reciprocal rank (MRR) and Hits@1, Hits@3, and Hits@10. For each test triple $(h,r,t)$, the model performs tail-entity prediction using the query $(h,r,?)$ and head-entity prediction using $(?,r,t)$. The latter is converted into the tail-prediction query $(t,r^{-1},?)$ through the corresponding inverse relation. During ranking, all other known correct entities except the current test answer are removed from the candidate set. A higher MRR indicates a better average rank of the correct entity, while Hits@k measures the proportion of queries for which the correct entity is ranked among the top k candidates. Unless otherwise specified, the main experimental results are averaged over head- and tail-entity prediction.

5.1.3. Baselines

We compare BiMLI with two categories of representative methods. The unimodal methods include TransE [15], DistMult [14], ComplEx [11], RotatE [12], GC-OTE [13], and TuckER [10]. The multimodal methods include IKRL [18], MMKRL [19], RSME [20], VBKGC [21], OTKGE [22], IMF [24], QEB [46], AdaMF [27], MyGO [28], MoMoK [25], MCKGC [26], and ROAD [38]. These baselines cover major technical paradigms including translation-based modeling, semantic matching, complex-space rotation, tensor factorization, visual enhancement, adaptive fusion, mixture-of-experts learning, and cross-modal consistency learning.

5.1.4. Implementation Details

The experiments are implemented using PyTorch 1.10.1 and CUDA 10.2 and conducted on a server equipped with an Intel Xeon Platinum 8260 CPU, 512 GB of memory, and two NVIDIA Quadro RTX 8000 GPUs. The initial structural, visual, and textual features are extracted using pretrained GAT, VGG16, and BERT encoders, with original dimensions of 256, 4096, and 768, respectively. These pretrained encoders remain frozen during the training of BiMLI.

The unified hidden dimension is set to 256, and the dropout rate of the RotatE decoder is set to 0.3. We optimize the model using RMSprop. The batch size is selected from $\{64,128,256,512,1024,2048,4096,8192,16384\}$. The learning rate, weight decay, and momentum are searched within their specified hyperparameter ranges. The loss weights are selected from $\alpha ,\beta \in \{0.1,0.2,\dots ,0.9\}$, and the number of attention heads is set to $H=2$. All hyperparameters are selected according to the validation MRR.

5.2. Main Results: Overall Performance (RQ1)

Table 2 compares BiMLI with both structure-based and multimodal knowledge graph completion methods. On DB15K, BiMLI achieves an MRR of 42.35%, a Hits@3 score of 48.31%, and a Hits@10 score of 63.59%, corresponding to relative improvements of 6.43%, 10.30%, and 16.34% over the strongest baselines for the respective metrics. Its Hits@1 score is 31.72%, which is slightly lower than the best result of 32.38%. This indicates that the advantage of BiMLI on DB15K is mainly reflected in improving the overall ranking of correct entities and their coverage among the top-3 and top-10 candidates, while further improvement remains possible for top-1 prediction. On MKG-W, BiMLI achieves the best results across all metrics, with MRR, Hits@1, Hits@3, and Hits@10 scores of 43.55%, 36.06%, 47.43%, and 57.41%, respectively. These results correspond to relative improvements of 11.98%, 12.76%, 17.40%, and 15.00%.

Compared with unimodal methods such as RotatE, the overall improvements obtained by BiMLI on both datasets demonstrate that multimodal attributes and local-link information provide effective complementary evidence for structural representations. Compared with existing multimodal methods, its advantages across multiple metrics suggest that modality encoding and fusion alone may be insufficient to fully exploit the conditional dependencies between multimodal semantics and local topology. BiMLI uses Link→Modality to select modality evidence relevant to each local link and Modality→Link to weight local links conditioned on each modality, thereby integrating semantic and topological evidence within a unified framework. Overall, the main results demonstrate the effectiveness of the complete BiMLI framework.

The Improve value in Table 2 is defined as the relative improvement of BiMLI over the strongest baseline in the corresponding column. Let $\mathrm{BestBaseline}$ denote the best result among all baselines for a given metric:

$$\mathrm{Improve}(\%)={\displaystyle \frac{\mathrm{BiMLI}-\mathrm{BestBaseline}}{\mathrm{BestBaseline}}}\times 100.$$

(50)

5.3. Ablation Study (RQ2)

To analyze the contributions of different components, we conduct two groups of ablation experiments. The modality-contribution experiments remove the structural (w/o S), visual (w/o V), and textual (w/o T) modalities, respectively. The model-design experiments remove Link→Modality (w/o LM), Modality→Link (w/o ML), the interaction-aware fusion and neighborhood aggregation module (w/o FA), and the cross-modal contrastive constraint (w/o CL), respectively. The results are reported in Table 3.

First, removing any individual modality reduces all evaluation metrics on both datasets, confirming the complementary roles of structural, visual, and textual information. On DB15K, removing the structural, visual, and textual modalities decreases MRR by 1.24, 2.01, and 1.43 percentage points, respectively, indicating that visual attributes make the largest contribution on this dataset. On MKG-W, the corresponding decreases are 3.57, 1.95, and 3.36 percentage points, suggesting that this dataset relies more heavily on structural and textual evidence. The different modality dependencies observed across the two datasets support the design of BiMLI, which adaptively selects modality information according to local-link contexts.

Second, removing Link→Modality decreases MRR by 2.04 and 5.99 percentage points on DB15K and MKG-W, respectively, whereas removing Modality→Link results in smaller decreases of 0.46 and 0.67 percentage points. These results show that both interaction directions are effective but contribute asymmetrically. Link→Modality is the primary source of performance gains, while Modality→Link provides consistent complementary improvements. This observation supports the complementarity of bidirectional interaction but does not imply that the two directions are equally important.

Removing the interaction-aware fusion and neighborhood aggregation module decreases MRR by 1.96 percentage points on DB15K and 7.28 percentage points on MKG-W, with the latter being the largest decrease among all ablated variants. This result indicates that the representations produced by bidirectional interaction require effective fusion and neighborhood propagation to form discriminative entity representations. Finally, removing the cross-modal contrastive constraint decreases MRR by 1.34 and 2.87 percentage points on DB15K and MKG-W, respectively, confirming the auxiliary role of cross-modal alignment in entity representation learning. Overall, all ablated variants underperform the complete BiMLI model, demonstrating that modality information, bidirectional interaction, interaction-aware fusion and aggregation, and cross-modal regularization jointly contribute to the final link prediction performance.

5.4. Parameters Sensitivity Analysis (RQ3)

Different hyperparameters affect the link prediction performance of the model in different ways and to varying degrees. All hyperparameter experiments are conducted on the validation sets. For each category of hyperparameters, we adopt either a controlled-variable strategy or a joint sensitivity analysis. Specifically, the hyperparameters under investigation are varied while the remaining settings are kept unchanged, and the resulting performance differences are compared. The batch size primarily affects the stability of gradient estimation, whereas the learning rate, weight decay, and momentum jointly influence the optimization process. In addition, the loss-weight coefficients $\alpha$ and $\beta$ exhibit a certain degree of interaction. The experimental results show that multiple configurations achieve comparable performance within reasonable parameter ranges, indicating that BiMLI has a reasonable degree of tolerance to hyperparameter variations.

Figure 5 presents the link prediction performance of BiMLI under different batch sizes. As the batch size increases, the MRR and Hits@K scores on DB15K and MKG-W generally exhibit upward trends. This observation suggests that larger batches provide more in-batch negative samples and may facilitate cross-modal contrastive learning and overall representation optimization. On MKG-W, the model performance improves markedly as the batch size increases from 64 to 4096 and gradually approaches saturation beyond 4096. Although some metrics show slight further improvements at batch sizes of 8192 and 16384, the gains over 4096 are limited and accompanied by greater computational and memory costs. Therefore, except for the batch-size sensitivity analysis, all experiments involving MKG-W consistently use a predefined batch size of 4096 to balance predictive performance and computational overhead and to ensure comparability under the same training configuration. The results obtained with larger batch sizes are reported only to characterize the model’s sensitivity to this parameter and are not used to adjust the configurations of other experiments post hoc. In contrast, DB15K continues to exhibit a relatively clear upward trend within the investigated range; therefore, a batch size of 16384 is used in the formal experiments. Overall, increasing the batch size facilitates the training of BiMLI, but the resulting benefits vary across datasets. The batch size should therefore be selected by balancing predictive gains against computational and memory costs.

Figure 6 and Figure 7 employ parallel-coordinate plots to analyze the sensitivity of BiMLI to changes in the learning rate, weight decay, and momentum, thereby characterizing its performance under different combinations of optimization parameters. Throughout the analysis, the batch size, loss weights, random seed, and all other training settings are kept unchanged. The results show that MRR and Hits@K generally exhibit consistent trends. Multiple configurations achieve comparable performance within reasonable parameter regions, indicating that BiMLI can tolerate moderate variations in the optimization parameters. Nevertheless, some mismatched combinations of the learning rate, weight decay, and momentum lead to marked performance degradation. Moreover, the two datasets exhibit different high-performing parameter regions, suggesting that the effects of the optimization parameters are dataset-dependent. This analysis characterizes the response of the model to parameter variations within the investigated grid and does not claim to identify a globally optimal configuration.

Figure 8 analyzes the joint effects of the link prediction loss weight $\beta$ and the cross-modal contrastive loss weight $\alpha$ on the performance of BiMLI. On DB15K, high MRR and Hits@10 values are distributed across multiple parameter combinations and do not vary monotonically with either individual weight, indicating that the two loss terms need to be appropriately balanced. On MKG-W, a small value of $\beta$ generally corresponds to lower predictive performance, whereas MRR and Hits@10 improve when $\beta$ is increased to an appropriate range. This result indicates that the link prediction objective should retain the primary role during joint training. In contrast, the effect of $\alpha$ is non-monotonic, suggesting that cross-modal contrastive learning is more suitable as an auxiliary regularizer than as the dominant optimization objective. Both datasets contain multiple high-performing regions with comparable results, although their specific distributions differ, further demonstrating the dataset-dependent effects of the loss weights. Overall, BiMLI exhibits a certain degree of tolerance within reasonable weight ranges, while inappropriate combinations of $\alpha$ and $\beta$ may still reduce its predictive performance.

5.5. Degree-Wise Analysis of Bidirectional Interaction (RQ4)

Table 4 further investigates how the structural richness of query head entities affects bidirectional interaction on MKG-W. We consider only tail-entity prediction queries $(h,r,?)$ and divide them into low-, medium-, and high-degree groups according to the degrees of their head entities in the original training graph. Entity degrees are computed only from the observed triples in the training graph, without including the inverse links introduced for head-entity prediction.

The Link→Modality-only variant consistently outperforms the Modality→Link-only variant across the three degree groups by 1.33, 2.75, and 2.95 percentage points, respectively. This indicates that link-conditioned modality selection is the primary source of BiMLI’s performance gains. The complete BiMLI achieves MRR scores of 41.60%, 48.57%, and 34.89% on the low-, medium-, and high-degree groups, respectively, outperforming both unidirectional variants in all three groups. Compared with the Link→Modality-only variant, the complete model improves MRR by 0.62, 0.20, and 1.05 percentage points, respectively, indicating that Modality→Link provides non-redundant complementary information beyond Link→Modality.

The complete model exhibits a relatively larger gain for high-degree entities. This observation is consistent with the motivation for Modality→Link: when a central entity is associated with more local links, the model must distinguish modality-relevant structural evidence from a larger candidate set, making modality-conditioned link weighting potentially more beneficial. In contrast, all three models achieve their highest MRR on medium-degree entities. This suggests that a moderate number of local links can provide sufficient topological evidence, whereas the complex neighborhoods of high-degree entities may contain more heterogeneous or weakly relevant links, thereby increasing the difficulty of evidence selection.

It should be noted that the high-degree group contains only 195 queries, accounting for 4.56% of all 4,274 queries, which is substantially fewer than the low- and medium-degree groups. Therefore, the gain observed for high-degree entities should be interpreted as trend-level evidence under the current experimental setting rather than as a statistically significant conclusion. Overall, the degree-wise results further support the functionally asymmetric yet complementary roles of the two interaction directions: Link→Modality provides the primary performance contribution, while the complementary effect of Modality→Link becomes more apparent for entities with richer local-link information.

5.6. Case Study (RQ5)

To illustrate how bidirectional interaction affects individual queries, Table 5 reports the filtered ranks of the gold tail entities for four representative test queries, where a lower rank indicates better performance. For the queries (R.E.M., genre, ?) and (The Couch, screenwriter, ?), the Link→Modality-only variant ranks the gold entities first, whereas the Modality→Link-only variant ranks them 1,383rd and 16th, respectively. These cases illustrate that link-conditioned modality selection can provide key semantic evidence for prediction. In contrast, for (Randolph Childress, position played, ?), the Modality→Link-only variant ranks the gold entity first, while the Link→Modality-only variant ranks it fifth, illustrating the value of modality-conditioned local-link selection.

For (Birkenhead, country, ?), the two unidirectional variants rank the gold entity 27th and 256th, respectively, whereas BiMLI improves its rank to second. This case illustrates that jointly modeling both interaction directions can alleviate ranking errors that remain in either unidirectional variant. Overall, BiMLI matches or outperforms the stronger unidirectional variant across all four queries.

6. Conclusions

This work investigated bidirectional conditional dependencies between multimodal attributes and local links in attribute-based multimodal knowledge graph link prediction. We argued that existing methods primarily model how links guide modality selection, while the reverse direction—conditioning on modality representations to distinguish and weight individual local links—remains underexplored. To address this gap, we proposed BiMLI, a framework that jointly models two complementary interaction directions: Link→Modality selects relevant semantic evidence along the modality dimension, and Modality→Link selects relevant topological evidence along the local-link dimension. Interaction-aware fusion, neighborhood aggregation, and a RotatE decoder complete the link prediction pipeline, with cross-modal contrastive regularization as an auxiliary constraint.

On DB15K and MKG-W, BiMLI achieved MRR scores of 42.35% and 43.55%, corresponding to relative improvements of 6.43% and 11.98% over the strongest baselines, and attained the best result on seven of eight metrics. Ablation studies confirmed that both directions are effective yet contribute asymmetrically: Link→Modality is the primary performance source, while Modality→Link provides complementary gains. Degree-wise analyses and case studies further demonstrated that the two directions correct ranking errors that persist in either unidirectional variant.

Several limitations point to future work. First, BiMLI currently operates under the transductive setting; extending it to inductive link prediction with unseen entities requires mechanisms for dynamically encoding new local links. Second, experiments were limited to DB15K and MKG-W with three modalities (structure, vision, text); scaling to larger knowledge graphs and additional modality types would test the generality of the framework. Beyond these extensions, the bidirectional conditioning perspective may generalize to other structured prediction tasks involving heterogeneous evidence and local relational contexts.