Visual Semantics in MT Evaluation: Do Image Descriptions Help with Assessment of Multimodal MT Quality?

1. Introduction

Machine Translation (MT) has achieved remarkable progress in recent years, largely driven by advances in neural text-generation architectures and large-scale pre-training [1]. End-to-end encoder–decoder models enabled learning broader representations instead of relying on phrase-based statistical alignments, allowing Neural Machine Translation (NMT) systems to capture long-range dependencies and contextual semantics more effectively. Developments in data augmentation, pre-training, and transfer learning have further facilitated multilingual NMT approaches [2], enabling a single model to handle numerous language pairs. These methods allow high-resource languages to enhance performance in low-resource settings and support more generalized translation capabilities.

Despite these advances, MT systems continue to struggle with linguistic ambiguity [3] and producing contextually appropriate translations [4], particularly in cases where text alone is insufficient to resolve meaning [5]. Words with multiple senses, pronouns referring to visually grounded entities, or culturally situated expressions often require extra-linguistic information for accurate interpretation. To address this challenge, Multimodal Machine Translation (MMT) has emerged as a research direction that integrates visual features—typically from paired images—alongside textual input to provide additional disambiguating context [6,7,8].

While architectural progress in MMT has been substantial, automatic evaluation methodologies to track these advances have lagged behind [4]. Commonly used automatic evaluation metrics (AEMs) such as BLEU [9], METEOR[10], and COMET[11] are largely image-agnostic, assessing translation quality primarily through lexical overlap or semantic similarity. As a result, they fail to capture the visual grounding capabilities that are central to MMT [12]. In addition, benchmark test sets often consist of short and syntactically simple sentences [13], which further limits the ability of existing metrics to reliably reflect system quality in multimodal settings. Consequently, the lack of modality-aware evaluation frameworks has led to ongoing uncertainty about whether reported improvements in MMT systems can be attributed to the inclusion of visual information [14]. The absence of standardized evaluation metrics and datasets further complicates cross-system comparison, making it difficult to measure progress in the field when different studies adopt different evaluation protocols [4].

In principle, image context should be incorporated directly into MMT evaluation, in a similar manner as human assessors rely on visual cues to resolve ambiguities in translation judgments [12]. Related work in image captioning evaluation demonstrates that metrics integrating visual features align better with human judgments [15,16,17], highlighting the importance of multimodal grounding for assessment. However, extending these approaches to MMT is non-trivial for two main reasons. First, in image captioning the image and caption are intrinsically aligned, whereas in MMT the image often serves only as an auxiliary cue for resolving ambiguity and may not correspond directly to all aspects of the textual content. Second, translation ambiguity is inherent to the MT task itself, meaning that evaluation methods must assess not only semantic similarity but also consistency and adequacy across languages.

In parallel, pre-trained AEMs have demonstrated improved performance through the use of document-level or multi-sentence context, enabling better modelling of discourse coherence and coreference resolution [18?,19]. These metrics operate either at the sentence level by augmenting embedding representations with surrounding context or at the document level by computing similarity scores across extended text spans [20,21]. Given their ability to integrate extra-sentential information and discourse signals, exploring the applicability of such metrics for MMT evaluation constitutes a promising direction.

In this paper, we investigate and compare multimodal and text-only MT systems through the lens of state-of-the-art AEMs, with a particular focus on the reliability of text-only metrics in multimodal settings. We propose a simple approach to incorporating visual semantics into existing metrics by using target-language image descriptions (IDs) as a proxy for visual context. Through a comparative evaluation of context-aware and context-agnostic metrics on multilingual multimodal benchmark datasets, we aim to address the following research question (RQ):

• RQ: Can image descriptions, when integrated with existing pre-trained AEMs, improve contextual sensitivity and evaluation accuracy in MMT?

The remainder of this paper is organized as follows. Section 2 provides an overview of MT, multimodal MT, and their evaluation. Section 3 reviews existing MMT evaluation methods and discusses their limitations. Section 4 introduces our methodology for integrating IDs into metric-based evaluation. Section 5 describes the experimental setup, followed by the meta-evaluation in Section 6. Finally, Section 7 concludes the paper by discussing how different AEMs capture different aspects of MMT system performance.

2. Background

2.1. Machine Translation

MT is the task of automatically translating text from one language into another. MT models aim to capture and model the complex linguistic features of languages—such as syntax, semantics, and pragmatics—that are fundamental to understanding language and generating accurate translations. Since its inception, MT research has undergone significant evolution, transitioning from rule-based approaches to statistical methods [22,23], advancing to state-of-the-art NMT systems [24,25,26], and now entering the emerging era of Large Language Models (LLMs) [27]. Modern MT systems, powered by vast amounts of bilingual data and advanced computational resources, can produce translations of considerable quality [28].

NMT currently dominates the MT field due to its simplified translation model, faster data processing, and more efficient memory utilization compared to earlier paradigms [29]. These models typically employ encoder–decoder neural network architectures: the encoder processes the source sentence as input, while the decoder generates the target sentence conditioned on the encoder’s output. NMT systems based on the Transformer architecture have achieved state-of-the-art performance [26].

While statistical and neural models have traditionally relied on labelled data for training, recent advances in pre-training have enabled the modelling of sequential problems through self-supervised and multi-task learning paradigms, as exemplified by LLMs [1]. These models, capable of understanding and generating human-like text, can perform a wide range of Natural Language Processing (NLP) tasks. The emergence of LLMs introduces new opportunities for understanding how knowledge is represented and transferred across languages, domains, and modalities, and how these models may redefine the scope and methodology of MT [30].

2.2. Multimodal MT

MMT extends traditional MT with integration of information from multiple modalities—such as text, images, video, or audio—to enhance the accuracy and contextual understanding of translation output. MMT aims to augment the latent representations derived from standard translation systems with information drawn from these additional modalities [8]. The fundamental intuition driving MMT is that visual context can resolve translation ambiguities and supply missing textual information, thereby leading to improved translation quality over text-only MT. This research area gained prominence through a series of dedicated shared tasks held at the Conference on Machine Translation (WMT)1 starting in 2016, primarily focusing on image-guided translation [6,31,32].

The growing interest in MMT is largely driven by the increasing availability of diverse multimodal datasets and the architectural flexibility of neural models, which can jointly process multiple modalities such as vision, speech, and text [2,33]. These non-textual modalities can provide helpful context, particularly in cases where MT systems struggle—such as the translation of ambiguous words whose correct interpretation and rendering depend heavily on contextual information [13,34]. For instance, the English word "bank" might translate differently in French depending on whether the image shows a financial institution ("banque") or a river edge ("rive"). Similarly, an image can clarify whether the English word "glasses" should be translated as ‘drinking vessels’ or ‘spectacles’, as depicted in example given in Figure 1. MMT systems can leverage the visual input to make these disambiguation decisions [35] as the visual context can aid in modeling complex grammatical features, such as gender marking (e.g., inferring the sex of a person in the image to correctly translate from a gender-neutral to a gendered language). Caglayan et al. [8] and Futeral et al. [13] highlight the robustness of MMT models against input noise, such as spelling mistakes or deliberately missing input words, suggesting that visual context can mitigate deficiencies when linguistic context is scarce [8,13].

MMT systems are primarily image-guided models build on the frameworks of neural networks [8,36], integrating visual signals in various architectural stages [13,37,38]. These methods are generally classified into early fusion and late fusion strategies. Early fusion aims for a closer, often pre-encoder, integration of textual and visual representations, typically utilising global visual features (e.g., averaged pooled ResNet-50 features) [12,31,32], such as through encoder/decoder initialization in Recurrent Neural Networks (RNNs) [39,40,41,42]. Other early techniques include treating the visual feature vector as a pseudo-word and prepending or appending it to the source embeddings, or using element-wise multiplication [43]. Advanced early fusion approaches, like the multimodal Transformer [44], integrate multimodal self-attention layers to fuse modality representations deeply within the encoder. Conversely, late fusion strategies rely on separate encoding of modalities and merge representations later, usually closer to the decoder, often leveraging hierarchical attention [45] and gating mechanism [14,40]. This involves an auxiliary visual attention layer over the image data to produce a distinct visual context vector [5], which is subsequently combined with the textual context using summation, concatenation [46], or sometimes specialized mechanisms like hierarchical attention [45,47]. Furthermore, other fusion and training paradigms exist, such as Multi-task Learning (MTL), which jointly trains the text encoder to predict the visual feature vector [35,48]; or modern approaches like ZeroMMT [38] and VGAMT [13], which adapt pre-trained text models via lightweight adapters and a visually-conditioned masked language modelling (VMLM) objective combined with a Kullback-Leibler (KL) divergence penalty to maintain translation fidelity [38].

In another growing area of research, large-scale vision-language models (LVLMs) [15,49,50] have demonstrated strong zero-shot and few-shot performance across a range of downstream multimodal tasks (e.g. image caption, text-to-image or image-to-text generation), often achieving impressive results with minimal or no task-specific data [16,51]. Models like CLIP [49], Flamingo [52], and BLIP [50] have recently been investigated for their potential of visually informed translation capabilities [53]. Notably, [16] aligns features from a multilingual text model with those extracted by CLIP using a lightweight mapping network, while [54] combines visual pre-training with cross-lingual masked language modeling and region classification to produce visually grounded representations that, when fine-tuned, outperform unimodal MT baselines.

The above discussion highlights both the growth and potential of MMT systems; however, a key obstacle remains the lack of standardized evaluation methods for reliably and accurately tracking research progress in this area. Current evaluation practices continue to rely heavily on string-based metrics such as BLEU, METEOR, and CHRF, often with little or no complementary analysis involving expert human assessment. As a result, there is ongoing debate about whether visual context contributes meaningful information beyond what is already available in the text [14].

2.3. MT Evaluation

Evaluation constitutes a fundamental component of MT research, serving as both a diagnostic tool and a benchmark for measuring system progress. Broadly, evaluation methods can be classified into human and automatic approaches.

2.3.1. Human Evaluation

Human evaluation (HE) remains the most reliable method for assessing translation quality, as it directly reflects human perceptions of fluency, adequacy, and contextual appropriateness. When conducted by professionals, HE has the advantage of capturing linguistic nuances, pragmatic meaning, and discourse coherence—dimensions that automated methods may fail to consider. Over the years, several HE metrics have been proposed, with adequacy and fluency remaining the most common criteria, typically assessed using various scales [55,56,57], and performed by either human experts or crowd-workers [58,59]. Metrics based on error typologies, such as MQM [60] and Error Span Annotation (ESA) [61], are considered more reliable than simple scoring methods. MQM requires human evaluators to explicitly identify and annotate errors in the translation output rather than assigning a single score to a segment or sentence [62], while ESA offers a lighter error identification metric, similar to Direct Assessment (DA) [58].

Post Editing (PE), in addition to its role in computer-aided translation and revision, is another widely used approach for MT evaluation [63] as PE effort is often interpreted as an indirect measure of MT quality. In semi-automatic settings, where MT output is corrected by a human translator, the Human Translation Edit Rate (HTER) score—calculated as the ratio of edits to the total number of words—serves as estimate of quality, with lower scores indicating better quality translations [64]. HE also plays a fundamental role in advancing automatic evaluation research, as many evaluation metrics rely on human-annotated data to train neural models and/or to align their scoring behaviour with human judgments.

However, despite its interpretative richness, human evaluation is inherently time-consuming, costly, and susceptible to subjective variation [57,65]. Inter-annotator inconsistencies, differences in linguistic proficiency, and variations in task design can all affect the reliability and reproducibility of results. As such, while important for final system assessment, human evaluation alone is impractical for large-scale or iterative system development where instant feedback and comparison with numerous models is essential.

2.3.2. Automatic Evaluation

Automatic evaluation (AE) provides a scalable and cost-effective alternative to HE, allowing for rapid benchmarking during system development and optimisation. Early heuristic-based metrics, most notably BLEU [9], TER [64], METEOR [10] and ChrF [66], offered a simple approach by quantifying n-gram overlaps between candidate and reference translations. Such methods provided an efficient proxy for translation quality and were instrumental in advancing statistical and early neural models. However, as MT systems have evolved—particularly with the advent of NMT—the limitations of these metrics have become increasingly evident [67]. Word- and n-gram–based measures tend to penalize legitimate paraphrasing, alternative phrasal reordering, or stylistic variation, and often perform poorly on morphologically rich languages [67,68,69]. In response, the field has seen a transition towards embedding-based and pretrained metrics [70,71], which utilise distributed semantic representations from language models such as BERT [70]. These approaches provide a more nuanced understanding of meaning beyond surface-level matching, yet they remain susceptible to biases inherent in training data, domain sensitivity, and cross-lingual inconsistencies [72,73]. To overcome limitations of LM based pretrained metrics, a more recent trend has become fine-tuning models on translations with human evaluation scores [11,67,74,75]. Most of these metrics either n-gram or neural (supervised or un-supervised) perform reference-based quality estimation, making their performance heavily dependent on the availability and quality of these references. In contrast, reference-free metrics tasked to predict the quality of MT outputs considering the segments in the source language without relying on gold-standard human translations [76], rendering them particularly suitable for low-resource settings. A few notable examples include BARTScore [77], PRISMsrc [78], COMET-QE [79].

While automatic metrics offer efficiency and objectivity, they may not always fully capture the multifaceted nature of translation quality, particularly in terms of contextual appropriateness, stylistic fidelity, and pragmatic meaning [57,80]. In addition, their numerical scores can sometimes be difficult to interpret. As a result, recent research has began exploring evaluation approaches that incorporate explicit error annotation and correction mechanisms [81]. More recently, studies have also investigated the use of LLMs for automatic translation error labeling [82,83]. While these approaches show promising results, their performance may vary across evaluation settings [84].

2.3.3. Context-Aware Evaluation

The study of context in MT evaluation focuses on incorporating document-level, paragraph-level, or inter-sentential information, rather than evaluating translations at the single-sentence level [85,86,87]. It is now widely accepted that both human and automatic evaluations should take context into account [20,27,88,89,90]. In particular, Castilho et al. [20] showed that nearly half of the translations they analysed required additional context for accurate assessment. Reflecting this shift, since 2022 the Conference on Machine Translation (WMT) has adopted document-level human evaluation for ranking systems in shared tasks [27,91]. Using platforms such as Appraise2 , human evaluators can assess either multi-sentence segments or individual sentences while having access to the full document context. However, despite these advances in human evaluation, adapting contextual principles to automatic evaluation remains challenging, and most existing automatic metrics still operate primarily at the word or sentence level.

Several attempts has been made to extend traditional AEMs to context-aware or discourse-level evaluation [86]. For example, Jiang et al. [18] introduced BlonDe, a document-level metric that incorporates features such as named entities, tense, pronouns, and discourse markers to assess translation quality beyond the sentence level. Building on this idea, Vernikos et al. [92] introduced a straightforward method to extend sentence-level metrics to document-level by taking additional sentences from test set as context. They demonstrated improved correlation scores with human judgments by adapting three widely used metrics—COMET, BERTScore, and Prism—and COMET-QE (a reference-free metric). Their approach involved concatenating each sentence with its two preceding sentences, which were then used for embedding the reference, hypothesis, and source. Although the metric extends evaluation to the document level, it still scores sentences individually, with additional context only used during the embedding process. In the reference-free setting, Raunak et al. [19] introduced SLIDE (SLIding Document Evaluator) that operates on sentence blocks using a sliding window over the test set and accumulating scores for the entire chunk. The authors reported higher pairwise system ranking accuracy compared to sentence-level baselines (COMET-Kiwi [79]) and comparable performance with reference-based metrics, even slightly surpassing Doc-COMET [92]. For paragraph-level evaluation, Deutsch et al. [85] explored using a sentence-level dataset with concatenated multiple sentences to train a neural metric on longer texts, reporting only marginal or no improvements over baselines.

LLMs typically operate with very large context windows and, together with their in-context learning and reasoning abilities, have attracted growing interest for automatic evaluation tasks [86,93]. LLM-based metrics generally rely on prompt-based methods that instruct models to assign quality scores or to identify and categorize translation errors [82,94]. By providing multiple examples within prompts, these models can learn and contextualize their scoring strategies through in-context learning [95]. Overall, their ability to handle extended context and multilingual input highlights the potential of LLMs to reshape MT evaluation. However, several challenges remain before LLM-based metrics can be reliably adopted for primary evaluation. These include reproducibility concerns associated with proprietary models [86], inconsistent adherence to task instructions [96], potential knowledge dilution in long prompts [97], and relatively weak correlations with human judgments at the segment level [84,96].

3. Current Approaches to Automatic MMT Evaluation and Challenges

As mentioned in Section 2.3, due to high temporal and monetary costs, automatic evaluation is often the only practical way to provide feedback during model development and to track progress in the field. However, the automatic evaluation of MMT systems is inherently more challenging than that of conventional MT systems. Since most MMT models are extensions of NMT architectures, they inherit common translation issues such as grammatical errors, inappropriate lexical choices, and semantic inadequacies [98,99], while simultaneously introducing additional complexities arising from the integration of visual information [14].

In practice, evaluation continues to rely heavily on text-based AEMs and relatively short and simple test sets derived from Multi30K. These metrics and datasets provide only a narrow view of model capabilities, particularly with respect to the multimodal dimension [100,101]. Despite well-documented limitations of surface-level metrics such as BLEU, many MMT studies still report performance based solely on these scores, often without human validation or statistical significance testing [102,103]. Even advanced neural AEMs, originally designed for traditional MT systems, are commonly applied without adaptation to non-textual modalities. Consequently, such metrics may struggle to distinguish between systems that genuinely leverage visual information for disambiguation and those that produce fluent translations independent of image input [104].

For example, Elliott [37], using METEOR text-similarity evaluation, showed that some MMT models are insensitive to visual input and produce nearly identical translations even when provided with unrelated images, suggesting that visual features are often ignored. Similarly, Raunak et al. [105] argued that the visual modality does not lead to significant gains in translation quality; however, their comparison relied on BLEU scores. Building on such findings, Wu et al. [14] questioned the validity of reported improvements in MMT and suggested that observed gains may instead result from a regularization effect. In response, Li et al. [101] highlighted limitations of evaluation benchmarks derived from Multi30K, noting that the dataset was not originally designed with multimodal evaluation in mind and advocating the creation of more suitable MMT datasets. In a similar vein, Futeral et al. [13] proposed a multilingual and multimodal contrastive evaluation set containing ambiguous sentence pairs paired with disambiguating images, requiring models to exploit visual information to generate correct translations. Datasets such as MultiSense [106] and MultiSubs [107] further expand evaluation to cross-lingual and cross-modal ambiguity.

While these efforts represent steady progress in test sets, the automatic metrics used for evaluation still require refinement to align with the domain-specific needs of MMT. As current evaluation lacks standardized vision-aware metrics capable of accurately justifying the visual context and quantifying the quality of translation. This also highlights the need for a deeper understanding of the limitations and potential of existing automatic, particularly context-aware, approaches in context of MMT evaluation.

Based on the discussion above, we argue that the evaluation of MMT systems is a twofold problem: (1) assessing the model’s visual awareness, i.e., its sensitivity to visual context, and (2) evaluating the quality and correctness of the generated translation. Therefore, evaluation metrics should capture not only linguistic fidelity but also the extent to which visual information contributes meaningfully to translation quality. This raises the question of whether existing evaluation metrics, originally developed for text-only MT systems, are suitable for MMT tasks and capable of distinguishing between text-only and multimodal MT systems. In the following sections, we examine the potential of existing pre-trained evaluation metrics (Section 4) for MMT tasks and conduct experiments (Section 5) to investigate their sensitivity to multimodal information in the MMT setting.

4. Context-Aware Evaluation of MMT Systems Using Document-Level Metrics

As discussed in Section 2.3.3, document-level metrics with extended context have shown better performance in text-only tasks over baselines in both reference-based and reference-free settings. This motivates us to explore their potential for context-aware evaluation of systems that incorporate multimodal information. However, most multimodal datasets consist of independent sentence-level instances, where each sentence represents a distinct conceptual unit [101]. As a result, the direct use of document-level metrics is not feasible, since these metrics rely on neighbouring sentences for context.

A practical alternative is to use IDs as contextual input for document-level metrics when surrounding sentences are not available. This integration can be seamlessly applied without modifying the architectures of existing metrics. Since IDs represent visual content in linguistic form, their inclusion can provide useful disambiguation signals during evaluation, particularly for context-dependent translations, following the same approach used to improve contextual awareness in MT models [39,108]. This evaluation setup investigates whether integrating IDs into AEMs improves their performance in MMT evaluation, as posed in the research question.

We hypothesise that adding visual context through IDs will yield higher correlations with human judgements than traditional text-only automatic metrics in MMT tasks. To address this question and validate the hypothesis, we conduct a series of experiments using both traditional and context-aware AEMs. We compare their performance across multiple multilingual multimodal datasets with respect to correlations with human judgments. In the sections below, we explain the proposed integration of IDs in metrics and introduce the selected document-level metrics.

4.1. Image Description as Context

Visual–language pre-training has emerged as a powerful paradigm for jointly learning representations from visual and textual data [50,109,110]. Models in this category integrate visual and linguistic information by pairing strong visual encoders with large, often frozen, language models to produce unified multimodal representations (e.g., CLIP [111] BLIP-2 [50], Flamingo [52] and Gemma-3 [112]). These architectures have achieved state-of-the-art performance across a range of multimodal tasks, including image description, object detection (OD), and object recognition (OR) [113]. Recent work has explored the use of such task-agnostic visual–language models (VLMs) for image caption evaluation, aiming to assess both the linguistic and visual dimensions of candidate captions [15,16,17].

In order to examine the potential of such models in MMT evaluation, we first obtained the multilingual IDs for each evaluation set (Table 2) using a VLM via prompting (see appendix A1 for prompts used for each language). The available test sets contain source-side ambiguous text in English, therefore to evaluate English–to–{German, Czech, French} translation direction, we generated the corresponding IDs in the target language. Directly generating visual context (which refers to the detailed contents of an image including objects, attributes and their relationships) in the target language, help eliminate the possible linguistic or translation ambiguity [17]. In the next stage, these IDs are incorporated as contextual information within document-level metrics to assess whether they enable AEMs to more accurately assess the translation quality and align with human judgemnts.

For ID generation, we employ Gemma-3-12B-IT, multimodal LLM from the Gemma family 3. This choice was guided by several factors including: the model must be open-source, multilingual, officially support image captioning and description generation, and achieve an accuracy above 50 on standard benchmark datasets. We also considered other models such as CLIP, BLIP, and Flamingo. However, following manual qualitative inspection of the generated descriptions, Gemma-3-12B-IT was selected for its better performance4. Table 1 presents illustrative examples from the CoMMuTE dataset, showing source sentences, target translations, and the corresponding automatically generated IDs.

4.2. Context Integration in Document-Level Metrics

Document-level metrics typically use surrounding source and reference sentences from the same document, when evaluating a given sentence. The context is extracted using configurable parameters that determine the number of preceding sentences to include [92]. In our experiments, we extend two existing document-level metrics—Doc-BERT and Doc-COMET—modifying their context integration module to use IDs instead of surrounding sentences, adapting them to MMT task. Both of these metrics have their sentence-level baselines, open-source up-to-date implementations and are widely applied for comparing and developing MT systems [27,96,114].

Doc-BERT is an extension of BERTScore5 to the document-level evaluation. Originally [92], the context is drawn from the reference text and added to both the hypothesis and reference sentences, then excluded when calculating the similarity score for the current segment. In our implementation, we replace this textual context with IDs, extracted from relevant images using a VLM. These textual descriptions are concatenated with the hypothesis and reference before the embeddings are computed, thereby enriching their contextual representations. For baseline pre-trained language model, we used multilingual transformer models such as roberta-large and microsoft/deberta-xlarge-mnli, selected based on their strong correlation with human judgments6.

Doc-COMET [92] extends the COMET7 framework by integrating source and reference context directly into the encoder. In the original version, COMET incorporates preceding sentences from the source text along with the current source sentence, while for the hypothesis, the context is taken from the reference to prevent potential error propagation that could occur if hypothesis context were used. Similar to sentence-level COMET, document-level extension is initialized from a pre-trained language model (RoBERTa) and fine-tuned on human judgment data. In our adaptation, we replace the textual reference context with IDs, thereby allowing visual semantic information to contribute to contextual encoding. Since IDs are available only in the target language, the source context is kept same as in the original Doc-COMET setup.

5. Experiments

In this section, we describe the experimental setup, including the datasets, multimodal and text-only MT systems, and evaluation procedure. To ensure transparency and reproducibility, all metrics are evaluated using their default configurations, and results are reported from a single run unless otherwise stated. Also, we report results obtained using publicly available datasets, open-source metrics, and pre-trained language models, including LLMs. For trained and pre-trained models such as BERTScore and COMET, we employ the best-performing (as per their official results) publicly available models with default hyper-parameter settings. For meta-evaluation, we consider system performances, translation accuracy on contrastive test-sets and correlation with human scores where human annotations are available.

5.1. Dataset

We use a combination of evaluation sets from image caption and MT domain, including Multi30K, MSCOCO-test2017, MLT, CoMMuTE, and THumB. These datasets cover a range of evaluation aspects including ambiguous lexical terms, verbs, and contrastive multilingual translations, providing the means to assess both lexical accuracy and multimodal sensitivity. These datasets are described below:

THumB 1.0 [115] comprises 500 images sourced from MSCOCO [116], each paired with one human-written caption and four automatically generated captions. The evaluation of candidate captions involves manual assessment along three dimensions: precision (the extent to which the caption accurately describes the image), recall (the degree to which it captures salient information within the image), and total (an overall quality score considering fluency, inclusivity, and conciseness). To support automatic evaluation, the dataset contains 4 reference captions for each image. The presence of both human-authored captions and human ratings makes this dataset a valuable resource for meta-evaluation. However, its main limitation lies in its monolingual nature, as the image captions are evaluated in English language only. We utilise THumB to estimate overall system performance and compute metrics’ correlation with human judgements.

MLT [34] is constructed by identifying 1,108 English words that are ambiguous(i.e., they have multiple possible translations in the target language). Each word appears across multiple sentences, resulting in a total of 98,647 MLT data points. The dataset, available for both English–German and English–French language pairs, is distributed under the Creative Commons Attribution–NonCommercial–ShareAlike 4.0 International License. Its multimodal design, with identified ambiguous lexical items can be helpful in studying visual disambiguation capabilities of models. In this study, we use MLT to investigate whether systems with stronger visual-awareness receive correspondingly higher ratings from AEMs compared to image-agnostic systems.

test2017 (MSCOCO) [31] is a standard evaluation set for the MMT task, comprising IDs that often contain ambiguous verbs. The dataset is derived from images sampled from the VerSe dataset [117], which in turn was adapted from the MSCOCO dataset [116]. Ambiguous verbs with multiple senses were first identified in VerSe using the IDs in English, and their corresponding translations were then collected in the German and French language. Owing to its design and relevance to lexical ambiguity, MSCOCO-test2017 has become a widely adopted benchmark for evaluating MMT systems [31].

test2016 (Multi30K) is another widely used benchmark for evaluating MMT systems. It is a test split of the Multi30K dataset [36], which itself is an extension of the Flickr30K dataset [118]. The original Multi30K corpus comprises 31,014 images paired with English descriptions translated into multiple languages. While the IDs were collected via Amazon Mechanical Turk, the translations were produced by professional native translators, ensuring high linguistic quality and cross-lingual consistency. This test-set comprises of 1000 English segments and their translations in Czech, German and French.

CoMMuTE (Contrastive Multilingual Multimodal Translation Evaluation) [13] contains 155 lexically ambiguous English sentences, each paired with two plausible translations in the target language. Typically, each translation is correct given the accompanying image, which serves to disambiguate the intended sense of the ambiguous source word. The ambiguous items are drawn either from the discourse sets of Bawden et al. [119] or newly introduced by the authors. Images were sourced online and chosen specifically to resolve the ambiguity in one of the two senses. The dataset includes English–French, English–German, and English–Czech language pairs, providing exactly two translations and one disambiguating image per instance. By offering contrastive multilingual translations, CoMMuTE supports and enables the examination of metric robustness in scenarios with noisy or suboptimal references.

5.2. Translation Systems

The evaluation datasets (Section 5.1) are provided as triplets comprising the source text, reference translation, and associated image. However, they do not include hypothesis translations—an essential component for automatic evaluation. To address this, we generate hypotheses using a diverse set of translation systems, including NMT, MMT, and LLM-based models, spanning both text-only and multimodal configurations. The translation systems used in this study are described in detail below.

ZeroMMT [120] is a zero-shot MMT approach that bypasses the need for large-scale multimodal parallel data by adapting a strong text-only MT model with lightweight visual adapters during training. Given the scarcity of multimodal data, MMT systems typically underperform compared to standard NMT models. To address this, the authors extend strong baseline NMT models with multimodal capabilities, removing the need of costly supervised data. The approach leverages visually-conditioned masked language modeling (VMLM) to integrate visual context into translation, and employs Kullback-Leibler (KL) divergence to align outputs between multimodal and text-only MT systems. Validation was performed on multiple languages, including German, Russian, French, Czech, Arabic, and Chinese, using BLEU and COMET. In our experiments, ZeroMMT represents purpose-build strong MMT to facilitate comparison with traditional MT systems.

Opus-MT [121] is a collection of open-source bilingual and multilingual MT models, primarily trained on the OPUS parallel corpus. The project aims to provide freely accessible translation services and tools, independent of commercial interests or restrictions. OPUS-MT hosts over 1,000 pre-trained models8 covering a wide range of language pairs. The models are based on the standard Transformer architecture and implemented using the Marian-NMT framework9. For our experiments, we use the bilingual models available on Hugging-Face and take Opus-MT as a representative of open-source bilingual text-only MT systems.

Aya-23-8B [122] is a multilingual LLM based on an optimized Transformer architecture and fine-tuned to follow human instructions. Aya-23-8B is a text-to-text model, accepting only text inputs and producing text outputs. The model supports 23 languages and contains 8 billion open-source parameters. Although Aya-23 was primarily trained on general text data and not specifically fine-tuned or evaluated for MT tasks, it demonstrated strong performance on the WMT24 translation benchmarks [27]. Many recent LLMs exhibit inherent multilingual and translation capabilities, yet there is no systematic and reliable evaluation of their MT performance. For this reason, we include text-only LLM (Aya-23-8B) in our experiments on MMT evaluation. Model selection was guided by reported language coverage in the original documentation, ensuring that both source and target languages of our test sets are supported. We further prioritised open-source models to ensure reproducibility of results. The prompt used for translation is given below:

"Translate the following sentence into {German | French | Czech}, output only the translation, nothing else."

Gemma-3-12B-IT [112] is an instruction-tuned multimodal LLM. Instruction tuning enables the model to follow task-specific prompts and natural language instructions, making it well suited for our task where multimodal context can be added via prompting. Gemma-3-12B-IT supports long context windows of up to 128K tokens and offers multilingual capabilities. Its visual grounding can help address lexical ambiguities and context-dependent meanings that are challenging for text-only MT systems. We employed Gemma-3-12B-IT for multimodal translation via prompting, instructing the model to translate the source text while considering the associated image, with following prompt:

"Translate the following sentence into {German | French | Czech}, using the image as context. Output only the translation, nothing else."

5.3. Baseline Metrics

We use several baseline metrics to compare our proposed methods against well-studied approaches commonly used to report progress in MMT research, we below describe those baselines.

BLEU [9] is a widely used metric for MT evaluation that measures the precision of n-grams between candidate and reference translations. Its popularity stems from computational efficiency and its ability to capture local word-level correspondence. However, BLEU is criticised for notable reasons: it disregards grammatical correctness, tends to favour shorter translations, and struggles to account for valid paraphrases or synonymous expressions that preserve meaning [69,102]. Despite these limitations, BLEU continues to be frequently used to report and interpret progress in recent MT research [38,96], mainly due to its language independence, computational simplicity, and long-standing acceptance within the research community.

TER (Translation Edit Rate) and its variant Human TER [64] quantify the number of edits required to transform a machine-translated output into a reference translation. By focusing on the post-editing effort, TER provides a direct and interpretable measure of the changes necessary to produce an acceptable translation. This emphasis on edit operations makes TER particularly useful for assessing practical translation quality and post-editing effort.

CHRF [66] is a character n-gram F-score metric that evaluates translations based on character-level matches rather than word-level correspondence. This design makes CHRF particularly effective for morphologically rich languages, as it is largely language-independent and does not require tokenization. As a result, CHRF has been shown to correlate well with human judgments—often outperforming more complex word-level metrics—and remains a reliable choice, especially for low-resource languages [114].

BERTScore [123] is an unsupervised metric for evaluating text generation that leverages contextual embeddings for soft comparison between reference and candidate translations. Tokens from both the reference and hypothesis are represented using embeddings from models such as BERT or RoBERTa, and precision, recall, and F1 scores are computed based on the best-matching token embeddings. BERTScore has been shown to outperform traditional untrained metrics, demonstrating stronger alignment with human judgments [71]. In our experiments, we only report F1 scores with TF-IDF weighting in our experiments and adapted the models from official implementation10.

COMET [11] is a supervised neural evaluation metric that differs fundamentally from string-based metrics (BLEU, TER, CHRF) and embedding-based approaches such as BERTScore. The COMET framework includes both reference-based and reference-free variants, trained on human judgment data such as Direct Assessments (DA) [58], HTER [64], and Multidimensional Quality Metrics (MQM) [60], and supports a wide range of languages. In our experiments, we use the default wmt22-comet-da model available on HuggingFace,11 which is initialized with XLM-RoBERTa as its backbone. This model computes scores using the source text, hypothesis translation, and reference translation, enabling it to capture semantic similarity and meaning preservation more effectively than surface-form metrics [124].

6. Meta-Evaluation

There are multiple approaches to metric meta-evaluation, which aims to quantify how well automatic metrics agree with human judgments of translation quality. Agreement can be measured using ranking-based approaches such as Spearman’s $\rho$ or Kendall’s $\tau$, or using linear correlations such as Pearson’s coefficient, each capturing different aspects of metric performance. Additionally, metric evaluation can be defined at various levels, including segment-level or system-level, and can focus on specific language pairs or domains.

One of the primary use cases of automatic evaluation metrics is ranking MT systems during development or when comparing systems from different competitors. Such rankings can be useful for shortlisting or clustering systems based on performance. For example, in WMT shared tasks, the large number of participating systems are shortlisted using AEMs for subsequent manual filtering [114]. Ideally, AEMs should rank MT systems in a manner consistent with human judgments, and their effectiveness is often evaluated based on how closely their rankings align with those of humans [114,125]. However, in our study, the number of systems is limited (four in total), which does not provide sufficient statistical power for pairwise system-level accuracy or significance testing between system rankings. Therefore, we rely on traditional correlation-based measures and accuracy on contrastive evaluation for meta-evaluation.

In the following sections, we first present the overall translation performance of the MT and MMT systems (Section 6.1). This is followed by an analysis of the metrics’ correlation with human judgment (Section 6.2) and an assessment of the contextual awareness of the document-level metrics (Section 6.3). Finally, we report the results from a small-scale human evaluation (Section 6.4) to provide qualitative insights.

6.1. System-Level Results

This section provides an overview of the performance of the selected MT systems across different metrics on the evaluation set. System scores are computed as the average of their segment-level performance. These averages can be further aggregated across systems or languages to obtain an overall score per metric or per system.

It is important to note that these results may not be intended for direct comparison between metrics or MT systems. Rather, they illustrate the score allocation strategies of different metrics and provide context for the meta-evaluation presented in subsequent sections. Additionally, these results highlight how the magnitude of scores can be different for same systems when evaluated by distinct metrics, as different AEMs employ different scoring strategies, reflecting their design and evaluation objectives. Some metrics adopt a more rigid approach (exact matches), while others allow for softer comparisons (embedding-based) between reference and candidate texts.

Table 3 presents the system-level scores, computed by aggregating scores across metrics and languages per system. Bold values in each row indicate the system with the highest average score for a given metric. Overall, the scores suggest that systems trained on multimodal data (ZeroMMT and Gemma-3) outperform text-only systems. Among the text-only systems (Aya-23 and Opus-MT) achieve comparable scores but consistently underperform relative to the multimodal systems.

According to document-level extension of COMET (i.e., Doc-COMET), the largest score difference is $0.05$ between Gemma-3 and Opus-MT. Standard COMET reports a similar difference of 0.04 points between the same systems (see the last two rows of Table 3). The next highest score difference, 3.89, is observed in BLEU for ZeroMMT over Aya-23. Notably, Aya-23, trained on 8 billion parameters, is still outperformed by ZeroMMT, which adapts a 1.3 billion parameter NLLB model [126] through visually conditioned language masking. Appendix C, provide more detailed comparison of metric scores according to language, system and corpora.

Overall system-level score support the Multimodal MT systems, for example, both embedding-based metrics and string-based metrics consistently rank ZeroMMT higher, while supervised metrics favor Gemma-3, the multimodal LLM, over standard MMT (i.e., ZeroMMT). This difference in metrics agreement likely arises from two factors: first, embedding-based and string-based metrics tend to favor lexically simpler translations, as seen in ZeroMMT, and may penalize the more varied vocabulary produced by Gemma-3. Second, trained neural metrics such as COMET and Doc-COMET, which are fine-tuned on human annotations, better capture linguistic variation and align with human preference, as discussed in Section 6.4.

6.2. Correlation Scores on Image Caption Test-Set

One of the ways to measure the performance of automatic evaluation metrics is find how well they correlate with human judgments [67,127]. For this, we use the THumB dataset, which collects human ratings for automatic image captioning (IC) along three dimensions: precision, recall, and overall quality. We focus on the overall quality score. Each image in THumB includes four candidate captions from different automatic captioning systems and five human-written references. For overall performance overview of AEMs, one human reference is treated as a candidate caption.12 Moreover, there are multiple correlation measures, such as Pearson, Spearman, and Kendall, each with its own score range, strengths, and limitations [128]. To ensure a fair comparison, we compute all three correlation coefficients to examine the performance and suitability of existing AEMs in the context of MMT evaluation.

Figure 2 presents the correlation scores on the THumB dataset. Overall, COMET shows the strongest alignment with human judgments at both the sentence and document levels, with Pearson correlations around 0.31–0.33, Spearman around 0.26, and Kendall around 0.19. Although the differences between Doc-COMET and COMET are not statistically significant, they are consistent across all correlation types. COMET considers both the source text and the reference when scoring MT outputs, while Doc-COMET additionally incorporates IDs as context, which likely helps produce automatic scores closer to human ratings.

Notably, the character n-gram–based metric CHRF achieves the second-highest correlation scores, outperforming embedding-based metrics (BERTScore and Doc-BERT) and showing the continued value of lexical overlap–based evaluation. The performance of CHRF aligns with recent findings from the WMT2025 Metric Shared Task, where CHRF achieved results comparable to the strongest neural submissions [96]. Surprisingly, Doc-BERT, despite incorporating additional IDs, performs worse than its sentence-level counterpart. Since BERTScore measures soft semantic similarity between embeddings, the increased token length introduced by additional context may dilute the similarity signal, making it harder to penalize semantically divergent yet fluent translations [72]. BLEU exhibits substantially lower correlations (Pearson ≈ 0.13, Spearman ≈ 0.11, Kendall ≈ 0.08), reflecting its limited alignment with human judgments.

Taken together, these results confirm that learned neural metrics provide a closer approximation to human evaluation on multimodal translation tasks, consistent with their demonstrated superiority in NMT [67]. Incorporating IDs slightly improves performance, highlighting the value of contextual information. However, additional context may not benefit all metrics; for example, BERTScore performance decreases after integrating context. Furthermore, the systematic decline in correlation values across metrics underscores the stricter nature of rank-based measures (e.g., Spearman, Kendall) and emphasizes the importance of reporting multiple correlation types to obtain a more comprehensive assessment of metric reliability [129].

6.3. Contrastive Evaluation

Contrastive or challenge sets are commonly used for targeted evaluation of NLP systems, enabling focused assessment of specific linguistic phenomena or revealing system strengths and weaknesses that are not apparent in standard test sets [96]. Unlike conventional evaluation corpora, which typically contain generic, real-world text, challenge sets include rare, adversarial, or ambiguity-focused examples. For instance, the CoMMuTE test-set [13] provides contrastive translation pairs accompanied by visual context: an ambiguous English source sentence, its correct human translation, and an incorrect (bad_reference) translation that intentionally mistranslates the ambiguous word.

To assess the robustness of metrics under adversarial conditions, we conducted a contrastive evaluation on the CoMMuTE dataset. We performed two evaluation runs: one using the correct reference and another using the bad_reference, which introduces a controlled semantic error (see Figure 1). A robust evaluation metric should assign a lower score to a system output when compared against the incorrect reference than when compared against the correct one. This setup enables us to assess not only the sensitivity of automatic metrics to subtle meaning shifts but also whether the MT systems themselves successfully resolve lexical ambiguities, as reflected by differences in metric scores across the two contrastive runs.

Figure 3 presents a heatmap of the average score differences for MT systems. The visualisation clearly separates text-only from multimodal systems. For the text-only systems (Opus-MT, a bilingual MT model, and Aya-23, an LLM-based MT system), most automatic metrics show almost no score differences between the good and bad_reference variations. In contrast, systems that incorporate visual information (ZeroMMT and Gemma-3) exhibit more pronounced differences, underscoring the usefulness of visual context for producing translations different from traditional systems. Results on the contrastive set further reinforce the overall system-level findings (Table 3), with multimodal MT systems achieving higher average scores than their text-only counterparts.

At the metric level, Doc-COMET demonstrates the highest sensitivity to quality variations, yielding slightly larger score differences than all other metrics—specifically, 2.9% more than the baseline COMET. N-gram-based metrics perform comparably to neural metrics in detecting lexical differences for multimodal systems. BLEU and CHRF, being precision-oriented metrics, tend to perform well on contrastive sets containing simple and repetitive sentences, where differences often hinge on subtle lexical changes [130]. Table 4 compute correlations between good and bad_reference scenario, where higher correlation coefficient indicate metric’s inability to detect noise and vice versa. Metrics such as COMET (r=-0.14) and Doc-COMET (r=-0.42) show strong sensitivity to corrupted references, producing low or negative correlations that suggest they sharply penalize mistranslations when reference quality declines. Similarly, chrF (r=0.45) and BERT-based metrics ($r\approx 0.41$–$0.47$) demonstrate moderate sensitivity, while BLEU (r=0.62) and TER (r=0.84) preserve relatively higher correlations, indicating weaker responsiveness to reference errors. In this framing, low correlations are not evidence of metric unreliability but rather a sign of robustness, as they demonstrate the metric’s capacity to detect and react to poor reference quality rather than smoothing over it.

In summary, both surface-level and learned metrics were able to distinguish translation variations in the contrastive evaluation. Doc-COMET, having IDs as context and source translation in consideration, shows better suitability for multimodal task. Overall COMET, as task-specific trained metrics, demonstrated higher sensitivity compared to BERTScore, which relies on general-purpose pre-trained embeddings.

6.4. Human Evaluation

To analyse how visual context affects human perception of translation quality, a human evaluation study was conducted on a subset of the CoMMuTE testset using the Appraise platform. The human evaluator rated translations on a 0–100 scale under two conditions: text-only (source and hypothesis only) and multimodal (source, hypothesis, and image). The interface for multimodal evaluation is shown in Appendix B. Each instance was evaluated multiple times to obtain robust averages. The evaluator was a bilingual German–English speaker with German as their native language and an educational background in computational linguistics.

Table 5 presents the averages of score assigned by the human evaluator and the automatic metrics, with correlation score in last column. For the without-image (w/o img) condition, all systems received an average human score above 90 on the 0–100 scale for English→German translation. This skewed average is obvious for two reasons. First, the MT systems—including the LLM-based models (Aya-23 and Gemma-3)—were trained on large, high-quality German corpora, and were tested on relatively simple image-caption translations. Second, without access to the image, the human annotator cannot judge whether the system resolved the lexical ambiguity correctly, and therefore tended to assign high scores to translations that were lexically and syntactically well-formed. However, when the image context was available, the average score drops, indicating the human evaluator revised the ratings.

This scoring strategy is visualised in Figure 4, which presents the segment-level score distributions for the human evaluator and the automatic metrics. The diagonal plots show the score distributions of individual metrics, while the off-diagonal plots illustrate pairwise comparisons between metrics. The distributions reveal clear shifts in both human and automatic scores depending on the availability of image during evaluation. The score distribution of human evaluator confirms that for text-only conditions the human evaluator tended to assign relatively high scores, particularly to fluent translations with plausible semantics. When the image was available, the score distribution became more dispersed, with pronounced peaks at both 0 and 100. These findings reinforce that even small mistranslations or subtle lexical errors can have a substantial impact on perceived translation quality when expert evaluators have access to visual context, in line with previous observations by Lala [12].

Among the AEMs, COMET and Doc-COMET exhibit similar bimodal distributions, following the human scoring behaviour. This pattern reflects cases where MT systems failed to resolve lexical ambiguities correctly, for example, Aya-23 and Opus-MT received scores at both extremes depending on whether ambiguity was resolved or not, as shown in Table 5. String-based metrics such as BLEU and CHRF show a comparable bimodal trend, distinguishing between text-only and multimodal MT outputs. However, due to their reliance on exact surface-form matching and sensitivity to lexical variation, these metrics produce lower absolute score ranges than neural metrics, resulting in larger peaks at the lower end of the scale.

In contrast, embedding-based metrics such as BERTScore and Doc-BERT assign more generous scores, leading to more normally distributed outputs. Because these metrics rely on cosine similarity between contextual embeddings rather than strict n-gram overlap, they may overestimate semantic similarity in some cases. This makes them vulnerable to structurally similar but semantically divergent translations (e.g., “man bites dog” vs. “dog bites man”), as previously highlighted by Kaster et al. [131].

Among the systems, Gemma-3 (a multimodal LLM) and ZeroMMT (an MMT system) achieved the highest and second-highest human scores (Table 5), respectively. In contrast, Aya-23 and Opus-MT exhibited substantial drops in human scores when evaluated with the associated image, indicating that these text-only systems often failed to resolve visual ambiguities that became apparent once the image was visible.

In summary, the human study confirms that context plays a critical role for both human evaluators and automatic metrics, as both human and some AEMs remained sensitive to visual context. The annotator substantially revised their judgments once the image was available, demonstrating that text-only evaluation can mask important differences between multimodal and text-only MT systems. Likewise, trained metrics—particularly learned neural metrics—show improved performance and stronger correlations with human scores when visual information is incorporated via IDs.

It is important to note, however, that the human evaluation was based on source–candidate comparison, whereas AEMs performed reference-based evaluation, with COMET being an exception in that it also considers the source text. Since each metric captures different aspects of translation quality (e.g., lexical overlap, semantic similarity, and syntactic structure), and lexical matching remains a common foundational signal across metrics [131], combining multiple metrics or reporting them jointly can provide a more comprehensive and reliable assessment.

7. Discussion and Conclusion

This study examined how well existing AEMs perform for the MMT evaluation task and whether incorporating IDs as context improves their reliability. We evaluated a range of metrics (including n-gram–based, embedding-based, and supervised neural metrics) across multiple test sets, comparing their behaviour on translations produced by text-only and multimodal MT systems.

Across a series of experiments on multimodal evaluation sets, supervised neural metrics demonstrated better performance. In particular, Doc-COMET, which incorporates visual semantics extracted from associated images, consistently outperformed both string-based and embedding-based baseline models. At the system level, Doc-COMET assigned the highest scores to Gemma-3, a multimodal LLM-based system, with a clear margin over text-only baselines. This outcome was also reflected in the human evaluation results (Table 5). The performance of Doc-COMET on both contrastive evaluation and correlation with human judgments highlights the importance of contextual information for assessing MMT output. Notably, Doc-COMET’s evaluation strategy resembles human assessment practices, as it considers the source text, the candidate translation, reference translations, and image-derived semantic information. These supervised metrics achieved the highest agreement with human ratings across all correlation measures.

In response to the research question (whether adding visual context in the form of IDs benefits neural metrics during automatic evaluation), the results indicate moderate but consistent improvements over context-agnostic counterparts, suggesting that contextual information can improve the reliability of metric predictions. However, adding visual context does not guarantee better performance. The effectiveness of contextual integration depends on the metric architecture. Doc-BERT, for example, did not outperform BERTScore despite using additional context, which contrasts with earlier findings [86,92].In most cases, Doc-BERT performed worse than its sentence-level variant, suggesting that the longer input sequences introduced by IDs may dilute semantic signals rather than strengthen them. Moreover, both BERT-based metrics underperformed compared to n-gram-based metrics, particularly in cases where incorrect translations remained lexically similar to the reference [72].

Overall, the performance gap between n-gram metrics and learned metrics (supervised or embedding-based) was small, and all metrics showed positive correlation with human ratings. Yet the contrastive and human evaluation results on the CoMMuTE corpus revealed clear differences in how metrics respond to visually grounded meaning. All metrics except COMET and Doc-COMET showed positive correlation with bad_reference, indicating limited ability to detect small but meaningful semantic changes that humans heavily penalised. In contrast, Doc-COMET showed the strongest ability to capture quality degradation under corrupted references. While n-gram metrics (BLEU, TER, CHRF) handled lexical mismatches reasonably well—sometimes better than embedding-based metrics in short-caption data—their reliance on surface overlap limits their ability to detect deeper semantic errors. The human evaluation further confirmed the importance of visual context: without images, the annotator assigned near-ceiling scores to all systems, but once images were shown, scores changed sharply, especially for text-only systems. This shows that textual fluency alone can mask semantic inaccuracies that become clear only when visual grounding is considered. Multimodal systems achieved the highest human scores, and trained metrics that used IDs showed better alignment with these expert judgments.

In summary, three main conclusions emerge:

-First, visual context plays a meaningful role in both translation generation and evaluation, particularly for content involving ambiguity or visually grounded meaning. Models incorporating visual information (Gemma-3 and ZeroMT) achieved higher average human ratings, and the inclusion of image context led to improved alignment between metric scores and human judgments.

-Second, learned metrics such as COMET and Doc-COMET are better suited than embedding-based metrics such as BERTScore and Doc-BERT for evaluating MMT quality, as they reflect human preferences more closely and better distinguish between multimodal and text-only systems.

-Third, the mixed behaviour of different metrics highlights the need for multi-metric evaluation. Each metric captures different aspects of translation quality, and no single metric is sufficient for a comprehensive assessment. Although learned metrics perform well, n-gram metrics remain essential for low-resource scenarios where language-independent tools are required [114]. Simple parameter-free averaging of multiple metrics has also been shown to improve correlation with human judgments by up to 13% [131].

While the findings support the usefulness of IDs in multimodal evaluation, the study also reveals limitations. The improvements from contextual integration remain moderate, and not all metric architectures benefit from additional inputs. Moreover, the human evaluation was small in scale, limiting broader generalisation. Future work should expand human evaluation efforts and explore more effective ways of integrating visual grounding directly into metric design, beyond the shallow use of descriptive context adopted here. One possible direction is to compute embedding similarity using multimodal-multilingual models (e.g., ViLBERT, CLIP) instead of unimodal BERT, as demonstrated in related work on image caption evaluation [15,16]. Another avenue is to initialise evaluation metrics from vision–language models rather than text-only language models, followed by fine-tuning on human evaluation data, to test whether multimodal pre-training helps reduce token bias and ambiguity in metric predictions.