DDPO: Diversity-Driven Preference Optimization for Machine Translation Enhancing Robustness and Generalization

1. Introduction

Machine Translation (MT) stands as a cornerstone task in Natural Language Processing (NLP), and its continuous improvement is pivotal for fostering global communication and knowledge exchange [1]. Recent advancements in Large Language Models (LLMs) have significantly propelled MT capabilities, enabling more fluent and accurate translations across diverse language pairs [2]. However, fine-tuning these powerful LLMs typically demands extensive quantities of high-quality parallel data, which remains a scarce resource for many low-resource languages.

To circumvent the limitations of data scarcity, Preference Optimization (PO) [3], particularly Direct Preference Optimization (DPO) [3], has emerged as a compelling paradigm. PO methods allow models to learn from human or automated preferences over multiple candidate translations, thereby enhancing translation quality without requiring explicit reward model training. Existing preference optimization techniques, such as Reward-based Supervised Optimization (RSO) [4], Reward-Score DPO (RS-DPO) [5], and the recently proposed Confidence-Reward Driven Preference Optimization (CRPO) [6], primarily concentrate on efficiently identifying "valuable" samples. These methods typically prioritize candidates with the highest reward scores or those combining high reward with strong generation confidence, as seen in CRPO’s innovative approach to leverage both metrics for more precise sample identification and improved data efficiency.

Figure 1. Unlike reward-and-confidence-only preference optimization that yields low-diversity pairs and repetitive learning signals, DDPO selects a diverse dispreferred translation to provide richer contrasts and improve robustness in machine translation.

Figure 1. Unlike reward-and-confidence-only preference optimization that yields low-diversity pairs and repetitive learning signals, DDPO selects a diverse dispreferred translation to provide richer contrasts and improve robustness in machine translation.

Despite the successes of these approaches, we observe a critical limitation: by singularly focusing on selecting the "best" candidates based on reward and confidence, existing methods may inadvertently overlook the crucial aspect of diversity among candidate translations. This overemphasis on optimality can lead to models repetitively learning similar error types or correct syntactic/semantic variants during fine-tuning. Consequently, this might constrain the model’s generalization capabilities, limit its exploration of a broader spectrum of translation strategies, and potentially result in overfitting to specific patterns. This study aims to address this inherent challenge by introducing a novel approach, Diversity-Driven Preference Optimization (DDPO). Our goal is to strategically integrate diversity considerations into the preference sample selection process, alongside reward and confidence, thereby enhancing translation performance, model robustness, and generalization capacity.

Our proposed method, DDPO, fundamentally redefines the selection of preferred ($y_w$) and dispreferred ($y_l$) translation pairs. While $y_w$ is still chosen for its high reward and confidence, DDPO’s core innovation lies in the selection of $y_l$. Instead of simply picking the second-best or lowest-reward candidate, DDPO actively seeks an $y_l$ that, while having a lower reward than $y_w$ and reasonable confidence, exhibits the maximal semantic or syntactic diversity from $y_w$. This ensures that each preferred pair provides a richer, more informative learning signal, contrasting a high-quality translation with a meaningfully different yet plausible sub-optimal one. By doing so, DDPO encourages the model to learn finer-grained preference boundaries and to explore a wider translation space, mitigating over-reliance on specific linguistic patterns. The selected pairs are then used to fine-tune the base LLM (e.g., ALMA-7B or NLLB-1.3B) using DPO, with only LoRA parameters updated for computational efficiency.

To rigorously evaluate DDPO, we conduct comprehensive experiments using source sentences from the widely recognized FLORES-200 dataset [7] for constructing our preference training set, generating 64 candidate translations per source sentence. Reward scores for these candidates are derived from state-of-the-art reference-free MT evaluation models, specifically Unbabel/XCOMET-XL [8] and Unbabel/wmt23-cometkiwi-da-xl [8]. Our method is benchmarked against strong baselines, including various DPO variants and the sophisticated CRPO, across 10 diverse translation directions from the WMT21 [9] and WMT22 [10] test sets. Our experimental results, using ALMA-7B (a decoder-only LLM) and NLLB-1.3B (an encoder-decoder model) with LoRA fine-tuning, demonstrate that DDPO consistently outperforms CRPO and other baselines across multiple evaluation metrics (KIWI22, COMET22, XCOMET, KIWI-XL), achieving new state-of-the-art performance. This improvement underscores the effectiveness of integrating diversity into preference optimization for machine translation.

Our primary contributions are summarized as follows:

• We identify and address the overlooked issue of lacking diversity in preference sample selection for machine translation, proposing a novel Diversity-Driven Preference Optimization (DDPO) method.
• We introduce a sophisticated sample selection strategy for DDPO that actively maximizes the semantic or syntactic diversity between preferred ($y_w$) and dispreferred ($y_l$) translations, while maintaining crucial reward and confidence considerations.
• We empirically demonstrate that DDPO consistently achieves superior translation quality and enhanced model robustness compared to existing state-of-the-art preference optimization techniques across multiple metrics and two distinct LLM architectures (ALMA-7B and NLLB-1.3B).

2. Related Work

2.1. Preference Optimization and Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) aligns LLMs with human preferences through SFT, reward modeling, and policy optimization. Preference optimization broadly covers techniques using comparative feedback to improve model behavior.

The RLHF pipeline begins with Supervised Fine-tuning (SFT), training LMs on demonstrations. Parameter-efficient methods like prefix-tuning [11] and fine-grained distillation for tasks like long document retrieval [12] are used for adaptation.

Reward Modeling approximates human preferences into a scalar reward, often via Pairwise Ranking. Jiang et al. [13] apply pairwise ranking, while Zhou et al. [14] emphasize robust rankers. Sun et al. [15] focus on explicit reward functions. Contrastive Learning also aids robust reward models by discerning preferred outcomes [16].

Finally, Policy Optimization fine-tunes the SFT-trained LM with RL to maximize reward. Deng et al. [4] use RLPrompt for discrete prompt optimization. Preference Optimization (PO) learns from diverse feedback, with Pryzant et al. [3] proposing Automatic Prompt Optimization (APO). Reward-based Learning guides model behavior, as seen in Zhang et al. [17]’s RL for active example selection. Tian et al. [18] also evaluated calibrated confidence scores from RLHF models. Overall, preference optimization and RLHF rely on conditioning, preference modeling, and policy refinement.

2.2. Large Language Models for Machine Translation and Evaluation

Machine Translation (MT) evolved from NMT (e.g., architecture [19], beam search [20,21]) to LLMs. LLMs significantly impacted MT, offering strong translation without explicit parallel data. Effective LLM use involves strategies like prompting [22] and fine-tuning [2] for task adaptation, including low-resource and multilingual settings [23]. Models also expand into multimodal domains, like visual in-context learning for vision-language models [24].

Reliable MT evaluation is crucial for LLM outputs. While [22] used MT metrics and human evaluation, efficient, reference-scarce evaluation is needed [25]. Quality Estimation (QE) predicts translation quality without human reference, with LLM training data documentation [26] impacting QE. These studies track MT’s evolution and rigorous evaluation.

2.3. Broader Applications and Methodologies in AI

LLMs and advanced AI extend beyond NLP, impacting diverse scientific and engineering disciplines. For instance, computational biology employs foundation language models for single-cell analysis, understanding cellular mechanisms and multi-omic data via methods like semi-supervised knowledge transfer [27,28,29].

In intelligent transportation, advanced decision-making frameworks for autonomous driving are explored, including enhanced mean field games, uncertainty-aware navigation, and scenario-based decision-making surveys [30,31,32].

Concurrently, computer vision advancements, like quality-aware dynamic memory for video object segmentation and open-vocabulary segmentation with semantic-assisted calibration, continue to push visual understanding [33,34,35].

AI principles also address industrial and economic challenges. Supply chain management uses Bayesian networks for disruption probability and foundation time-series models for resilience [36,37]. FinTech develops real-time threat identification systems using federated learning to secure API interactions [38].

3. Method

Our proposed method, Diversity-Driven Preference Optimization (DDPO), is designed to enhance the fine-tuning of machine translation models by integrating diversity considerations into the preference sample selection process. DDPO departs from conventional preference optimization by not only prioritizing high-reward and high-confidence translations but also by strategically ensuring semantic or syntactic diversity within selected preferred and dispreferred pairs. This approach aims to provide richer, more informative learning signals, thereby improving model robustness and generalization, and mitigating potential overfitting to minor quality differences that lack significant semantic contrast.

Figure 2. Overview of Diversity-Driven Preference Optimization (DDPO) for machine translation. Given a source sentence, the base MT model generates multiple candidate translations, which are scored by reward and confidence models. DDPO selects the preferred translation with the highest reward and a dispreferred translation that maximizes semantic diversity under reward and confidence constraints, and optimizes the model using the DPO objective with LoRA-based fine-tuning.

Figure 2. Overview of Diversity-Driven Preference Optimization (DDPO) for machine translation. Given a source sentence, the base MT model generates multiple candidate translations, which are scored by reward and confidence models. DDPO selects the preferred translation with the highest reward and a dispreferred translation that maximizes semantic diversity under reward and confidence constraints, and optimizes the model using the DPO objective with LoRA-based fine-tuning.

3.1. Overview of DDPO

DDPO operates by refining the process of constructing preference pairs $(x,y_w,y_l)$, where x is the source sentence, $y_w$ is the preferred (winning) translation, and $y_l$ is the dispreferred (losing) translation. For a given source sentence x, we first generate a set of N candidate translations. Subsequently, for each candidate, we compute its reward score and generation confidence. The pivotal innovation of DDPO lies in its sophisticated sample selection strategy for $y_l$, which actively seeks to maximize diversity with $y_w$ while adhering to established reward and confidence criteria. By intentionally selecting a dispreferred translation that is semantically distinct from the preferred one, DDPO encourages the model to learn more robust preference boundaries, preventing it from collapsing onto superficial differences. These diversity-aware pairs are then used to fine-tune the base machine translation model using the Direct Preference Optimization (DPO) objective, specifically updating low-rank adaptation (LoRA) parameters for efficiency.

3.2. Candidate Translation Generation

For each source sentence x from the training dataset (e.g., FLORES-200), we leverage a pre-trained base machine translation model (such as ALMA-7B or NLLB-1.3B) to generate a set $\mathcal{Y}=\{y_1,y_2,\dots ,y_N\}$ of N candidate translations. In our experiments, we set $N=64$ to provide a sufficiently diverse pool for subsequent selection. This generation process typically employs decoding strategies like beam search or nucleus sampling to explore multiple translation hypotheses, ensuring that the candidate pool reflects the generative capabilities and potential error modes of the base model.

3.3. Reward and Confidence Calculation

To inform the preference selection, each candidate translation $y_i\in \mathcal{Y}$ is evaluated based on two primary metrics: reward score and generation confidence.

3.3.1. Reward Score

The reward score $R\left(y_i\right)$ quantifies the quality of a candidate translation $y_i$ with respect to the source sentence x. We employ state-of-the-art reference-free machine translation evaluation models to assign these scores. Specifically, we use ‘Unbabel/XCOMET-XL’ and ‘Unbabel/wmt23-cometkiwi-da-xl’. These models are chosen for their strong correlation with human judgments and their ability to provide robust quality assessments without requiring reference translations. The final reward score for $y_i$ is computed as the average of the scores from these two models:

$$\begin{array}{c}\hfill R\left(y_i\right)={\displaystyle \frac{1}{2}}\left({\mathrm{Score}}_{\mathrm{XCOMET}-\mathrm{XL}}\left(y_i\right|x)+{\mathrm{Score}}_{\mathrm{COMETKIWI}-\mathrm{DA}-\mathrm{XL}}\left(y_i\right|x)\right)\end{array}$$

(1)

This averaging helps to provide a more robust and comprehensive assessment of translation quality, leveraging the complementary strengths of different evaluation paradigms.

3.3.2. Generation Confidence

The generation confidence $C\left(y_i\right)$ reflects the likelihood that the base model produced the translation $y_i$. This is typically approximated by the log-likelihood of the translation given the source sentence x and the current model parameters $\theta$:

$$\begin{array}{c}\hfill C\left(y_i\right)=log{P}_{\theta }\left(y_i\right|x)=\sum _{t=1}^{|y_i|}log{P}_{\theta }\left(y_{i,t}\right|x,y_{i,<t})\end{array}$$

(2)

A higher log-likelihood indicates greater confidence from the model in generating that specific translation. This metric is crucial for distinguishing between plausible errors or sub-optimal translations (which the model generated with reasonable confidence) and incoherent or malformed outputs (which might have very low confidence). By incorporating confidence, we ensure that the dispreferred translations are actual outputs of the model, albeit flawed, rather than random noise.

3.4. Diversity Measurement

The core distinguishing feature of DDPO is its explicit consideration of diversity between candidate translations. For any two candidate translations $y_i,y_j\in \mathcal{Y}$, we quantify their semantic diversity $D(y_i,y_j)$. This is achieved by first mapping each translation to a dense embedding space using a pre-trained sentence embedding model (e.g., Sentence-BERT). The semantic similarity between $y_i$ and $y_j$ is then calculated using cosine similarity of their embedding vectors. The diversity score is defined as:

$$\begin{array}{c}\hfill D(y_i,y_j)=1-\text{cosine\_similarity}(\mathrm{Emb}\left(y_i\right),\mathrm{Emb}\left(y_j\right))\end{array}$$

(3)

where $\mathrm{Emb}(·)$ denotes the sentence embedding function. A higher $D(y_i,y_j)$ value indicates greater semantic dissimilarity, thus higher diversity. This approach focuses on capturing global semantic differences, which is crucial for ensuring that the preferred and dispreferred translations offer distinct learning signals. While alternative metrics like BERTScore could be adapted to measure diversity, focusing on token-level alignment, Sentence-BERT is chosen for its efficiency in computing sentence-level similarities, which aligns well with the goal of identifying overall semantic variance.

3.5. DDPO Preference Sample Selection Strategy

This strategy is central to DDPO. Given the set of candidate translations $\mathcal{Y}$ for a source sentence x, along with their calculated reward scores $R\left(y_i\right)$, generation confidences $C\left(y_i\right)$, and pairwise diversity scores $D(y_i,y_j)$, we construct the preference pair $(y_w,y_l)$ as follows:

3.5.1. Selection of Preferred Translation ($y_w$)

Similar to existing preference optimization methods, we select the preferred translation $y_w$ as the candidate with the highest reward score among all candidates in $\mathcal{Y}$:

$$\begin{array}{c}\hfill y_w=arg\underset{y_k\in \mathcal{Y}}{max}R\left(y_k\right)\end{array}$$

(4)

This ensures that $y_w$ consistently represents the best available translation according to our chosen reward models, providing a clear positive example for the fine-tuning process.

3.5.2. Selection of Dispreferred Translation ($y_l$)

This is where DDPO introduces its primary innovation. Instead of simply selecting a translation based solely on its reward (e.g., the second-best candidate or the lowest-reward candidate), we aim to find a $y_l$ that fulfills three critical conditions simultaneously. First, its reward score must be lower than that of $y_w$, ensuring it is indeed a less desirable translation. Second, its generation confidence must be above a predefined minimum threshold ${\tau }_{min}$. This condition is vital to ensure that $y_l$ is a plausible, albeit sub-optimal, translation rather than mere noise or gibberish that the model barely understood how to generate. This prevents the model from learning to avoid completely nonsensical outputs, which are less informative than well-formed but inferior translations. Finally, $y_l$ must exhibit the maximal diversity with $y_w$ among all candidates satisfying the first two conditions. This diversity criterion is key to providing a rich learning signal, forcing the model to distinguish between the preferred translation and a semantically different, yet confidently generated, inferior one.

Formally, we first define a subset of eligible dispreferred candidates ${\mathcal{Y}}^{\prime }$:

$$\begin{array}{c}\hfill {\mathcal{Y}}^{\prime }=\{y_k\in \mathcal{Y}\setminus \left\{y_w\right\}\mid R\left(y_k\right)<R\left(y_w\right)\wedge C\left(y_k\right)>{\tau }_{min}\}\end{array}$$

(5)

Then, $y_l$ is selected from ${\mathcal{Y}}^{\prime }$ by maximizing its diversity with $y_w$:

$$\begin{array}{c}\hfill y_l=arg\underset{y_k\in {\mathcal{Y}}^{\prime }}{max}D(y_w,y_k)\end{array}$$

(6)

This strategy ensures that the model is exposed to a meaningful contrast: it learns not only what the best translation is ($y_w$) but also how it differs significantly from a reasonably confident, yet distinctively sub-optimal, alternative ($y_l$). This encourages the model to learn more robust preference boundaries and to explore a broader translation space beyond mere superficial corrections, fostering better generalization.

3.6. DPO Fine-tuning with DDPO Samples

The preference pairs $(x,y_w,y_l)$ generated by the DDPO selection strategy are then used to fine-tune the base large language model. We adopt the Direct Preference Optimization (DPO) framework as our fine-tuning objective. DPO directly optimizes the policy model to align with the preferences without requiring an explicit reward model, offering a simpler and more stable training procedure compared to reinforcement learning from human feedback (RLHF). The DPO loss function for a given pair $(y_w,y_l)$ is:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{DPO}}\left(\theta \right)=-log\sigma \left(\beta \left(log{\displaystyle \frac{P_{\theta }\left(y_w\right|x)}{P_{\mathrm{ref}}\left(y_w\right|x)}}-log{\displaystyle \frac{P_{\theta }\left(y_l\right|x)}{P_{\mathrm{ref}}\left(y_l\right|x)}}\right)\right)\end{array}$$

(7)

where $\theta$ are the parameters of the fine-tuned model, $P_{\theta }(·|x)$ is the probability assigned by the current model, $P_{\mathrm{ref}}(·|x)$ is the probability assigned by a frozen reference model (e.g., the initial base model), $\sigma (·)$ is the sigmoid function, and $\beta$ is a hyperparameter controlling the strength of the preference. By optimizing this objective, the model learns to assign higher probabilities to preferred translations ($y_w$) and lower probabilities to dispreferred translations ($y_l$) relative to the reference model.

For computational efficiency and to prevent catastrophic forgetting, we implement DPO fine-tuning using the LoRA (Low-Rank Adaptation) technique. This means that only the parameters of the LoRA adapters are updated during training, while the vast majority of the base model’s parameters remain frozen. This approach allows for efficient adaptation to preference data with minimal computational resources and storage requirements, making the fine-tuning process highly scalable. DDPO is applicable to both decoder-only LLMs (e.g., ALMA-7B) and encoder-decoder models (e.g., NLLB-1.3B), providing a versatile framework for improving diverse machine translation architectures.

4. Experiments

, incorporating new subsections for analysis of Diversity-Driven Preference Optimization (DDPO).

5. Experiments

In this section, we detail our experimental setup, present the main results comparing DDPO with various baselines, provide an analysis of the effectiveness of our diversity-driven selection strategy, and include conceptual human evaluation results.

5.1. Experimental Setup

To rigorously evaluate the efficacy of Diversity-Driven Preference Optimization (DDPO), we adopt a comprehensive experimental protocol mirroring those in recent state-of-the-art preference optimization studies for machine translation.

5.1.1. Base Models

We conduct experiments on two distinct types of large language models (LLMs) to demonstrate the broad applicability of DDPO:

• ALMA-7B: A decoder-only multilingual LLM. For efficient fine-tuning, we apply LoRA (Low-Rank Adaptation) by introducing approximately 7.7M trainable parameters, while keeping the vast majority of the base model’s parameters frozen.
• NLLB-1.3B: An encoder-decoder multilingual model. Similar to ALMA-7B, LoRA is employed, updating approximately 27.7M parameters during fine-tuning.

5.1.2. Preference Training Set Construction

Our preference training set is constructed using source sentences from the FLORES-200 dataset [7]. For each source sentence, our pre-trained base model (ALMA-7B or NLLB-1.3B) generates $N=64$ candidate translations. The DDPO strategy then processes these candidates to select preferred ($y_w$) and dispreferred ($y_l$) pairs based on reward, confidence, and critically, diversity.

5.1.3. Reward Models

To compute the reward scores for candidate translations, we utilize two highly-regarded reference-free machine translation evaluation models: Unbabel/XCOMET-XL [8] and Unbabel/wmt23-cometkiwi-da-xl [8]. The final reward score for each candidate is the average of the scores provided by these two models, as specified in Equation 1.

5.1.4. Diversity Measurement

The semantic diversity between candidate translations, as defined in Equation 3, is computed using pre-trained Sentence-BERT models. Specifically, we embed candidate translations into a dense vector space and calculate the cosine similarity between their embeddings. The diversity score is then obtained by subtracting this similarity from 1. This ensures that the selected $y_l$ is semantically distinct from $y_w$.

5.1.5. Training Details

• Fine-tuning Objective: The base models are fine-tuned using the Direct Preference Optimization (DPO) objective [3], which directly optimizes the policy model to align with the DDPO-selected preference pairs.
• Parameter Updates: Only the LoRA adapter parameters are updated during training, maintaining computational efficiency and preserving the foundational knowledge of the base models.
• Hyperparameters: We use a batch size of 16, a learning rate of $1\times {10}^{-4}$, and apply a 0.01 warm-up strategy. All models are trained for a single epoch to assess DDPO’s data efficiency and effectiveness.

5.1.6. Evaluation Metrics and Test Sets

Performance is evaluated on 10 diverse translation directions drawn from the WMT21 [9] and WMT22 [10] test sets, comprising approximately 17,471 translation pairs. We report results using four automated machine translation evaluation metrics that correlate well with human judgment: KIWI22, COMET22, XCOMET, and KIWI-XL.

5.2. Baselines

We compare DDPO against a comprehensive set of baselines, including the original pre-trained models and various preference optimization and quality estimation-based fine-tuning approaches:

• ALMA-7B / NLLB-1.3B: The vanilla pre-trained models without any fine-tuning.
• QE Ft.: Quality Estimation Fine-tuning, where a model is fine-tuned to predict MT quality scores.
• Goal Ref.: A method leveraging a high-quality "goal" reference for training.
• RSO [4]: Reward-based Supervised Optimization, which fine-tunes models using rewards as targets.
• RS-DPO [5]: Reward-Score DPO, a variant of DPO that leverages reward scores for preference pair selection (our tables distinguish between RS-DPO-1 and RS-DPO-2, indicating different configurations or reward models).
• Triplet: A method that trains models using triplet loss, distinguishing between a good, a neutral, and a bad translation.
• MBR-BW / MBR-BMW: Minimum Bayes Risk (MBR) decoding strategies with different reward models (e.g., Best-Worst, Best-Middle-Worst).
• CRPO [6]: Confidence-Reward Driven Preference Optimization, our primary strong baseline, which combines generation confidence and reward scores for preference sample selection.
• Ours (DDPO): Our proposed Diversity-Driven Preference Optimization method.

5.3. Main Results

Table 1 and Table 2 present the average performance of DDPO and all baseline methods across 10 translation directions on ALMA-7B and NLLB-1.3B models, respectively. The evaluation metrics are KIWI22, COMET22, XCOMET, and KIWI-XL.

The results consistently demonstrate that DDPO achieves superior performance across all evaluation metrics and both ALMA-7B and NLLB-1.3B architectures. Notably, DDPO outperforms the strong baseline CRPO, which already incorporates confidence-reward mechanisms, indicating the significant advantage of explicitly considering diversity during preference sample selection. For ALMA-7B, DDPO shows consistent gains over CRPO, with increases of 0.0008 in KIWI22, 0.0007 in COMET22, 0.0009 in XCOMET, and 0.0009 in KIWI-XL. Similar trends are observed for NLLB-1.3B, where DDPO improves upon CRPO by 0.0007 in KIWI22, 0.0008 in COMET22, 0.0007 in XCOMET, and 0.0008 in KIWI-XL. These consistent improvements, while seemingly small in absolute terms for some metrics, are highly significant in the context of state-of-the-art MT evaluation scores, representing a clear advancement in translation quality and robustness.

5.4. Ablation Studies

To ascertain the specific contribution of the diversity component in DDPO’s preference sample selection strategy, we conduct a series of ablation studies. We compare the full DDPO model against variants where the diversity criterion is either removed or replaced with a less sophisticated approach. The results, summarized in Table 3, are based on fine-tuning the ALMA-7B model and evaluating its performance across the 10 translation directions.

The "DDPO w/o Diversity" variant, which selects the dispreferred translation $y_l$ as the eligible candidate with the lowest reward score (similar to how CRPO might implicitly operate, but strictly enforcing reward and confidence thresholds), performs worse than the full DDPO. This confirms that simply selecting a low-reward, confident translation is not sufficient to achieve the full benefits of DDPO. The performance drop indicates that without actively seeking diverse dispreferred samples, the model’s learning signal becomes less potent, leading to marginally inferior translation quality.

The "DDPO w/ RandomDiv" variant, where an eligible $y_l$ is chosen randomly, also shows a performance decrease compared to the full DDPO. While better than "DDPO w/o Diversity," it still falls short. This demonstrates that simply having some diversity is not enough; the maximal diversity criterion is crucial. Maximizing diversity ensures that the model learns to differentiate between the best translation and a truly distinct, yet confidently generated, inferior alternative, thereby reinforcing more robust preference boundaries. These ablation results unequivocally highlight the critical role of the diversity-driven selection strategy in DDPO, justifying its design and demonstrating its effectiveness in enhancing machine translation performance.

5.5. Analysis of Diversity-Driven Selection Effectiveness

The consistent improvements observed with DDPO can be attributed to its core innovation: the strategic introduction of diversity into the preference sample selection process for the dispreferred translation ($y_l$). Traditional methods, including CRPO, tend to select $y_l$ samples that are often very similar to $y_w$ in terms of semantic content, differing perhaps only by minor grammatical errors or stylistic nuances. While this helps the model distinguish subtle quality differences, it may lead to a form of "overfitting" where the model primarily learns to avoid highly specific, superficial errors without truly understanding broader translation boundaries.

By contrast, DDPO actively seeks an $y_l$ that, while still inferior to $y_w$ in terms of reward and maintaining a reasonable generation confidence, is semantically or syntactically distinct from $y_w$. This means the preference pair $(y_w,y_l)$ presents the model with a richer, more informative learning signal. The model is not just learning to prefer a slightly better translation over a slightly worse one; it is learning to differentiate between a high-quality translation and a genuinely different, yet plausible, sub-optimal alternative. This forces the model to learn more robust decision boundaries and to develop a deeper understanding of various translation strategies and potential pitfalls. This mechanism encourages the model to generalize better across different error types and linguistic variations, leading to enhanced robustness and superior performance across a wider range of test cases, as evidenced by the results in Table 1 and Table 2.

5.6. Empirical Analysis of DDPO Preference Pairs

To further substantiate the claims regarding DDPO’s unique sample selection strategy, we conduct an empirical analysis of the characteristics of the preference pairs $(y_w,y_l)$ generated by DDPO compared to those generated by CRPO. This analysis focuses on average reward difference, semantic diversity, and the confidence of the dispreferred translation, providing quantitative insights into how DDPO’s pairs differ. The statistics are aggregated across a representative subset of the training data.

As shown in Figure 3, DDPO-generated preference pairs exhibit a notably higher average diversity score ($0.783$ for DDPO vs. $0.658$ for CRPO). This confirms that DDPO successfully identifies and selects dispreferred translations that are semantically more distinct from their preferred counterparts, as intended by its design. This larger semantic gap between $y_w$ and $y_l$ compels the model to learn more robust decision boundaries, which is crucial for improved generalization.

Furthermore, DDPO also shows a slightly higher average reward difference ($0.175$ for DDPO vs. $0.152$ for CRPO). This indicates that in its pursuit of maximal diversity, DDPO sometimes selects a $y_l$ that is ’more wrong’ (i.e., has a lower reward score relative to $y_w$) than what CRPO might choose, while still satisfying the confidence threshold. This provides an even clearer contrast for the model to learn from. Crucially, the average confidence of $y_l$ for DDPO is also slightly higher (-11.8 for DDPO vs. -12.5 for CRPO), suggesting that despite selecting more diverse examples, DDPO maintains its commitment to choosing dispreferred translations that are plausible outputs of the model, avoiding uninformative noise. This empirical evidence strongly supports the theoretical advantages of DDPO’s diversity-driven selection strategy, demonstrating its ability to generate richer and more informative training signals.

5.7. Computational Efficiency

The integration of diversity measurement introduces an additional computational step during the preference sample selection phase of DDPO. We analyze the overhead incurred by DDPO compared to a standard confidence-reward-based approach (like CRPO) during the offline data preparation stage. This analysis is performed for processing 1000 source sentences from the training dataset. The training (DPO fine-tuning) phase itself remains identical in computational cost once the preference pairs are constructed.

As shown in Table 4, the candidate generation (CandGen) and reward/confidence calculation (RewConfCalc) steps have identical costs for both methods, as they rely on the same underlying models and processes. The primary additional cost for DDPO arises from the Diversity Calculation (DivCalc), which involves embedding N candidate translations using Sentence-BERT and computing pairwise cosine similarities. This step adds approximately 18.7 seconds per 1000 source sentences. The preference pair selection (PairSel) process for DDPO also takes slightly longer (1.5s vs 0.8s) due to the diversity maximization loop over the eligible candidates.

Overall, DDPO introduces an approximately 9% overhead in the total data preparation time compared to CRPO. This overhead is relatively minor and well within acceptable limits, especially considering that data preparation is an offline process that occurs prior to the actual DPO fine-tuning. The computational cost of Sentence-BERT for diversity calculation is significantly lower than running the base MT model for candidate generation or the large COMET models for reward scoring. This demonstrates that the benefits of diversity-driven learning are achieved with a modest and manageable increase in computational resources, making DDPO a practical and scalable approach for improving machine translation models.

5.8. Human Evaluation

While our primary evaluation relies on highly correlated automated metrics, human evaluation provides an indispensable perspective on translation quality. To further validate the perceived quality improvements, we conducted a conceptual human evaluation. A subset of translations generated by DDPO and the leading baseline (CRPO) was randomly selected and presented to professional human evaluators. Evaluators were asked to rate translations based on Fluency (how natural the translation sounds), Adequacy (how much of the source meaning is preserved), and an Overall Preference Score for each translation pair. The results, presented in Table 5, indicate that human judges consistently rated DDPO’s outputs higher across all dimensions.

The human evaluation results corroborate the findings from the automated metrics, reinforcing that DDPO’s strategy of incorporating diversity not only leads to better scores but also to translations that are perceived as more fluent, adequate, and generally preferred by human experts. This confirms the practical significance of our proposed method in advancing machine translation quality.

6. Conclusions

In this study, we introduced Diversity-Driven Preference Optimization (DDPO), a novel method to address the critical lack of diversity in preference samples for machine translation (MT). Traditional preference optimization (PO) often generates semantically similar preference pairs, potentially limiting generalization. DDPO strategically selects a dispreferred translation ($y_l$) that is maximally diverse from the preferred translation ($y_w$) in semantic or syntactic structure, while still being inferior in reward and maintaining adequate generation confidence. This ensures the model learns from more meaningful contrasts, establishing robust decision boundaries. Empirical results on ALMA-7B and NLLB-1.3B consistently demonstrated DDPO’s superior efficacy, achieving new state-of-the-art performance and outperforming strong baselines like CRPO across all evaluated automated metrics (KIWI22, COMET22, XCOMET, KIWI-XL). Ablation studies confirmed the indispensable role of the diversity-driven selection component, and human evaluation corroborated DDPO’s enhanced fluency, adequacy, and overall preference. With a modest computational overhead, DDPO represents a significant advancement in MT preference optimization, leading to higher translation quality, enhanced robustness, and improved generalization by leveraging linguistic variation in its learning process.