Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India

1. Introduction

Kokborok is one of the indigenous languages spoken by the Tiprasa people of Tripura, Northeastern India. The name itself is a compound of kok (language) and borok (people), literally meaning “language of the people.” With approximately 1.5 million speakers across Tripura, the Chittagong Hill Tracts of Bangladesh, and other parts of Northeast India, Kokborok holds official language status in Tripura alongside Bengali. It belongs to the Bodo-Garo branch of the Tibeto-Burman language family within the larger Sino-Tibetan family, and is characterised by SOV word order, postpositions, and tonal phonology.

Despite its official status and substantial speaker population, Kokborok remains severely under-resourced in natural language processing. Prior computational work has been largely limited to morphological analysis [1], POS tagging, and rule-based named entity recognition. Machine translation for Kokborok has received minimal attention: the WMT Low-Resource Indic Language Translation shared tasks [2,3,4] have included Kokborok since 2023, providing the only published NMT baselines. The best WMT 2025 submission [5] achieved BLEU scores of 6.99 (en→trp) and 2.99 (trp→en). These low scores reflect the extreme data scarcity and domain restriction of available training data, rather than any inherent untranslatability of the language.

In this paper, we address this gap with the following contributions:

• We develop KokborokMT, a significantly improved NMT system for Kokborok, by fine-tuning NLLB-200 [6] with a novel trp_Latn language token.
• We construct a 36,052-sentence parallel corpus combining professional translations from SMOL [7], WMT Bible data, and synthetic back-translations generated from Tatoeba English sentences using Gemini Flash.
• We demonstrate that synthetic data augmentation via LLM back-translation yields consistent improvements across all evaluation metrics and test conditions.
• We provide comprehensive ablation studies comparing zero-shot NLLB, fine-tuning without synthetic data (System 1), and fine-tuning with synthetic data (System 2).
• We investigate LaBSE-based quality filtering for synthetic data and report that filtering scores are unreliable for Kokborok due to the language’s absence from LaBSE’s training data, an important negative finding for the community.
• We conduct human evaluation with three annotators including a linguistic expert and a domain specialist, yielding mean scores of 3.74/5 (adequacy) and 3.70/5 (fluency).
• We release the model and evaluation scripts publicly to facilitate further research on Kokborok NLP.

2. Background and Related Work

2.1. Kokborok: Language and Script

Kokborok has a native script called Koloma, used during the reign of the Tripura kings, which is currently undergoing a revival effort. However, contemporary digital use and NLP research overwhelmingly employs the Roman script, which we adopt throughout this work. The language has nine dialects named after tribal communities (Debbarma, Reang, Jamatia, Noatia, etc.). Kokborok is an SOV language, uses suffixes as tense markers (-o for present, -kha for past, -nai for future), and employs two phonemic tones (level and high). Adjectives follow the nouns they modify, and plural markers appear sentence-finally on nouns.

2.2. Prior NLP Work on Kokborok

Computational work on Kokborok has been sparse. [1] developed a morphological analyser achieving approximately 80% accuracy. POS taggers using CRF and SVM approaches have been reported at around 84% accuracy. A rule-based NER system achieved 83% F-score [8]. Vowel recognition using LPCC features has also been explored. For machine translation, the WMT Low-Resource Indic Language Translation shared task has included Kokborok since 2023 [2,3,4], providing the only published NMT baselines. The best WMT 2025 submission [5] achieved BLEU of 6.99 (en→trp) and 2.99 (trp→en). We also note that OPUS and HuggingFace contain no publicly available Kokborok parallel data beyond the WMT shared task releases, confirming the extreme scarcity of resources for this language.

2.3. Low-Resource MT and Back-Translation

Back-translation [9] is a well-established technique for augmenting low-resource MT training data by translating monolingual target-side text into the source language. Recent work has demonstrated that LLMs can serve as effective generators of synthetic parallel data for low-resource settings. The NLLB-200 model [6] has become a standard backbone for low-resource MT fine-tuning, covering 200 languages with strong multilingual representations that transfer well to unseen languages through continued training. Adding new language tokens to multilingual models and fine-tuning on target language data is an established approach for extending coverage beyond the original training set. Fine-tuning NLLB on domain-specific or language-specific data consistently improves over zero-shot performance for languages at the margins of its coverage.

3. Data

3.1. Parallel Corpus Construction

Our training corpus combines three sources, totalling 36,052 sentence pairs after preprocessing and deduplication.

3.1.1. SMOL (9,284 Sentences)

The SMOL dataset [7] provides professionally human-translated parallel data for 123 low-resource languages. For Kokborok, SMOL comprises two sub-datasets: SMOLDOC (6,016 sentences), consisting of LLM-generated English documents on diverse topics professionally translated into Kokborok; and GATITOS (4,211 sentences), a token-level resource. An additional 57 SMOLSENT sentences were found to have reversed source and target columns during preprocessing and were corrected prior to use. SMOL represents the highest-quality component of our training data, covering diverse domains including health, education, culture, technology, and everyday conversation.

3.1.2. WMT Bible Corpus (1,769 Sentences)

The WMT Low-Resource Indic Language Translation shared task [4] provides 2,269 Kokborok-English parallel sentence pairs derived from Bible translations. We reserve 500 sentences as a held-out test set and use the remaining 1,769 for training. While this corpus is domain-restricted, it provides additional training signal and enables direct comparison with prior WMT baselines on the same test distribution.

3.1.3. Synthetic Back-Translation (24,999 Sentences)

We generate synthetic en→trp parallel data using English source sentences from the Tatoeba project (agentlans/tatoeba-english-translations on HuggingFace) [10], a well-known multilingual sentence collection used widely in MT research. We apply a length filter retaining sentences between 5 and 20 words, followed by deduplication using exact string matching, yielding 25,000 unique English sentences. Kokborok translations were generated in batches using the Google Gemini Flash API (gemini-2.5-flash-preview model) with the following system instruction: “You are a professional English to Kokborok translator. Translate each line accurately. Maintain the line order. Output ONLY the translations.” The total API cost was approximately INR 600 (USD $7). Tatoeba sentences cover everyday conversational and factual domains, complementing the more formal register of SMOLDOC.

3.2. Quality Filtering Investigation

Following standard practice in synthetic MT pipelines, we investigated LaBSE-based quality filtering [11] to identify and remove low-quality synthetic pairs. We computed cosine similarity between LaBSE embeddings of English source and Kokborok target sentences across all 24,999 pairs. The resulting score distribution (mean: 0.287, std: 0.216) was substantially lower than typical cross-lingual similarity scores for supported languages. Manual inspection of pairs across all score ranges confirmed that translation quality was consistently acceptable even at very low similarity scores (e.g., 0.04–0.15). We attribute the low scores to Kokborok’s absence from LaBSE’s training languages, rendering the embeddings unreliable for cross-lingual alignment with Kokborok. We therefore retain all 24,999 synthetic pairs without filtering and report this as a cautionary finding: LaBSE-based quality filtering is not applicable to languages absent from the model’s training data.

3.3. Data Splits and Deduplication

We construct evaluation sets from the two highest-quality sources:

• SMOL Test Set (500 sentences): Randomly sampled from SMOLDOC sentences only, ensuring domain diversity and professional translation quality.
• WMT Test Set (499 sentences): Randomly sampled from the WMT Bible corpus, enabling direct comparison with WMT shared task results.
• Development Set (500 sentences): Randomly sampled from remaining SMOL sentences after test extraction.

Zero overlap between training data (including synthetic pairs) and all test sets is verified by exact English-side string matching prior to any model training. The final training set contains 36,052 sentence pairs across all three sources.

3.4. Data Statistics

Table 1 summarises the corpus composition. Figure 1 (right) shows the relative contribution of each source.

4. Methodology

4.1. Base Model and Language Token

We fine-tune facebook/nllb-200-distilled-600M [6], a 600M parameter sequence-to-sequence transformer covering 200 languages. Kokborok is not among NLLB’s supported languages. We add a new special token trp_Latn to the tokenizer vocabulary (assigned ID 256204) and resize the model’s embedding matrix accordingly. This allows the model to condition generation on Kokborok as a distinct target language while leveraging representations from typologically related Tibeto-Burman languages already present in NLLB such as Burmese and Tibetan.

4.2. Training Setup

We train both translation directions simultaneously by concatenating the original and direction-flipped datasets, yielding 72,096 training pairs. This joint training approach encourages shared representations for both directions.

4.2.1. Hyperparameters

We train for 10 epochs using the AdamW optimiser with learning rate 2e-5, linear warmup over 500 steps, weight decay 0.01, and batch size 32. Mixed precision training (fp16) is used throughout. Maximum sequence length is 128 tokens. Training is conducted on a single A40 GPU, completing in approximately 3.5 hours for System 2 and 1.1 hours for System 1.

4.2.2. Model Selection

Checkpoints are saved at each epoch and the best model is selected based on validation loss on the SMOL development set. For both systems, validation loss continued to decrease through epoch 10, with final checkpoints selected (System 2 val loss: 0.2422; System 1 val loss: 0.2278).

4.3. Experimental Conditions

We evaluate three systems:

• Zero-Shot NLLB: The base NLLB-200-distilled-600M model with trp_Latn token added but no fine-tuning.
• System 1 (No BT): Fine-tuned on SMOL + WMT only (11,053 pairs; 23,098 with both directions). No synthetic data.
• System 2 (Full): Fine-tuned on SMOL + WMT + Gemini synthetic data (36,052 pairs; 72,096 with both directions). Primary system.

5. Evaluation

5.1. Automatic Metrics

We evaluate using a comprehensive suite of automatic metrics matching the WMT shared task evaluation protocol [4]:

• BLEU [12]: Computed using sacreBLEU [13] with default tokenisation.
• chrF [14]: Character n-gram F-score via sacreBLEU.
• ROUGE-L [15]: Longest common subsequence F-measure.
• METEOR [16]: Alignment-based metric using WordNet.
• TER [17]: Translation edit rate (lower is better).
• Cosine Similarity: Semantic similarity via LaBSE embeddings [11].
• COMET [18]: Neural evaluation using Unbabel/wmt22-comet-da.

All systems use beam search with beam size 4. We evaluate both directions on both test sets (four conditions per system).

5.2. Human Evaluation

5.2.1. Data

Human evaluation was conducted on 50 en→trp translations generated by System 2. Source sentences were sampled from the Tatoeba dataset [10] (agentlans/tatoeba-english-translations, HuggingFace), separately from all training and automatic evaluation data. From 6,765,220 total rows, we first deduplicated on the English column (1,417,346 unique sentences), then filtered for readability ≥ 2.5 and quality ≥ 0.0, yielding 222,317 candidate sentences. We randomly sampled 50 sentences (random_state=99) covering everyday conversational and factual domains.

5.2.2. Annotators

Three annotators independently rated all 50 translations: (1) a linguistic expert with expertise in Kokborok language structure; (2) a native Kokborok speaker; and (3) a native researcher specialising in Kokborok linguistics. Annotators were not informed of each other’s ratings.

5.2.3. Criteria

Each translation was rated on two 1–5 scales: Adequacy (does the translation preserve the meaning of the source?) and Fluency (is the translation natural and grammatically correct?), following standard MT human evaluation practice.

5.2.4. Results

Table 2 presents individual and aggregate scores. Agreement between the two trained evaluators (linguistic expert and native researcher) was substantial ($\kappa =0.67$ for both adequacy and fluency), while the untrained native speaker showed lower agreement ($\kappa =0.13$), consistent with known scale interpretation variability among non-expert annotators in MT human evaluation [19]. Mean scores across all annotators of 3.74/5 (adequacy) and 3.70/5 (fluency) indicate that KokborokMT successfully preserves meaning in most cases and produces largely natural output.

5.3. Reproducibility

Full sacreBLEU signatures for System 2 are provided in Appendix A for exact reproducibility.

5.4. Automatic Evaluation Results

Table 3 presents full automatic evaluation results. Figure 1 (centre) visualises BLEU scores across systems.

6. Analysis

6.1. Impact of Fine-Tuning

Zero-shot NLLB produces near-zero BLEU scores (0.09–0.63) for both directions, confirming that Kokborok lies outside NLLB’s effective coverage despite the addition of the trp_Latn token. Fine-tuning with even a modest gold corpus (System 1, ∼11k pairs) produces dramatic improvements, reaching BLEU 32.91 (trp→en, SMOL) and 14.87 (en→trp, WMT). This represents a roughly 30× improvement over zero-shot, demonstrating the critical importance of even modest amounts of high-quality parallel data for languages absent from multilingual model coverage.

6.2. Impact of Synthetic Back-Translation

System 2 consistently outperforms System 1 across all four evaluation conditions and all metrics. The gains are most pronounced for trp→en on the SMOL test set (+5.65 BLEU), where the model benefits most from the additional Tatoeba-derived training signal. For en→trp, gains are more modest (+1.90 BLEU on SMOL, +2.43 on WMT), reflecting the asymmetric nature of our synthetic data: Gemini translated English → Kokborok, so the synthetic data primarily strengthens the en→trp direction at training time, yet the bidirectional training setup propagates improvements to both directions. The consistent improvements validate Gemini Flash as a practical source of augmentation data for extremely low-resource languages at minimal cost.

6.3. LaBSE Quality Filtering

We investigated LaBSE-based quality filtering on the 24,999 synthetic pairs. Mean cross-lingual cosine similarity was 0.287 (std: 0.216), far below typical values for supported language pairs. At thresholds of 0.3, 0.4, 0.5, and 0.6, only 44.6%, 32.8%, 20.0%, and 10.0% of pairs would be retained respectively. Manual inspection confirmed translation quality was acceptable across all score ranges, including pairs with similarity scores below 0.1. We conclude that LaBSE similarity scores are not a reliable quality signal for Kokborok due to its absence from LaBSE’s training data. This is an important negative finding: quality filtering methods that rely on multilingual embeddings should be validated on a per-language basis before application to truly unseen languages.

6.4. Human Evaluation Analysis

The human evaluation scores of 3.74/5 (adequacy) and 3.70/5 (fluency) are consistent with the automatic metric performance. The native researcher in Kokborok linguistics gave the highest scores (3.84/3.80), while the untrained native speaker gave the lowest (3.64/3.54), a pattern commonly observed in MT human evaluation where expert annotators tend to apply scales more consistently [19]. The strong agreement between the two trained evaluators ($\kappa =0.67$) provides confidence in the reliability of the human assessment. Overall, the scores indicate the system is practically useful — preserving most meaning and producing largely natural output — while acknowledging room for improvement, particularly in fluency.

6.5. Domain Generalisation

Both systems achieve competitive scores on both the general-domain SMOL test set and the Bible-domain WMT test set, demonstrating reasonable cross-domain generalisation despite training on data from both domains. The higher trp→en BLEU scores on SMOL (38.56) versus WMT (28.03) reflect domain match with the SMOL training component.

6.6. Translation Direction Asymmetry

trp→en consistently outperforms en→trp across all systems and test sets. System 2 achieves BLEU 38.56 (trp→en) versus 15.25 (en→trp) on the SMOL test set. This asymmetry is expected: translating into English benefits from NLLB’s strong English generation capabilities, while generating Kokborok requires producing a low-resource language with limited representation in the base model. Closing this gap likely requires more gold Kokborok-side training data and potentially monolingual continued pretraining.

6.7. Comparison with Prior Work

Our System 2 achieves BLEU 17.30 (en→trp) versus the WMT 2025 best of 6.99, and BLEU 38.56 (trp→en) versus 2.99. We note that this comparison is not strictly controlled: WMT systems trained exclusively on Bible-domain data, while our systems additionally use SMOL professional translations covering diverse domains. The performance gap reflects both the richer training data and the effectiveness of our fine-tuning approach. Even on the WMT Bible test set alone, our system substantially outperforms WMT baselines, suggesting that the improvements are not solely attributable to domain match.

7. Limitations

This work has several limitations. First, our synthetic back-translation data was generated from English source sentences only (en→trp direction via Gemini), limiting Kokborok-source diversity in synthetic data. Future work could generate trp→en synthetic data from monolingual Kokborok text. Second, automatic metrics have known limitations for low-resource languages where reference translations may have limited lexical overlap with model outputs; our human evaluation partially addresses this concern. Third, we evaluate only on Roman-script Kokborok; the Bengali-script variant used in some official contexts is not addressed. Fourth, we do not attempt tokenizer adaptation or monolingual continued pretraining for Kokborok, which may further improve generation quality. Fifth, our comparison with WMT 2025 systems is indicative rather than strictly controlled due to differences in training data. Sixth, human evaluation was conducted only for the en→trp direction; trp→en human evaluation is left for future work.

8. Conclusion

We have presented KokborokMT, a significantly improved neural machine translation system for Kokborok, achieving BLEU 17.30 (en→trp) and 38.56 (trp→en) and substantially surpassing prior published results. Human evaluation by three annotators confirms practical translation quality with mean adequacy 3.74/5 and fluency 3.70/5. Our work demonstrates that a combination of professionally translated data from SMOL, domain-specific parallel data, and LLM-generated synthetic back-translations can bootstrap effective MT for an extremely low-resource Tibeto-Burman language. We additionally report that LaBSE-based quality filtering is unreliable for languages absent from the model’s training data, a cautionary finding for the community. We will release our model and evaluation pipeline publicly under CC-BY-4.0 to support further research on Kokborok and other under-resourced languages of Northeast India.