Assessing the Quality of Some Machine Translation Systems Using an Information-Theoretic RS-Method

1. Introduction

Translation from one language to another plays an increasingly important role in modern society, which is why many IT companies are developing and researching various machine translation systems. The first demonstration of machine translation capabilities took place on January 7, 1954, in New York City, at IBM headquarters. Organized by Georgetown University and IBM, the experiment demonstrated the fully automatic translation of more than 60 sentences from Russian into English. It was assumed that the problem of machine translation would be solved within the next ten years, but more than 70 years have passed, and the problem of translation quality remains unsolved. Naturally, the issue of translation quality is of great importance to the users of translated data and, consequently, to the developers of translation system.

Currently, there are many different approaches and methods for evaluating translation quality (see [1,2,3,4,5,6,7] and the reviews there in). In this paper, we develop an approach based on evaluating translation quality by the degree to which the author’s style is preserved [7]. But first, let’s take a brief look at the concept of author’s style. We’ll start with the definition provided by the renowned mathematician A.N. Kolmogorov [8]. He identified the following three sources of variability in a text: content, form, and the unconscious individual authorial style. Content is a straightforward concept; form corresponds to genres such as prose and poetry; whereas the unconscious individual authorial style reflects of the writer’s personality [6], which includes the set of words and expressions used, sentence structure, metaphors, and so on [6]. Today, this classification is widely accepted and is being further developed by many researchers and literary scholars [9,10,11,12,13]. Furthermore, according to the generally accepted view, an author’s individual style is, in a sense, as unique as a fingerprint. It is important to note that individual style is inherent not only to writers but also to ordinary people. For our study, we selected texts by famous writers because their authorial style is most clearly expressed and obvious. It is worth recalling that it has been shown that author style can be identified in authors writing in different languages (Amharic, Chinese, English, Russian) based on an analysis of texts of approximately the same length [14].

The issue of preserving the author’s style in translation was discussed by many literary scholars even before the advent of computers, and it was established and acknowledged that there are significant differences among translators in the degree to which they preserve the author’s style. (In particular, there are translations of classic novels in which the author’s style is invisible and completely replaced by the translator’s style [7]).

It is now generally accepted among literary scholars and linguists that the better a translation preserves the author’s style, the better the translation (see [7] for an overview).

In this article, we develop a method for evaluating translation quality based on the following premises: i) there are currently many machine translation systems that provide an acceptable level of content translation; ii) an important and challenging task for a translator is the ability to preserve the author’s style.

Based on these premises, we have expanded and refined the RS-method for quantitatively assessing the degree to which the author’s style is preserved in a translation.

It should also be noted that in this paper we focus on machine translation systems that deal exclusively with the translation of prose (or “common”) texts (the translation of poetic texts appears to be much more complex and is not yet a priority area for machine translation).

We apply this method to cross-translations to twelve literary works across six language-pair directions: English → Russian, Russian → English, Amharic → English, English → Amharic, Chinese → English, and Chinese → Russian.

Our research has shown that modern machine translation systems can preserve the author’s style to varying degrees, and this ability can vary across different language pairs. (For example, the same machine translation system may perform well for the Chinese–Russian pair but poorly for the English–Amharic pair.)

Overall, the proposed method can be useful both for those who use translated data and for machine translation developers.

2. Key Concepts

2.1. Author’s Style, Authorship Identification and Stylometry

In computational linguistics, authorial style is a probability distribution of sequences of linguistic units [10,13]. General characteristics of the author’s text—lexical units, syntactic constructions, punctuation, sentence length, etc.—together form statistical properties that can be estimated based of sufficiently large text samples. An ideal translation should be a text in the target language whose statistical properties reflect the properties of the original text [10,12]. This view is consistent with the tradition of stylometry, which views authorial style as a multidimensional statistical “fingerprint”. [9] Stylometry is a quantitative method in linguistics, literary studies, and other humanities that allows one to determine authorial style using statistical indicators [11,14]. Research in the field of authorship attribution using stylometry has shown that relatively simple characteristics—the distribution of n-grams and the frequency of function words—are sufficient to identify the author in texts in European languages [12,13,14].

If we want to apply stylometry to Eastern languages, we immediately run into a problem: how to count words if the text isn’t separated by spaces? This is impossible without writing specialized programs (parsers) [6] containing language-specific rules that can break sentences into words and words into morphemes. This is a very labor-intensive and not always reliable process.[6]

The RS- method we are developing is significantly simpler and more convenient to use and allows for quantitative assessment of translation quality using mathematical statistics.

2.2. Background to the Creation of the Proposed Method

The method of evaluating the preservation of the individual unconscious author’s style in translations is based on the use of lossless data compression algorithms, i.e., archivers. Their purpose is to encode texts in such a way that the length of the encoded message is shorter than the original, and if necessary, the encoded text can be decoded into the original. The essence of this action is as follows. The archiver receives text data, which it encodes into files of shorter length, i.e., “compresses.” This is achieved by finding irregularities in the frequencies of letters and words, as well as other hidden patterns, using various methods of information theory and formal grammar theory, and, in recent years, artificial intelligence.

Let us briefly describe the application of the information-theoretic approach within the framework of our method. Suppose we are given three essays E1, E2, and E3. Suppose we know that E1 and E2 were written by specific authors P1 and P2, while essay E3 was written by either P1 or P2. Let Z be a compression tool. If it is applied to a file Y, then we denote the length of the “compressed” file by Z(Y). The method works as follows: first, we pair the texts: the essay by a known author with the text by an unknown author E1E3 and E2E3, and compress both pairs. We also compress the texts of essays E1 and E2 separately, after which we calculate the difference in the lengths of the compressed files: Z(E1E3) minus Z(E1), and then similarly Z(E2E3) - Z(E2). If Z(E1E3) - Z (E1) is less than Z(E2E3) - Z(E2), then we conclude that text E3 belongs to author P1. If Z(E1E3) - Z(E1) > Z(E2E3) - Z(E2), then essay E3 belongs to author P2. This result occurs because the archiver, when compressing the added portion of text from essay E3, uses the statistical features it discovered when compressing the first portion, namely E1 or E2. Therefore, text E3 compresses better after text with the same information source. A simple example explaining the essence of the method: let E1 be a text in Spanish, E2 a text in French, and the unknown text E3 also be in Spanish. Then Z(E1E3) - Z(E1) will be less than Z(E2E3) - Z(E2), because in the first case the text E3 in the combined file E1E3 is compressed after the archiver is “configured” to “its” statistics and becomes shorter.

This idea was proposed by Tehan [16,17] and in the work of the authors [18], this idea was combined with methods of testing statistical hypotheses, which made it possible to construct a method for proving the reliability of results (within the framework of mathematical statistics). The described method was successfully applied by the authors to solve text attribution problems in the work [18], where it was experimentally demonstrated that each author has his own individual style, which is quite accurately manifested in his text of 4 KB (approximately two pages). It is important to note that the texts of writers in four very distant languages, from different language families, were considered and it turns out that an author’s style can be identified for any language using roughly the same amount of data [15].

2.3. Translation Quality and Cramer’s Coefficient

To make the description clearer, we will illustrate it with an example of constructing a method for recognizing the author’s style of various English-language writers.

Let’s describe the method: N different texts E1, E2, …, EN (e.g., works by different writers) are given. Each text Ei is represented as two sets, called the training set (Xi, i = 1, …, N) and the experimental set, which, in turn, consists of M parts (slices), which we will denote by Yij, i = 1, …, N; j = 1, …, M.

As an example, we compiled a set of texts from the following English-language writers: Beresford, London, O. Henry, and Twain (N = 4, M = 16).

From the works of these authors, we formed four training sets X1, X2, …, X4, each 64 KB long. Then we formed verification samples - 16 files Y1j, j= 1, …, 16, 4 KB long each, from the works of Beresford, Y2j, j= 1, …, 16 from the works of London, and then test samples - Y3j, j= 1, …, - Y4j, j= 1, …, from the works of O. Henry and Twain. Then, file Y1,1 was sequentially “compressed” with training samples X1, X2, ...X4, and it was determined with which of them it compressed “better” (i.e., Z(X1Y1,1) - Z(X1), ..., Z(X4Y1,1) - Z(X4) were calculated and i was found for which Z(Xi Y1,1) - Z(Xi) is minimal. All Yij, i = 1, ..., 4; j = 1, ..., 16, were processed similarly.

Table 1 presents the obtained data for the LZMA archiver, a 64 KB training sample (Xi), and a 4 KB slice (Yij).

Let us explain the meaning of these numbers: the number 16 in the third row of the table means that out of 16 files Y3j, j= 1, …, 16, all “compressed” better with X3, (in other words, all 16 “slices” from O. Henry’s works “compressed” better with the training set from his own work.) That is, it turned out that O. Henry’s authorial style is uniquely determined by a 4 KB slice with a training set of 64 KB. The numbers in the second row mean that out of 16 files Y2j, j= 1, …, 16, 15 “compressed” better with X2, (i.e., 15 slices from London’s works “compressed” better with his training set, but 1 slice is more similar to Twain). In the first and last rows, “recognizability” is worse: 13 slices out of 16 compressed better with their own writers, and 3 slices with other people’s texts.

We will present the results of the transition from the source texts E1, E2, …, EN in the form of a contingency table (of size NxN), and we will denote the contingency table itself by W( E1, E2, …, EN) or W (depending on the context) and present this table as

e_1,1 e_1,2 …. e_1,N

W( E₁, E₂, … , E_N) = e_2,1 e_2,2 …. e_2,N

e_N,1 e_N,2 …. e_N,N

We’ll explain the significance of the W contingency table in more detail. As we saw in the example, the numbers in the cells of the contingency table indicate the number of fragments attributed to a particular author. If the method works “correctly,” that is, correctly identifies the author’s style from the fragments, the values in the table will be concentrated primarily on the main diagonal. Otherwise, when the fragments provide no indication of the author’s style, the values in the table will be evenly distributed among the different cells attributed to different authors. This effect can be quantified using Cramer’s C coefficient [19], which is calculated as follows: First, calculate P=${\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{j=1}^N}\mathrm{e}\mathrm{ᵢ}\mathrm{ⱼ},\mathrm{p}\mathrm{ᵢ}\mathrm{ⱼ}={\displaystyle \frac{\mathrm{e}\mathrm{ᵢ}\mathrm{ⱼ}}{\mathrm{P}}},\mathrm{p}\mathrm{ᵢ}.={\displaystyle \sum _{j=1}^N}\mathrm{p}\mathrm{ᵢ}\mathrm{ⱼ},\mathrm{p}.\mathrm{ⱼ}={\displaystyle \sum _{i=1}^N}\mathrm{p}\mathrm{ᵢ}\mathrm{ⱼ},$ and then calculate the following:

x2 =${\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}(\mathrm{e}\mathrm{ᵢ}\mathrm{ⱼ}-\mathrm{N}\mathrm{p}\mathrm{ᵢ}.\mathrm{p}.\mathrm{ⱼ})²]/(\mathrm{N}\mathrm{p}\mathrm{ᵢ}.\mathrm{p}.\mathrm{ⱼ}\left)\mathrm{ⱼ}\right)$ and Cramer’s coefficient C = $\sqrt{x2/\left(PN\right)}$.

Note that the Cramer coefficient C=1 if all nondiagonal elements equal to 0, and C is equal to 0, if all e_i,j are equal.

It should be noted that the Cramer’s coefficient is used to assess the relationship or interdependence, and it takes values between zero and one.

Another important purpose of a contingency table W is to test a hypotheses about the heterogeneity of the data using the Chi-square test [7,15,18].

2.4. Slice-Based Analyses

A distinctive feature of the RS- method is its reliance on contiguous text fragments rather than individual sentences extracted from the text.

In our experiments, we investigated three fragment sizes: 4 kilobytes, 2 kilobytes, and 1 kilobyte, applied to a 64 kilobyte source corpus for each text studied. Smaller fragments increase the number of observations but reduce statistical information per fragment; larger fragments provide richer samples but fewer data points [18]. The consistency of the results across fragment sizes serves as a test of the robustness of our findings. We examined the impact of the archiving methods on the study results [7,15,18]. It turned out that the choice of archiving mechanism has virtually no effect on W. So, for further work, we selected the file archiver LZMA.

2.5. Selecting Machine Translation Systems

System	Type/Architecture	Released	Owned By	Official Website
Google Translate	Statistical MT (2006); rebuilt on Neural MT (Nov 2016); general-purpose, high-resource coverage [20]	2006 (SMT); 2016 (NMT)	Google LLC (Alphabet Inc.)	translate.google.com
Microsoft Bing Translator	Statistical MT (2007); rebuilt on Neural MT (Nov 2016); combines NMT, syntax-based/phrase-based SMT, n-gram modeling [21]	2007 (SMT); 2016 (NMT)	Microsoft Corporation	www.bing.com/translator
Claude (Anthropic)	Large language model (transformer-based, RLHF-aligned); prompt-driven translation, no dedicated MT pipeline [22]	March 2023 (Claude 1); translation-relevant generation: 3.5 (2024)	Anthropic PBC	www.anthropic.com (claude.ai)
GPT (OpenAI)	Large language model (transformer-based, RLHF-aligned); prompt-driven translation [23]	Nov 2022 (ChatGPT/GPT-3.5); March 2023 (GPT-4); May 2024 (GPT-4o)	OpenAI, Inc. (OpenAI Global LLC)	www.openai.com (chatgpt.com)
Gemini (Google)	Large language model (transformer-based, multimodal); prompt-driven translation with large-scale knowledge integration [24]	Dec 2023 (Gemini 1.0); 2025 (Gemini 2.5 Pro)	Google DeepMind (Alphabet Inc.)	gemini.google.com
DeepSeek	Large language model (Mixture-of-Experts transformer); open-weight, Chinese-English-emphasized training corpus [25]	Nov 2023 (DeepSeek LLM); Dec 2024 (V3); Jan 2025 (R1)	Hangzhou DeepSeek Artificial Intelligence Co., Ltd. (High-Flyer)	www.deepseek.com

2.6. Literary Base of the Study

For our study, we selected literary works in four languages, two from each. It is important to note that the selected languages belong to different language families with different scripts, different syntax and grammar

Table 3. List of literary works used.

No.	Language	English Author Name	English Title	Author Name in Source language	Book Title in source language	First Published	Revised Edition
1.	English	Florence L. Barclay	The White Ladies of Worcester	Florence L. Barclay	The White Ladies of Worcester	1917	-
2.		Virginia Woolf	To the Lighthouse	Virginia Woolf	To the Lighthouse	1927	-
3.	Russian	Anton Chekhov	The Lady with the Dog	Антoн Чехoв	Дама с сoбачкoй	1898	1991
4.		Nikolai Gogol	Dead Souls	Никoлай Гoгoль	Мертвые души	1842	1994
5.	Chinse	Yu Hua	To Live	余华	活着	1993	2014
6.		Shen Congwen	Border Town	沈从文	边城	1934	2009
7.		Zhang Ailing	Love in a fallen city	张爱玲	倾城之恋	1943	2012
8.		Lao She	Rickshaw boy	老舍	骆驼祥子	1936	1982
9.	Amharic	Haddis Alemayehu	Love to the Grave	ሀዲስ አለምየሁ	ፍቅር እስከ መቃብር	1968	-
10.		Bealu Girma	Oromaye	በዓሉ ግርማ	ኦሮማይ	1983	-

Table 3. List of literary works used.

No.	Language	English Author Name	English Title	Author Name in Source language	Book Title in source language	First Published	Revised Edition
1.	English	Florence L. Barclay	The White Ladies of Worcester	Florence L. Barclay	The White Ladies of Worcester	1917	-
2.		Virginia Woolf	To the Lighthouse	Virginia Woolf	To the Lighthouse	1927	-
3.	Russian	Anton Chekhov	The Lady with the Dog	Антoн Чехoв	Дама с сoбачкoй	1898	1991
4.		Nikolai Gogol	Dead Souls	Никoлай Гoгoль	Мертвые души	1842	1994
5.	Chinse	Yu Hua	To Live	余华	活着	1993	2014
6.		Shen Congwen	Border Town	沈从文	边城	1934	2009
7.		Zhang Ailing	Love in a fallen city	张爱玲	倾城之恋	1943	2012
8.		Lao She	Rickshaw boy	老舍	骆驼祥子	1936	1982
9.	Amharic	Haddis Alemayehu	Love to the Grave	ሀዲስ አለምየሁ	ፍቅር እስከ መቃብር	1968	-
10.		Bealu Girma	Oromaye	በዓሉ ግርማ	ኦሮማይ	1983	-

3. Research Results

We’ll now move on to a description of translation quality analysis. We’ll begin with an example of translating Nikolai Gogol’s novel “Dead Souls” from Russian into English. A 128-kilobyte corpus of the novel’s text was extracted. Only this portion was translated into English by each of the six machine translators we selected. We then divided each translated text into two samples—a training sample and a control sample. (This yielded 12 samples: six test samples and six training samples corresponding to each translator.) The test samples were then divided into slices of a specific length. Different slice lengths were used in different experiments—1 kilobyte, 2 kilobytes, or 4 kilobytes—to observe the impact of this parameter on the experimental results. Each slice was then compressed in turn using the trained sample.

After this, a W table was constructed for each specific slice length, and Cramer’s coefficient was calculated. Table 4 is an example of this processing.

As shown earlier, the number 11 on the left in the first row means that 11 slices from Claude’s test sample were “better compressed” after the training sample translated by itself, while one slice was better compressed after Gemini, and so on. The main diagonal shows that most slices are better compressed with their own training set, as evidenced by the relatively high Cramer coefficient. We then performed similar calculations for different groups of translators.

We separately constructed a table in a completely similar manner and calculated the Cramer coefficient for the translators Claude (Cl), DeepSeek, and Gemini; see Table

Table 5. – Translation of Gogol’s novel “Dead Souls”, C=0.51.

	Claude	DeepSeek	Gemini
Claude	14	2	0
DeepSeek	13	2	1
Gemini	9	3	4

Table 5. – Translation of Gogol’s novel “Dead Souls”, C=0.51.

	Claude	DeepSeek	Gemini
Claude	14	2	0
DeepSeek	13	2	1
Gemini	9	3	4

It turned out that in this case C= 0.51 and, as can be seen from this table, the translator’s style is not determined by the slices, as evidenced by the main diagonal.

Another pair of translators with which we conducted similar calculations – Google and GPT – yields C = 0.93; i.e., these data show that when translated by Google and GPT translators, the author’s style is almost completely replaced by the translator’s style. Cramer’s coefficient of C = 0.93 means that, based on a single slice, it is almost reliably possible to determine which translator a given slice belongs to. On the other hand, translators Claude, DeepSeek, and Gemini impose the translator’s style instead of the author’s style to a significantly lesser extent (C = 0.51).

Finally, if we consider all six translators (Table 4), we see that C = 0.79, i.e., the situation is intermediate: on average, all translators suggest their own style over the author’s, but to a greater extent than the top three in this example (Claude, DeepSeek, and Gemini) and to a lesser extent than the Google and GPT.

We performed similar calculations not only with 4 KB slices, but also with 2 KB and 1 KB slices, and with the same training set. All the obtained results are presented in Table 6:

The sixth column corresponds to a slice size of 4 kB, the seventh and eighth to 2 kB and 1 KB, respectively.) In all cases, we see that Google Translate and GPT Translate barely preserve the author’s style. Claude, DeepSeek, and Gemini Translate, however, preserve the author’s style significantly better, while the comparison of all translators yields an intermediate result.

We present data for three slice sizes to, firstly, demonstrate that the results and conclusions are virtually independent of this parameter and, secondly, to demonstrate that these data may be useful for translation system developers seeking to reduce the influence of machine translation on translation style.

The main conclusion is that the machine translators under consideration substitute the author’s style to varying degrees. For some translators, their style (not the author’s!) is almost unambiguously determined by the translation; in other cases, the machine translator’s style is barely noticeable. In other words, such translators do not replace the author’s style with their own!

The second important conclusion is that none of the translation systems considered in this study has a consistently low Cramer coefficient for all language pairs. In other words, there are currently no machine translators that preserve the author’s style for all pairs of languages.

Third, the contrast between the G + B pair (almost always C ≥ 0.87) and the Cl + Ge pair (often C < 0.50) suggests that commercial translation systems differ not only in accuracy but also in their fundamental approach to stylistic transformations. Systems with a high Cramer coefficient are not necessarily worse translators in a semantic sense, but they are more stylistically biased—they more persistently impose their own patterns on the target text.

Fourth, the slice-size effect (C decreasing as slice size decreases from 4 kB to 1 KB) confirms that style attribution requires a minimum amount of text. For the practical deployment of this method in quality audits, we recommend a minimum slice size of 2 kB and a training corpus of at least 64 kB per translator, consistent with the recommendations from papers of authors [7,15,18].

4. Conclusions

This study, based on a large-scale cross-lingual experiment, demonstrates that: (1) individual machine translators convey author style to varying degrees of accuracy; (2) the effectiveness of any given machine translator in preserving author style depends on the pair of language families used, with the greatest intertranslational divergence observed for Afroasiatic-Germanic (Amharic-English) and Sino-Slavic (Chinese-Russian) languages; and (3) the information-theoretical basis of the RS- method [7,15,18] provides the mathematically rigorous, language-independent foundation necessary for the recommended method of translator quality assessment based on author style preservation.