Improved Script Identification Algorithm Using Unicode-Based Regular Expression Matching Strategy

1. Background and Summary

Script identification (SI) is the first step in many text processing techniques, especially those used for language identification (LI). LI involves determining which language a piece of text in written in and classifying it as a pre-set language [1]. LI is the first step in many text processing tasks, including information retrieval, search engine use, and machine translation speech synthesis [2,3,4]. However, LI for short texts written in similar languages is a problem which has not yet been resolved [5,6,7,8]. Therefore, it is of great practical significance to optimize the performance of SI and LI algorithms [9,10,11].

In the Unicode encoding scheme, each script is arranged separately, which can be leveraged to identify the parts of text that are described by different scripts. SI is easier to implement than LI [9]. In LI, when more languages need to be recognized, the identification performance may be negatively affected [2,3]. Thus, some LI algorithms identify the script first, then identify the language among those that use the same script.

SI algorithms can be designed by taking advantage of the characteristics of the different scripts arranged in the Unicode encoding scheme. Hanif et al. used the beginning and end codes of different scripts to identify different parts of script in texts [12]. In a previous study, we constructed regular expressions according to the Unicode encoding ranges of different scripts and designed an SI algorithm using a regular expression matching strategy [13]. At the same time, a sentence-level SI data set containing 23 kinds of script and 261 languages was constructed using the Leipzig Corpora Collection, where each script was written in one or more language.

During the previous study, it was found that some characters are shared between different scripts. As it is impossible to be familiar with the use of every character in each script, we cannot grasp which scripts have shared characters or how many there are. Therefore, it is difficult to further optimize the performance of SI algorithms. Another disadvantage is that punctuation, numbers, and other symbols should be removed before script identification. Therefore, the text corresponding to each script obtained after identification may not be readable or cohesive.

To solve these problems, we use the coding regions of different scripts in the Unicode scheme, utilizing those that are unique to design a SI algorithm. The SI algorithm can recognize all 169 scripts in the current version of Unicode. In particular, we do not need to manually count the characters shared by different scripts when designing the SI algorithm, and the proposed algorithm is scalable. Another advantage is that it numbers, punctuation marks, and other symbols do not need to be filtered during SI and, so, these items are also included in the SI results. This ensures the integrity of the information provided in the SI results.

2. Data Description

2.1. Unicode Data

When designing an SI algorithm, it is generally necessary to identify the Unicode encoding range of each script. In this study, the script data provided on the Unicode website were used to achieve this. The web address is https://www.unicode.org/Public/UNIDATA/Scripts.txt [14]. This page contains the data for 170 script types. The first script type is “common,” which includes codes that appear in several scripts at the same time, such as numbers, punctation marks, characters, and symbols. The part of data for common-type script is shown in Figure 1. The other 169 types of script include characters which appear in only one script. The data for some of these script types are shown in Figure 2. Using the relationship between Unicode encoding and the script types, we designed an SI algorithm that can identify all 169 types (not including common-type script).

From Figure 1 and Figure 2, it can be concluded that, except for the line that starts with the symbol "#" and a blank line, the other lines on the web page represent Unicode encoding. The lines include four types of data: The Unicode encoding, the script, the character type, and the character name. In this study, an SI algorithm was designed using the Unicode encoding and its relevant scripts. We do not use the character name, as the same name may appear in different scripts.

2.2. Script Identification Data Set

Using the relationship between Unicode encoding and scripting discussed in Section 2.1, an SI algorithm was designed that can recognize the 169 scripts. Notably, at present, there is no open-source SI data set. Therefore, to assess the efficiency of the designed SI algorithm, we used the Leipzig Corpora Collection [15,16] to construct an SI data set which includes 24 scripts related to 261 languages. We used the Leipzig Corpora Collection as this data set is written in sentences, the source text had been downloaded from news or other websites, and does not contain foreign sentences. Thus, it is useful for building LI data sets. It includes most of the languages for text resources that are currently available on the internet.

In earlier research, we used the Leipzig Corpora Collection to construct a different SI data set that included 23 different scripts related to 261 languages, which included a CJK script group covering Chinese, Japanese, Korean, and other Chinese text types. However, in the data set used in this study, there was no CJK group. Based on the information obtained from the Unicode website [14], we re-organized the collected data. The Han and Hangul scripts were also included. The Han script includes Chinese, Japanese, and other Chinese text types, while the Hangul script only includes Korean.

SI is a text classification task. Therefore, when developing an SI algorithm, annotating text data is necessary. More than 270 languages can be downloaded from the Leipzig Corpora Collection’s website, with a few languages including different script data. To test the proposed SI algorithm’s efficiency, we selected the language data which include only one script. When constructing the SI data set, we first needed to determine what kind of script each text was written in, and then label the text. The details of the SI data set are presented in Table 1.

3. Script Identification

3.1. Script Identification Algorithm in Previous Research

In the first stage of the previously proposed SI algorithm [13], the Unicode encoding ranges of the different scripts were analyzed, and corresponding regular expressions were constructed. There are spaces between words when writing in languages other than Chinese and Japanese; therefore, the Unicode encoding of spaces was added to the regular expression of each script. To distinguish the SI algorithm proposed in this study, the SI from the previous study is expressed as PSI. The working process of the PSI algorithm proposed in our previous study is as follows:

• Remove all symbols in the text, except for characters and spaces, and replace them with spaces.
• Replace consecutive spaces with a single space. Filter out spaces at the beginning and end of the text.
• The text is matched with the regular expressions of all scripts separately. Replace consecutive spaces in each match with a single space. If the matching result is not empty, it means that the text contains the content described in the corresponding script, and the matching result and the corresponding script are stored in a dictionary.
• After the matching between the text and all the regular expressions is completed, the SI result is returned, which describes the different script content contained in the text and the corresponding script.

We previously found that some characters are shared by multiple scripts. Therefore, when constructing the regular expressions, the Unicode encodings of characters were added to the corresponding scripts. As it is difficult to fully account for these shared characters, our preliminary research focused on improving the SI performance. Thus, in the pre-processing stage, all the numbers, punctuation marks, and various symbols (except for the characters and spaces) were filtered as they appear in a variety of scripts. Although the performance of the algorithm was good, it is still difficult to ensure the integrity of the identified script due to the filtering of numbers, punctuation marks, and various symbols.

3.2. Constructing Regular Expressions for Scripts

The relationship between Unicode encoding and scripts can be summarized using the information provided on the Unicode website (https://www.unicode.org/Public/UNIDATA/Scripts.txt). There are four types of Unicode encoding data on the web page. In Table 2, “X” represents a hexadecimal number. Among them, types 1 and 3 represent only one form of Unicode encoding, while types 2 and 4 represent several Unicode encodings of the same type arranged consecutively. The Unicode encoding length of types 1 and 2 is a four digit-long hexadecimal number, while the other two are five digit-long hexadecimal numbers.

There are 170 scripts on the Unicode web page. The common script data represent Unicode encoding that appears in several scripts at the same time. This type contains numbers, punctuation marks, symbols, and certain characters. The unique codes correspond to other scripts.

Table 3. Some script’s regular expression.

Script	Unicode encoding
Katakana	[\\u30A1-\\u30FA\\u30FD-\\u30FE\\u30FF\\u31F0-\\u31FF\\u32D0-\\u32FE\\u3300-\\u3357\\uFF66-\\uFF6F\\uFF71-\\uFF9D\\U0001AFF0-\\U0001AFF3\\U0001AFF5-\\U0001AFFB\\U0001AFFD-\\U0001AFFE\\U0001B000\\U0001B120-\\U0001B122\\U0001B155\\U0001B164-\\U0001B167]
Hiragana	[\\u3041-\\u3096\\u309D-\\u309E\\u309F\\U0001B001-\\U0001B11F\\U0001B132\\U0001B150-\\U0001B152\\U0001F200]
Bopomofo	[\\u02EA-\\u02EB\\u3105-\\u312F\\u31A0-\\u31BF]

Table 3. Some script’s regular expression.

Script	Unicode encoding
Katakana	[\\u30A1-\\u30FA\\u30FD-\\u30FE\\u30FF\\u31F0-\\u31FF\\u32D0-\\u32FE\\u3300-\\u3357\\uFF66-\\uFF6F\\uFF71-\\uFF9D\\U0001AFF0-\\U0001AFF3\\U0001AFF5-\\U0001AFFB\\U0001AFFD-\\U0001AFFE\\U0001B000\\U0001B120-\\U0001B122\\U0001B155\\U0001B164-\\U0001B167]
Hiragana	[\\u3041-\\u3096\\u309D-\\u309E\\u309F\\U0001B001-\\U0001B11F\\U0001B132\\U0001B150-\\U0001B152\\U0001F200]
Bopomofo	[\\u02EA-\\u02EB\\u3105-\\u312F\\u31A0-\\u31BF]

There are two types of regular expression. One is the common type’s regular expression (CRE). We analyzed the encoded region of the common type of script to construct the CRE, which was used to identify the common Unicode script parts in text when designing the SI algorithm. The other type is each script’s regular expression (SRE). We analyzed the coding area of each script to construct SREs, Some SREs are shown in Figure 3. To subsequently compare our work with that of other researchers, the type of encoded data and the order on the original web page were maintained when constructing regular expressions, and different types of contiguous regions were not merged. An example regular expression is shown in Table 1 in Appendix A, while all SREs are shown in Table A2 in Appendix A.

In our earlier research, we only retained characters and spaces, while the other types were removed (e.g., punctation marks, numbers, and other symbols). To retain the semantic information in the source sentence as much as possible, as well as for the convenience of subsequent text processing, the regular expression of the script was constructed by merging the CRE and SRE of each script. This type of regular expression is called an MCSRE. An example of a constructed MCSRE is shown in Table A1 in Appendix A. In this example, the parts other than the bold and underlined text are the CRE. Due to spatial limitations, the MCSREs of all the scripts are not listed here. Using the CRE and SREs given in the appendix, we constructed each script’s MCSRE.

3.3. Improved Script Identification Algorithm

Numbers, punctuation, symbols, and a few characters that are common to several scripts also appear in the text. To ensure the readability and the accuracy of various subsequent natural language processing tasks, the semantic integrity of each recognized script should be ensured during SI. To solve these problems, we use the CRE and MCSRE to design an improved script identification algorithm (ISI). The working process of the ISI algorithm proposed in this study is described as follows, and the flow diagram is shown in Figure 3.

1)

The text is matched with the CRE to obtain the common parts of the script in the text and calculate their length.

2)

The text is matched to the MCSRE and the results for each script, respectively. When analyzing the MCSRE matching of a text and a script, one of the following two situations will occur:

a)

If the length of the MCSRE match result is equal to the length of the CRE match result, the current script content will not be included in the text. Then, study the text with the next script for the MCSRE operation.

b)

If the length of the MCSRE match result is not equal to the length of the CRE match result, the current script content is included in the text. As each MCSRE contains whitespace encoding, if the text contains multiple scripts, some matches will have consecutive spaces. After replacing the consecutive spaces in the matching result with spaces and filter out spaces at the beginning and end of the text, add the corresponding script and the matching result to the result dictionary. Then, study the text with the next script for the MCSRE operation.

3)

The text is matched with the MCSREs of all the scripts, and the resulting dictionary is returned after the operation has been completed. The resulting dictionary contains different script content than that which appears in the text.

3.4. ISI Example

To illustrate the ISI process, we chose a sentence containing multi-scripted content selected from the corpus. The sentence is as follows: “Bloomberg News сo ссылкoй на прoект заявления G7 пo итoгам заседания.”. Space limitations were not allowed, such that we could submit results that matched the MCSREs of all the scripts; only the key and partial matching results were submitted. An example is shown in Table 5.

3.5. Script Identification Experiment

To test the performance of the ISI algorithm, the SI data set was divided into training and test set at a ratio of 8:2. Then, the ISI algorithm was trained and tested. At the same time, comparative experiments were conducted to compare the performances of the ISI and SI algorithms proposed in this and the previously published study, respectively.

SI is a text classification task, in which each entry in the text classification corpus must have a label and, so, the text classification algorithm must return a label when predicting the text. When evaluating the performance of an SI algorithm, it is used to determine which script is contained in the text; therefore, the number of words in each script was counted, and the script with the most words was selected as the main script of the text. Languages other than those in the Han group are written with spaces between the words. For matches in these script groups, the number of words separated by a space were counted. As Han scripts encode more information in a shorter number of characters than, for example, Latin scripts, the Chinese and Japanese languages in the Han group are written without spaces between the words. Thus, for the Han group, the length of the matching result was considered, except for the spaces.

The performance of the text classification algorithms was evaluated using Precision, Recall, and F-score. This study used the $\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}$, $\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}$, and ${\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{F}}_1$ metrics to evaluate the performance of the SI algorithm, as shown in Equations 1–3, in which C represents all the scripts in the SI data set; |C| denotes the number of scripts in the SI data set; and the meanings of TP, FP, and FN are given in Table 6.

Comparative SI experiments were performed using the training and test data, and the results are shown in Table 7 and Table 8. The performance of the improved SI algorithm is almost identical to that of the SI algorithm designed in our previous study. The improved SI algorithm does not require pre-processing before SI, and the script content returned after SI includes numbers, spaces, and punctuation marks. All 169 kinds of script in the current version of Unicode were recognized.

$$\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}}{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}+\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{F}\mathrm{P}}_{\mathrm{i}}}},$$

(1)

$$\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}}{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}+\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{F}\mathrm{N}}_{\mathrm{i}}}},$$

(2)

$${\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{F}}_1={\displaystyle \frac{2\times \mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}\times \mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}}{\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}+\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}}}.$$

(3)

4. Conclusions

SI is the first step in many text processing tasks. When SI is performed on text, the integrity of the original information in each script should be maintained. As some characters are shared by multiple scripts, we used the relationships between Unicode encoding and scripts provided on the Unicode website to construct corresponding regular expressions for each script and designed an ISI algorithm on the basis of these expressions. The improved SI algorithm proposed in this study can identify all 169 kinds of script in the current version of Unicode.

We used the Leipzig Corpora Collection to construct an SI data set that included 24 scripts related to 261 languages, which was then utilized to conduct comparative experiments in order to test the performance of the proposed algorithm. The ISI algorithm proposed in this study performed almost as well as the SI algorithm proposed in our previous study.

In the early stage of research relating to the design of SI algorithms, the pre-processing stage involved filtering out numbers, punctuation marks, and any symbols other than characters, thus affecting the integrity of the SI results. In contrast, the algorithm proposed in this study does not require numbers, punctuation marks, or other symbols to be filtered during SI; therefore, these items in the text are also included in the SI results, thus ensuring the integrity of the information provided in the SI results.