Improving the Time Efficiency of a Script Identification Algorithm Using a Unicode-Based Regular Expression Matching Strategy

1. Introduction

Script identification (SI) is a technology in which computers are used to determine the script in which text content is written. It is the first step in most text processing systems, especially in those designed for language identification (LI)—that is, the process of determining which language a piece of text is written in and classifying it as a pre-set language [1,2]. LI is the first step in many text-processing tasks, including information retrieval, search engines, machine translation, and speech synthesis [3,4,5,6]. SI is easier to implement than LI and has a higher accuracy, but reducing the number of recognized languages in LI systems can improve their efficiency [3,4]. Therefore, some researchers have proposed techniques in which the script is first recognized and then the language within the same script is identified, effectively improving the efficiency of LI [7,8,9]. With the explosive growth in online information, the timely collection of multilingual text content and its language classification has become a primary problem. Therefore, it is necessary to improve the time efficiency of SI systems.

In the Unicode encoding scheme, a separate area is allocated for each script’s characters and symbols (relevant information can be found on the official Unicode website [10]). Generally, SI algorithms are designed using the Unicode encoding scheme’s features [11,12]. In preliminary research, we analyzed the common and specific parts of all 169 scripts on the Unicode website, constructed regular expressions for each script and based on this, designed an SI algorithm which is capable of recognizing these scripts while ensuring the integrity of the information in the SI results [13]. Although the focus of this preliminary research was improving the accuracy of SI, time efficiency enhancements were not considered.

In this study, the regular expressions constructed after analyzing the Unicode website scripts were applied and the following two optimization strategies were combined to design an SI algorithm with improved time efficiency.

1)

According to statistics reported by Statista [14], the number of web pages in different languages varies, with some languages having a higher proportion than others. Using these statistics, we can deduce which scripts are used on more webpages. When designing a script recognition algorithm, the scripts used in most web pages should be considered first, as they are more likely to occur. Once the target script is identified, there is no need to continue analyzing the remaining scripts.

2)

Sometimes, the text (or even individual sentences) contains content in different scripts. In these cases, it is necessary to identify the different scripts within the text. The length of the text is fixed, and the sum of the lengths of text in different scripts is equal to the total length of the text. Therefore, when designing a script recognition algorithm, it is necessary to analyze the lengths of different script contents being recognized: if the sum of the lengths of contents in different scripts is equal to the total length of the text, there is no need to analyze the remaining scripts.

2. Data Description

2.1. Unicode Character Database

At present, the Unicode website’s character database provides encoding blocks for 169 scripts [15], as illustrated in Figure 1. In addition to the encodings for each script, it also contains symbols and characters shared by multiple scripts, categorized as “Common,” as shown in Figure 2. Analysis of the web pages that provide the data tables reveals that, except for lines starting with “#”, the remaining lines all contain information related to Unicode characters, including the Unicode encoding, script, character type, number, and name. As the same character name sometimes appears in different scripts, when designing the SI algorithm, the relationship between the character encoding and the script was carefully taken into consideration.

2.2. Constructing Regular Expressions for Scripts

There are four types of Unicode encoding data in the Unicode character database, as shown in Table 1, where each ‘X’ represents a single hexadecimal digit. Among these, Type 1 and Type 3 represent Unicode encodings with only one form, while Type 2 and Type 4 represent multiple consecutive Unicode encodings of the same type.

In the preliminary study, to identify all scripts on Unicode websites while ensuring the integrity of the information in the SI results, the Unicode character database was analyzed and two types of regular expressions were constructed. The first was the common regular expression (CRE), the purpose of which is to identify several characters and symbols shared by scripts that appear in the text. This regular expression was constructed by analyzing the data in the Unicode character table classified as ‘common’. The CRE are shown in Table A1 in Appendix A; all content in the table (other than that in bold) are CRE.

The second type of regular expression is the script regular expression (SRE). This type was constructed by analyzing the Unicode codes of characters in scripts other than those labeled as ‘common’; all SREs are shown in Table A2 in Appendix A. In addition to characters, text may also contain various symbols. To ensure information integrity, these symbols must also be retained in the SI results. However, integrity cannot be guaranteed when only using SREs. Therefore, the SRE for each script was combined with the CRE to construct another regular expression, called the Mixed Common–Script Regular Expression (MCSRE). An example of a constructed MCSRE is shown in Table A1 in Appendix A; due to spatial limitations, the MCSREs of all scripts are not listed here. Using the CRE and SREs provided in Appendix A, the MCSRE for each script was constructed. These MCSREs were then used alongside the CRE and SREs in the SI algorithm designed in this study.

2.3. Script Identification Dataset

At present, there is no open-source dataset designed specifically for SI. The Leipzig Corpora Collection’s source texts were downloaded from news or other websites, and doesn’t contain sentences in various foreign languages. The texts are organized by language, with sentences as the basic unit, and datasets of different sizes are provided for each language [16]. It is very useful for building LI datasets as it includes most of the textual resources currently available on the Internet. In a preliminary study, data covering 273 languages were downloaded from the Leipzig Corpora Collection, and 261 languages that are written using only one script were selected. Among these 261 corpora, 208 contained at least 10,000 sentences each, while the other 53 contained fewer than 10,000 sentences [17,18]. When constructing the SI dataset, we first determined which script each text was written in and then labelled the text with that script. The details of the SI dataset are presented in Table 2, Table 3 and Table 4, in which language ISO codes are used to list the languages considered in this study. Through language encoding, the corresponding language name and country can be searched for on the relevant website [19,20].

3. Script Identification Algorithm

3.1. Script Identification Algorithms in Previous Research

To identify scripts and ensure the integrity of the SI results, the character database provided on the Unicode website was analyzed to construct CRE and MCSREs, on the basis of which the improved SI (ISI) algorithm was designed [13]. The ISI algorithm can recognize all scripts currently recorded on the Unicode website and has good scalability. Its workflow follows the flow diagram shown in Figure 3, comprising the following steps:

1)

The text is matched with CREs to extract the parts of the text for which the script tag is ‘common’, and the collective length of these parts is calculated. The matched content consists of symbols, numbers, and other elements shared by several scripts.

2)

The text is matched with the MCSRE of each script. When analyzing the matching results for each script, one of the following two situations will occur:

a)

If the length of the MCSRE match result is equal to the length of the CRE match result, the text does not contain content in the current script. Then, MCSRE matching with the next script will be performed on the text.

b)

If the length of the MCSRE match result is not equal to the length of the CRE match result, the text contains content in the current script. As each MCSRE includes space encoding, if the text contains multiple scripts, some matches will have consecutive spaces. After replacing consecutive spaces in the match results with a single space and removing spaces at the beginning and end of the text, the script and match results are added to the results dictionary. Then, MCSRE matching with the next script will be performed on the text.

3)

After MCSRE matching is completed for the text and all scripts, a results dictionary is returned, which provides the contents in different scripts and the associated script names.

3.2. Improving the Time Efficiency of Script Identification

In the proposed algorithm, the time efficiency of SI is improved through implementation of the following four methods.

1)

The text may contain content in multiple scripts. As the text length is fixed and the sum of the lengths of the content in each script is equal to the total text length, if the total length of the recognized script content equals the length of the text during the SI process, there is no need to analyze the remaining scripts and the process should be terminated immediately.

2)

On the Internet, the proportion of text written in different languages varies. Based on the correspondence between languages and scripts, the proportion of commonly used scripts in online text information can be deduced. Based on this, in an SI algorithm, scripts that appear more frequently on the Internet should be given higher priority. Some websites [14] provide statistics on Internet content in different languages, but there are no corresponding corpora. Therefore, in this study, the constructed script recognition corpus was analyzed and the weight of a given script was set according to the number of sentences contained in the associated corpus. Table 4 shows the number of sentences in each script corpus. When constructing SREs and MCSREs, regular expression sequences were sorted according to the number of sentences in descending order. When using the SI algorithm to identify scripts, the text should be matched with the regular expression of the script with the highest weight first, then analyzed with the script with the next highest weight, and so on. Ideally, a text will contain only one type of script content; if this is the case, the target script can be quickly located based on above matching pattern. Once located, there is no need to continue analyzing the remaining scripts. This can improve the time efficiency of SI.

3)

During the SI process, determining whether the text contains content written in a specific script is a key operation. If there are many scripts that need to be identified, this operation must be performed multiple times. In preliminary research, MCSRE matching operations were used to analyze whether the text contains content written in the current script. As SREs are shorter than MCSREs, we chose to use SRE matching operations to improve the matching speed. Each time SRE matching is performed, if a match result is returned, the corresponding script and matching result are added to the SI result dictionary.

4)

The results obtained from SRE matching do not include punctuation marks, numbers, spaces, and certain characters which are common to multiple scripts. If these symbols are not used for SI, the completeness of the text information cannot be ensured, which affects subsequent text processing results. To ensure the integrity of the information in the SI results, MCSREs were constructed in the preliminary stage of research. This type of regular expression includes not only symbols specific to each script, but also various symbols shared by multiple scripts. The MCSRE matching results can ensure the integrity of the SI results. For sentences containing multiple scripts, where the sentence structure is relatively complex, executing MCSRE matching may result in some symbols that belong to other scripts being included in the results. Therefore, in this study, based on the results obtained through SRE matching, the indices of the first and last characters of the result are first located in the original text. Then, according to these two indices, the text segments where the script content appears are obtained. Finally, the text segments are matched with the MCSREs of the corresponding scripts and consecutive spaces are replaced with a single space, resulting in the final script content.

3.3. High-Time-Efficiency Script Identification Algorithm

In this study, the above four methods are combined to improve the execution efficiency of an SI algorithm based on the ISI algorithm proposed in previous research [13], and the high-time-efficiency script identification (HTESI) algorithm is proposed. In the preliminary study, two types of regular expressions—CRE and MCSREs—were considered to analyze the SI process. Three regular expressions for matching are used in this study, as follows:

1)

CRE matching reveals characters shared by multiple scripts in the text, including punctuation, spaces, numbers, characters and other symbols;

2)

SRE matching is used to obtain the characters in the text that are used only by the current script. If the match length is non-zero, this indicates that the text contains content written in the current script. Otherwise, the text does not contain content written in the current script;

3)

MCSRE matching is performed after determining the type of script and matching result appearing in the text. Each script identified in the SI results is matched with the original text using MCSRE to obtain complete recognition information.

The workflow of the HTESI algorithm follows the flow diagram shown in Figure 4, comprising the following steps:

1)

The text length $L_t$ is calculated and the variable $L_{sum}$ is defined to calculate the total length of the SI result.

1)

This text is matched with CREs to extract the parts of the text where the script tag is ‘common’ and the length $L_c$ of these texts is calculated. At this point, $L_{sum}=L_c$

3)

The text is matched with the SRE of each script. When analyzing the matching results for each script, one of the following two situations will occur:

4)

If the matching result length $L_m$ is equal to zero, execute the SRE matching operation of the next script.

5)

If the match result $l_m$ is not zero, add the corresponding script to the SI result dictionary and calculate $L_{sum}=L_{sum}+L_m$. Then, check if $L_{sum}$ is equal to $L_t$: if so, stop the SI; otherwise, perform the matching operation for the next script’s SRE.

6)

After completing SI, for each script and matching result in the SI dictionary, the start and end characters of the matching result are located in the original text to determine the text segment in which the script appears. Then, a matching operation is performed between this text segment and the corresponding script’s MCSRE, consecutive spaces are replaced with a single space, and the matching results—along with the corresponding script—are added to the final SI result dictionary.

3.4. HTESI Example

To illustrate the HTESI process, we selected the following sentence containing multiple scripts from the corpus: “Bloomberg News сo ссылкoй на прoект заявления G7 пo итoгам заседания.” The HTESI process for this sentence is detailed in Table 5. In this study, weights were set based on the number of sentences in each script corpus and the MSE and MCSRE sequences were sorted in descending order of weight. Most content in the corpora was in Latin script, followed by Slavic languages. Therefore, in this case, the text was first matched with the Latin SREs. As the sum of the SI results after matching was less than the length of the original text, the text was then matched with the SREs corresponding to the second-ranked Slavic language in the SRE sequence. When the sum of the SI results after matching was found to be equal to the length of the original text, there was no need to continue analyzing the remaining scripts in the SRE sequence and the SI process stopped. After SI was completed, the segment of the original text where the script content appears was located for each result, which was then matched using MCSREs to form a complete SI result.

4. Results

To verify the feasibility of the algorithm proposed in this study, training and testing experiments were conducted using the SI dataset mentioned in Section 2.3. The ratio of the training set to the test set was 9:1. To verify the proposed SI algorithm’s efficiency, we designed three different experiments and performed comparative experiments using the ISI algorithm proposed in previous research [13]. The experimental environment of this study was a MacBook Pro with an Apple M1 chip,8GB of memory and the program was written in Python 3.11.5 using the Spyder 5.4.3 IDE.

SI is a text classification task, in which each entry in the text classification corpus must have a label and, so, the text classification algorithm must return a label when predicting the text. When evaluating the performance of an SI algorithm, it is used to determine which script is contained in the text; therefore, the number of words separated by a space was counted, and the script with the most words was selected as the main script of the text. As Han scripts encode more information in a shorter number of characters than, for example, Latin scripts, the Chinese and Japanese languages in the Han group are written without spaces between the words. Thus, for the Han group, the length of the matching result was considered, except for the spaces.

The performance of the text classification algorithms was evaluated using Precision, Recall, and F-score. This study used the $\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}$, $\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}$, and ${\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{F}}_1$ metrics to evaluate the performance of the SI algorithm, as shown in Equations (1)–(3), in which C represents all the scripts in the SI data set; |C| denotes the number of scripts in the SI data set; and the meanings of TP, FP, and FN are given in Table 6.

$$\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}}{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}+\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{F}\mathrm{P}}_{\mathrm{i}}}},$$

(1)

$$\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}}{\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}+\sum _{\mathrm{i}=1}^{\left|\mathrm{C}\right|}{\mathrm{F}\mathrm{N}}_{\mathrm{i}}}},$$

(2)

$${\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{F}}_1={\displaystyle \frac{2\times \mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}\times \mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}}{\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{P}+\mathrm{M}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{R}}}.$$

(3)

4.1. Impact of SRE Matching on SI

In the preliminary study, when determining whether a text contains a certain type of script, the text was matched with the corresponding MCSRE of that script. The purpose of this was to ensure the integrity of the SI result. The main operation in SI is to match text with the regular expressions of each script. In this study, SREs were initially used for matching to reduce the matching time, as they are simpler than MCSREs.

To verify this method’s validity, an algorithm was designed and labeled SI-1. The workflow of SI-1 is almost the same as that of the ISI algorithm, with the difference being that when determining whether the text contains a certain script, an SRE is used instead of an MCSRE, and segments which contain a certain script are determined after identifying different scripts in the text and then matched with the MCSRE of the corresponding script to obtain the SI results. To better understand the experimental process, Figure 5 shows the flowchart of SI-1.

The results of a comparative experiment between SI-1 and ISI are shown in Table 7. These results show that using SREs improved the matching speed; in particular, the training and testing times of the newly proposed method were reduced by 8.61- and 8.56-fold than those of previous study, respectively, respectively, while the Micro F1 score for script identification was slightly higher than those reported in our previous study.

4.2. The Impact of Calculating Match Length on SI Algorithm Performance

The sum of the lengths of all script texts that appear in the text is equal to the total text length—a potential advantage that was not considered in previous research. If the sum of the identified SI results is equal to the text’s length, there is no need to perform further regular expression matching operations between the remaining scripts and the text. In this research, this advantage is leveraged to improve the proposed algorithm’s time efficiency. To verify the validity of this method, an algorithm was designed, labeled SI-2. The workflow of the algorithm is the same as that of the algorithm proposed in this research (shown in Figure 4); the main difference is that the frequency with which each script appears in the corpora is not considered—the order of the regular expressions for each script used in the comparative experiment is instead consistent with the order of scripts on the official Unicode website.

The results of the comparative experiment between SI-1, SI-2, and ISI are shown in Table 7, showing that the algorithm’s time efficiency was further improved by considering the length of the SI results, the training and testing times were reduced by 8.34- and 8.23-fold than those of previous study respectively, respectively, without reducing the algorithm’s performance.

4.3. The Impact of the Frequency of Different Scripts on SI Performance

On the Internet, the proportion of text written in different scripts varies. When designing an SI algorithm, if the regular expressions for each script are sorted in descending order based on their frequency on the Internet, it is likely that fewer pattern analyses will need to be performed, and the most-used scripts can be identified rapidly; in this way, the algorithm’s time efficiency can be further improved. As there is no relevant open-source corpus available for this purpose, in this study the weights were set based on the number of sentences in each script corpus (as detailed in Table 4), and the SREs and MCSREs were sorted according to these weights. After ranking the SREs and MCSREs in this way, the HTESI algorithm proposed in this study could be implemented; its process is shown in Figure 4. The results of a comparison of the relevant experimental results are shown in Table 7, the training and testing times were reduced by 8.61- and 8.56-fold than those of previous study, respectively. It is indicating that the SI algorithm’s time efficiency was further improved through weighting by the prevalence of different scripts, without any adverse effects on the algorithm’s performance.

5. Conclusions

SI is the first step in most multilingual text processing tasks. To improve the time efficiency of the SI algorithm, it was optimized in the following three ways in this study:

1)

The main operation in the SI process is to determine whether a certain type of script is present in the text. In previous research, to provide complete SI results for subsequent text processing, each script-specific character encoding and the character encodings shared among scripts were merged to construct an MCSRE for each script. As the content represented by this regular expression is relatively extensive, performing regular expression matching requires a significant amount of time. To reduce the time needed for this process, we constructed SREs for each script using only script-specific character encodings. As these SREs carry far less information than MCSREs, performing SRE matching requires less time. Therefore, in this study, SREs are used to determine whether certain scripts are present in the text. If present, MCSREs are then used to obtain complete SI results.

2)

In preliminary research, to determine which script is present in a text, the text was matched against the regular expressions of various scripts using regular expression matching. This study takes advantage of the fact that the sum of the lengths of content written in different scripts within a text is equal to the total length of the text by calculating the sum of the lengths of each recognition result during the SI process. If this sum is equal to the length of the text, there is no need to analyze the remaining scripts and the SI process is stopped.

3)

The frequency of text in different languages on the Internet varies; similarly, the frequency of content written in different scripts also varies. Therefore, when designing an SI algorithm, it is preferable to first analyze those scripts that appear more frequently to identify them rapidly. This study also takes advantage of this practical situation: when designing the list of regular expressions, the expressions were sorted in descending order according to the frequency of content written in each script in the corpus. This allows the SI process to determine content in different scripts present in the text more quickly.

The experimental results show that, after applying these optimization strategies, the time efficiency of the resulting HTESI algorithm noticeably improved without reducing the identification rate, thus demonstrating the effectiveness of the optimization methods proposed in this study.