On Move with Interleaving (MwI) Implementation

1. Introduction

Careful implementation of algorithms was a strong focus in the past due to considerably limited computer resources [1,2,3,4]. Today, however, there is an illusion among average programmers that this problem is no longer relevant. The compilers generate efficient executable code and offer additional optimisation features [5,6]. Better personal computers are equipped today with gigabytes of on-board memory, and their CPUs can perform calculations at a rate of hundreds of megaflops [7]. However, with better computers, even more data are produced everyday [8]. Supercomputers [9], GPGPU installations [10] and cloud-based solutions [11] have been developed for processing them. Unfortunately, the majority of users do not have access to such installations; indeed, they operate modest personal computers, processing manageable amounts of data for which they expect to receive results quickly. This is why even a short segment of programming code, executed frequently, should be understood, not only arithmetically, but also from an implementation perspective. Programming code could be optimised in various aspects, such as, for example, in terms of spent CPU time [12], required memory [13], or energy consumption [14]. An approach to reduce the spent CPU time of a recently proposed string transformation technique named Move with Interleaving [15] is considered in this paper. This transformation reduces the information entropy [16,17] as efficiently as the combination of the Burrows-Wheeler transform [18,19,20] followed by Move-To-Front transform [21,22] or Inversion Frequencies transform [23]. Move with Interleaving can, therefore, be used as a replacement for these transforms in the preprocessing stage of data compression algorithms [22,24], reducing the length of the programming code. Although its theoretical time complexity is linear with respect to the length of the sequence to be transformed, it requires careful implementation to achieve an acceptable spent CPU time. The main contributions of the paper are as follows:

• Highlighting the awareness of computer scientists, and especially programmers, that better computers do not solve the problems of shortening CPU time without careful implementation;
• Providing information about CPU spent time using two popular programming languages, C++ and Java;
• Testing the efficiency of the implementations on various platforms using different compilers.

The paper is organised in five Sections. Move with Interleaving is, together with the Move-To-Front transform, introduced briefly in Section 2. Three various implementations of Move with Interleaving are described in Section 3. The experimental results are given in Section 4, while the last, fifth Section contains the conclusion.

2. Background

2.1. Move-To-Front Transform

Move-To-Front (MTF) is one of the self-organising list strategies [21,25,26,27] used in caching algorithms [28] and data compression [29,30]. Let $X=\langle x_i\rangle$, $0\le i<n-1$, represent the input sequence of n elements $x_i$ from an alphabet ${\Sigma }_X$, where ${\Sigma }_X$ consists of m elements. MTF reads the elements $x_i\in X$ sequentially, and maps them to the domain of natural numbers, including 0, to obtain the output sequence $Y=\langle y_i\rangle$, $0\le i<n$, $y_i\in {\Sigma }_Y=\{0,1,2,\cdots ,m-1\}$. The list L is employed in this mapping process and it is populated with all the elements from ${\Sigma }_X$ initially. The algorithm finds the position l, $0\le l<m-1$, in L, where $x_i=L_l$ and sends l to Y. After that, the positions of all the elements $L_e$ in L between $0\le e<l$ are increased, while $x_i$ is placed in front of L. The pseudocode for the MTF transform is shown in Algorithm 1. The main loop contains only three functions. The first one, on Line 7, iterates through the elements in L and stops when $L_l=x_i$. It is expected that $x_i$ is found in the constant time $O\left(1\right)$, since MTF is a self-organising list. The function in Line 8 terminates in $O\left(1\right)$, as it just writes l to an array representing Y. The most time-consuming task is performed by the last function in Line 9, which needs to replace the first $l-1$ values in L. It can be realised in various ways. Four different implementations were proposed and analysed in [31].

MTF is used most frequently in data compression [26,27,29], as it reduces the information entropy of X when X contains local correlations. For example, a sequence of the same elements is converted into a sequence of zeros; a sequence of alternating elements gives a sequence of ones; a sequence of alternating triples results in a sequence of twos, and so on. This is the reason why MTF is used typically after BWT, which tends to group the same elements close together [19,20].

Algorithm 1 Pseudocode of MTF transform

2.2. Move with Interleaving

The Burrows-Wheeler transform, (BWT) should be used before MTF to enhance the local correlations of X, as mentioned in the previous Subsection. BWT was introduced in 1994 [18], and is a well-studied transform [22]. Unfortunately, it has $O(n^{2}log\left(n\right))$ time complexity, which is why X should be split into small blocks to achieve a fast enough response [18]. A direct correlation between suffix array and the BWT was determined later [19]. Straightforward implementation of the suffix array also works, unfortunately, in $O(n^{2}log\left(n\right))$ [32,33]. Fortunately, more algorithms were developed for constructing a suffix array of X in $O\left(n\right)$ [34,35,36]. This also enables obtaining the BWT in linear time. However, the implementation of these algorithms is relatively demanding. This is why it would be desirable to replace the pipeline of the three algorithms – an algorithm for constructing the suffix array, an algorithm to generate BWT from the suffix array, and the MTF algorithm – with only one compact algorithm. It was shown recently that, at least for the domain of Raster Images, this can be done by the transform named Move with Interleaving (MwI) [15]. Pixels from a continuous-tone image are arranged into a sequence X at first. Pixels in the close neighbourhood of the considered pixel are, in the majority of cases, similar, but not completely the same. However, this similarity is sufficient for MTF to produce small indices $y_i$. But, when a region of different pixel values is encountered, MTF has to generate many large indices $y_i$ before bringing enough values from this region close to the front of L. This has, unfortunately, a negative effect on the reduction of the information entropy, and this is why BWT should be applied before MTF. Because of this, MwI monitors the value $y_i=MTF\left(x_i\right)$, and reacts when $y_i$ exceeds the given threshold t. MwI assumes that an area with different values is found and that the values similar to $x_i$ will follow. Because of this, it inserts, in addition to $x_i$, $2t$ interleaved values around $x_i$ in front of L, i.e., the values from a generated auxiliary sequence $A=\langle x_i,x_i+1,x_i-1,x_i+2,x_i-2,\cdots ,x_i+t,x_i-t\rangle$.

The MwI pseudocode is shown in Algorithm 2. The algorithm first populates the list L in Line 5. It uses the first element from X and interleaves it with $2t$ values. These values are inserted in front of L, and the remaining values from ${\Sigma }_X$ are filled in alphabetical order. The first element $x_0$ is stored immediately in the output Y in Line 6, as the decoder should perform the same initialisation of L. The algorithm then enters the main loop and processes $X=\langle x_i\rangle$, $1\le i<n$, sequentially. The position l of $x_i$ is determined in L (Line 8). It checks after that whether l is within the given threshold t. If so, the MTF procedure is performed in Lines 11 and 12; otherwise, the algorithm fills the auxiliary sequence A with interleaved elements around $x_i$ in Line 14. The elements from A are then inserted in L in the MTF-style within Lines 16 and 17.

Algorithm 2 Pseudocode of MwI transform

Finally, let us analyse the MwI time complexity. $l>t$ in each iteration for the worst-case scenario. Therefore, for each $x_i$, $1\le i<n$, the algorithm fills A in time $T_A=2t+1$, and then performs the MTF $2t+1$ times. MTF has the expected time complexity $O\left(n\right)$, and, as a result, the expected execution time of MwI is $T_{MwI}=(2t+1)O\left(n\right)$. Since $t\le m<<n$, it can be concluded that MwI’s time complexity is the same as MTF, i.e., $O\left(n\right)$.

3. Materials and Methods

Three different implementations of MwI, denoted as I-1, I-2, and I-3, are considered in this Section. The programming code is presented in C++ [37], with a note that the paper does not make use of its object-oriented features.

3.1. Implementation I-1

List L is represented as the standard C++ vector in this implementation. Algorithm 3 shows the function that initialises L. This initialisation assumes that the alphabet represents the numbers from the range $[lBound,uBound]$. The last parameter N in function InitL_I controls the number of elements to be inserted. It is set to $uBound-lBound+1$ initially. Variables u and l hold the incremental interleaving values around FirstEl.

Algorithm 3 Implementation I-1: initialization of L

The main function of implementation I-1 is shown in Algorithm 4. The position l of each $x_i$ in L is determined in Line 9 and then stored in Line 10. After that, l is tested (Line 11): if it is smaller than t, the MTF is executed in Lines 12-14; otherwise, more elements interleaved around $x_i$ should be moved-to-front of L. The auxiliary vector A is filled with at most $1+2t$ interleaved values by calling the function InitL_I in Line 17. In Algorithm 3, this time the parameter N gets the value 2t. After that, the content of L is updated in Line 18.

Algorithm 4 Implementation I-1: MwI main function

Algorithm 5 Implementation I-1: L update function

Finally, let us explain the updating procedure for L, when more elements should be moved to the front (see Algorithm 5). In Line 4, $1+2t$ interleaved values from vector A are first copied to a temporary vector Temp. The remaining values of L are added incrementally within the Lines 6– 17 to Temp if they have not yet been added in Line 4. When the for-loop terminates, vector Temp contains the updated status of L, which is then returned in Line 18 for use in the next iteration of Algorithm 4.

3.2. Implementation I-2

In implementation I-2, the data structure used to store L is once again a standard vector. Consequently, the initialisation process remains identical to that given in Algorithm 3. The primary drawback of implementation I-1 is in the update function outlined in Algorithm 5, which contains two nested loops. The first loop iterates through all the elements of L. The nested while loop scans the auxiliary vector A to verify whether the element L[i] also exists in A. If not, the element is added to the temporary vector Temp. Although the UpdateL function is not called for each element being transformed, and the length of the vector A is expected to be small, $1+2t<<m$, it opens up room for an improvement. Namely, the nested while loop can be omitted by introducing a lookup table $LUT$. Its purpose is similar to the lookup table in the Counting-sort algorithm [39]. In our case, $LUT$ consists of m flags set to FALSE initially. The flags in $LUT$ are set during the creation of the auxiliary array A, i.e. each flag is raised at the position $LU{T}_{A_i}$. During the updating of L, for each element in L it checks the status of the flag at the location $LU{T}_{L_i}$. The element is added into the temporary vector if the flag is FALSE. Details, including the construction of vector A and the updating of L, are presented in Algorithm 6.

Algorithm 6 Implementation I-2: MwI with LUT

An array of flags LUT is allocated in Line 5. It consists of m elements, i.e. the size of the alphabet ${\Sigma }_X$. In this implementation it is assumed that the alphabet $\Sigma$ contains elements from the interval $[0,m-1]$. If this is not the case, a simple mapping of the alphabet’s element into this interval should be done before calling the function MwIwithLUT. When the position l of the considered element $x_i>t$, all flags in LUT are set to FALSE in Line 19. The flag belonging to l is set to TRUE after that in Line 20. The same is applied for all the remaining interleaved values in Lines 26 and 30. The update of L is started in Line 34 after A has been prepared. The for loop operates for all elements of ${\Sigma }_X$. For each element of L, it is checked whether its flag in LUT is set (Line 35). If it is not, the considered element is inserted into A. Finally, A is copied into the new L in Line 37.

3.3. Implementation I-3

Implementation I-3 utilises a dynamically linked list of records to represent L. Each record has only two components: el, containing the value, and a pointer next that points to the subsequent record. Concerning MTF, it seems that this data structure could offer benefits, as there is no requirement to move the elements at positions 0 through $l-1$.

Let us consider an example of L shown in Figure 1a, containing 10 elements. Suppose the element that should be moved to the front of L has the value 7. The record containing this value has already been found and is pointed to by pointer w; its preceding pointer is denoted as wPrev. The new status of L is established in constant time by repointing just three pointers, as illustrated in Figure 1b. The details are provided in Algorithm 7.

Algorithm 7 Implementation I-3: MTF

The concept of how the MwI transformation can be realised using a dynamic linked list is shown in Figure 2. Let us suppose that $x_i=7$ is again the considered value in L, which is shown in Figure 2a. In addition, let us suppose that $t=2$. The position of $x_i=7$ is at $l=5$, and as $l>t$, the auxiliary list A of $1+2t$ values should be appended in front of L. The auxiliary list A is shown in Figure 2b. L is inspected, and all its records holding values within the range $[x_i-t,x_i+t]$ are omitted (see Figure 2c). Finally, L and A are linked together using the pointer $ATail$; the result is shown in Figure 2d. We can observe that $1+2t$ new records have been created, and an equal number of them have been deleted. Memory allocation and release are, unfortunately, computationally expensive operations, and should be avoided whenever possible [38]. Therefore, records that should be released, are placed in a stack named Pool in the continuation. Whenever a new record is needed during the process of constructing A, it is taken from the Pool. Details of the implementation can be found in Algorithm 8. In Line 8, the function PreparePool inserts $1+2t$ dynamically allocated records into the stack pointed to by Pool. Function GetPosition (Line 10) performs the same tasks as the programming code in Lines 8 to 14 of Algorithm 7. After that, the records containing the values from the interval $[x_i-t,x_i+t]$ are inserted into Pool by the function in Line 14. The function ConstructA (Line 15) constructs list A with interleaved values around the value $x_i$ using the records from the Pool. It returns pointers A and $ATail$. In Lines  16 and 17, lists L and A are merged into a new list L. In Lines 20, 21, and 22, the record is moved in front as explained earlier.

All implementations are accessible at [40].

Algorithm 8 Implementation I-3: MwI

4. Experiments

All three proposed implementations of MwI were tested on 32 benchmark 8-bits greyscale images. The images can be seen in [15], and, in this paper, they are denoted in the same way. The values of pixels were arranged into the sequence X in the scan-line order. All three implementations were coded in both C++ and Java, as the most popular programming languages, and they were executed on three different platforms:

P1:

A personal computer with AMD Ryzen 9 7950X3D 16-Core Processor clocked at 4.20 GHz, equipped with 64 GB of RAM, running the Windows 11 operating system. The code was compiled using MS Visual Studio, Version 17.4.2. OpenJDK-8-JRE was used for Java.

P2:

The same personal computer as in P1, but using Linux Kernel 6.6.8. with the Manjaro distribution. The code was compiled with G++ compiler Version 13.2.1. for C++. OpenJDK-8-JRE was used for Java again.

P3:

The Raspberry Pi 2 Model B running Raspbian GNU/Linux 11 (bullseye). The C++ code was compiled with gcc version 10.2.1 20210110 (Raspbian 10.2.1-6+rpi1), while OpenJDK-8-JRE was used for Java.

Table 1 illustrates the CPU time spent for all three implementations across the three platforms for code written in C++.

The results were indeed stunning. Implementation I-2 was 562% faster than I-1. However, Implementation I-3 surpassed I-2 by 334% and outperformed I-1 by an astonishing 1883% on platform P1. Similarly, on platform P2, I-3 was 1908% faster than I-1 and surpassed I-2 by 246%. Interestingly, despite using the same computer, platform P2 outperformed platform P1 consistently. While I-1 and I-3 differed by 4% and 5%, respectively, platform P2 surpassed P1 for I-2 significantly by 42%. The pattern repeated on platform P3, but with a more pronounced effect. I-3 was, this time, faster than I-1 by more than 2300%, and by more than 550% from I-2. However, in this case, the absolute spent CPU time also becomes important. The user should wait for 7 and a half minutes to transform all 32 images using implementation I-1, while I-3 requires only 19.5 seconds.

The results for the Java implementation are provided in Table 2. Implementation I-3 again surpassed I-1 and I-2 by more than 1400% and 680%, respectively, on platform P1. Similarly, on platform P2, I-3 outperformed I-1 by over 1270% and I-2 by 580%. Platform P2 was faster than platform P1 for I-1 and I-2, but not for I-3. On platform P3, implementation I-3 outperformed I-1 by more than 6700%, and I-2 by almost 1000 %.

As expected, the Java implementation was slower than C++ implementation (compare the results in Table 1 and Table 2). Let us consider the results on the personal computer, i.e., on platforms P1 and P2. The most evident difference is at I-2, where platforms P1 and P2 lag behind the C++ implementation by more than 360% and 480%, respectively. Platform P2 was faster for I-1 and I-2, while it was slightly slower for I-3 (by less than 7%). The Java implementations were significantly slower than the C++ implementations on platform P3. Implementation I-1, written in Java, was slower by a hardly believable 1358%, I-2 by more than 800%, and I-3 by more than 460%. In the case of I-1 coded in Java, the user would need to wait for 1 hour and 42 minutes to transform all 32 images, while, for I-3, this process is completed in one and a half minutes.

5. Conclusion

This paper considers three different implementations of the Move with interleaving (MwI) transform. Although MwI operates with the expected $O\left(n\right)$ time complexity, its CPU time usage differs considerably between various programming solutions. Although efficient implementations are necessary in embedded systems or computationally weaker platforms like Raspberry Pi, processing larger datasets can also be noticeably faster on modern personal computers.

Three different implementations of MwI were considered in the paper: the first utilised a standard vector to store the MwI status, the second incorporated a lookup table, while the third was based on a dynamic one-way connected list and a mechanism for reusing allocated memory chunks. All the implementations were encoded in C++ and Java. Additionally, three different platforms were used for the experiments: MS Windows 11, Linux with the Manjaro distribution, and Raspberry Pi with Linux. The first two platforms were run on the same computer to ensure a fair comparison. The experiments involved 32 continuous-tone greyscale raster images with various resolutions and contexts. Their pixels were transformed into a sequence using a scan-line approach and then processed with MwI. The results show that a careful implementation reduces the CPU time spent significantly. For example, the implementation based on the dynamic linked list was 1800% faster than the implementation using the standard vector. The lookup table accelerated the implementation, using a standard vector significantly; however, it did not reach the speed of the implementation with the dynamic linked list. Linux turned out to be slightly more efficient that Windows. As expected, the C++ implementations outperformed the Java implementations.

The Raspberry Pi platform cannot be compared directly to the two platforms running on a modern personal computer. However, the importance of careful implementation and the choice of programming language becomes even more evident. For example, the fastest implementation using a dynamic one-way connected linked list took 19 second if coded in C++, but one and a half minutes if coded in Java. The slowest implementation, using a standard vector, required 7 minutes and 32 seconds, while the same implementation coded in Java needed even 1 hour and 42 minutes.