Next generation blockchain technology: The Entropic Blockchain

1. Introduction

A blockchain is an online global database accessible to anyone via the Internet, at any time [1]. In blockchain databases, the information is recorded, maintained and shared by a community. The blocks within a blockchain contain information about transactions. A typical transaction is a data structure that defines a transfer of information or value. In this sense, a transaction can be the operation of storing information to the block, extracting information from the block, transferring value from one entity to another, a contract, and so on. Blockchain technology combines many other technologies, like cryptography, peer-to-peer networks, smart contracts, and consensus mechanisms to make it nearly impossible to hack or tamper with the transactions and information stored within the blocks. Crypto currencies such as Bitcoin [2] are an application of blockchain, which are related to day-to-day digital payment-based systems. In order to create a new block or to verify a transaction, computers must solve complicated math problems in a process called mining. The node that completes the mathematical puzzle the fastest receives a small Bitcoin reward and gets the right to certify the transaction. Blocks are then appended to the blockchain in chronological order, as they are created.

The main issue with the blockchain technology is the ever-increasing size of the blockchain, which makes its storage and validation process more complicated. A single block in the Bitcoin blockchain is around 1-2 Mb size, while the entire Bitcoin blockchain is today 578.48 Gb (on 12 June 2024), making it almost impossible to run it on conventional laptops and desktops [3]. Moreover, its daily growth rate is around $0.04\%$, and its annual growth rate is $16.91\%$.

Another issue is the increasing computational power requirements to mine blocks and to perform the validation. The proof of work consensus mechanism uses a lot of power. For example, the electricity consumption of Bitcoin is estimated to be 172.26 terawatt-hours (TWh) per year (on 12 June 2024), which amounts to 96.08 Mt CO2 equivalent. The average energy consumption of Bitcoin per transaction is 766.64 kilowatt-hours (kWh), which has a carbon footprint of 427.60 kg CO2 [4]. These huge numbers are having a direct impact to the power requirements to run the blockchains, but they are also having a measurable environmental impact.

In this article, we propose a novel method that allows the scalability of blockchain databases in the download, validation and confidentiality process, by developing a lightweight blockchain technology called Entropic Blockchain, for simultaneous data compression and encryption, while also facilitating a unique methodology for reaching consensus in real-time. This novel method uses Shannon’s Information Entropy (IE) [5] to generate the Entropic Barcode of a dataset, and it is the subject of a recent patent application [6].

2. Calculating the Information Entropy

Let us assume that a given set X contains N characters, each chosen from a set with radix $U\ge 2$. Let $N_l$, where $l=1,2,3,\cdots ,U$, be the number of $a_l$ characters in the set X. The fractions $p=\{p_1,p_2,\cdots ,p_U\}$ can be defined for the set X, where $p_l=N_l/N$ is the fraction of the occurrence of a character $a_l$ within the set X. According to Shannon [5], the average information extracted per character, or the number of bits of information per character, or the information entropy of the set X is:

$$\mathrm{IE}=-\sum _{j=1}^U{p}_j·{log}_b\left(p_j\right),$$

(1)

where the base b gives the units of information (e.g., $b=2$ for bits), and the total bit content of the set is $N·\mathrm{IE}$.

For a given set, the maximum IE is obtained when the fractions are equal to each other (similarly as normal numbers), that is $p_j=1/U$, so:

$$max\left(\mathrm{IE}\right)={log}_b\left(U\right).$$

(2)

The IE given by (1) is computed when the fractions refer to single characters within a set. However, a useful extrapolation could be the generalization of relation (1) to m-block information entropy [7]:

$${\mathrm{IE}}^{\left(m\right)}=-\sum _{j=1}^{\Omega }{p}_j^{\left(m\right)}·{log}_b\left(p_j^{\left(m\right)}\right),$$

(3)

where instead of single characters, combinations of m characters are used to define a new set of characters, called m-blocks. In this case, $p^{\left(m\right)}$ are the fractions of the m-block characters, and the summation extends over all possible combinations of distinct m-blocks. For a given set of N characters each chosen from a set of radix U the maximum number of distinct m-blocks of the newly constructed set of m-block characters is:

$$\Omega =U^m.$$

(4)

The maximum value of the IE theoretically permitted for the new set of m-blocks is:

$$max\left({\mathrm{IE}}^{\left(m\right)}\right)={log}_b(\Omega )={log}_b\left(U^m\right)=m{log}_b\left(U\right)$$

(5)

Combining (2) and (5), we deduce that $max\left({\mathrm{IE}}^{\left(m\right)}\right)=m·max\left(\mathrm{IE}\right)$, so using m-blocks increases the IE value by a factor of m relative to the set of single characters. In order to clarify the methodology proposed here, it is useful to show a few examples. Let us assume that a given set of characters contains characters form a set of radix $U=2$ (only two distinct single characters). Using bits, $b=2$ and relations (4) and (5), then:

If $m=1$, $\Omega =U^m=2$, indicating that we have two possible states and each state encodes $\mathrm{IE}=1$ bit per character:

$$\{0,1\}$$

If $m=2$, $\Omega =U^m=4$, indicating that we have four possible states and each state encodes $\mathrm{IE}=2$ bits per character:

$$\{01,00,10,11\}$$

If $m=3$, $\Omega =U^m=8$, indicating that we have eight possible states and each state encodes $\mathrm{IE}=3$ bits per character:

$$\{000,001,011,101,110,100,111,010\}$$

If $m=4$, $\Omega =U^m=16$, indicating that we have 16 possible states and each state encodes $\mathrm{IE}=4$ bits per character:

$$\{0000,0001,0011,0111,1111,1001,0100,0101,0111,0010,1100,1110,1101,1110,1000,0110\}$$

If the set has N characters and we take $m=N$, then $\Omega =U^m=2^N$ is the number of possible states and each state encodes $\mathrm{IE}=N$ bits per character.

3. Data Segmentation into Windows

Consider again our set of N characters $X=\{x_1,x_2,\cdots ,x_N\}$. First, we create a subset called “window”, containing a number of characters called “window size”, WS. Taking a number of characters from $x_1$ to $x_{\mathrm{WS}}$ where $\mathrm{WS}\le N$, creates the first window. Starting from left to right, one slides the segment of size WS across the whole set, where the position of each new window is obtained by sliding WS from left to right for a fixed number of characters, called “step size”, SS. In order to ensure that all sections of the set are captured by this process, the SS must be at least 1 and maximum WS, so $1<\mathrm{SS}\le \mathrm{WS}$. By doing this, a given set of N characters, will result in a new set of $N_W$ windows, $\{W_1,W_2,\cdots ,W_{Nw}\}$, given by the formula:

$$N_W=1+\frac{N-\mathrm{WS}}{\mathrm{SS}}.$$

(6)

Ideally, the SS is taken to ensure the ratio $(N-\mathrm{WS})/\mathrm{SS}$ is an integer, by selecting SS to satisfy the relation

$$N=A·\mathrm{SS}+\mathrm{WS},$$

(7)

where A is positive integer. However, in order to ensure that $N_W$ is an integer, another alternative is to augment the dataset by $N_P$ additional blank characters to cover the entire dataset with windows:

$$N_P=N_W·\mathrm{SS}+\mathrm{WS}-N.$$

(8)

The new augmented set has $\widehat{N}=N+N_P$ characters. Figure 1 shows this procedure applied to a set of bits. In the case illustrated in Figure 1, $\mathrm{WS}=20$ and $\mathrm{SS}=9$. Starting from left to right, the first window is formed, then sliding this to the right by SS, the second window is obtained, and so on until the whole set is split into $N_W$ windows, forming a new set of $N_W$ elements.

The link between the index of a given $j^{\mathrm{th}}$ window and the index of the first character in the set corresponding to the $j^{\mathrm{th}}$ window is given by the formula:

$$N_j=\mathrm{SS}·W_j-\mathrm{SS}+1$$

(9)

where, $N_j=1,2,3,\cdots ,N$ and $W_j=1,2,3,\cdots ,W_{N_W}$, with $N_W$ given by relation (6).

4. Generating a Set of $\mathbf{m}$-Blocks

We already established that a new set constructed using m-blocks from a given set of N single characters, containing U distinct characters, will have $\Omega =U^m$ distinct m-blocks/characters. We are now working out how many m-block elements will be in the newly formed set of m-blocks. Just as the “windows” segmentation of the set of N characters resulted in a new set containing $N_W$ windows, the same set could be transformed into a new set of m-blocks containing $N_m$ elements. The set of m-blocks is constructed using the same procedure used for generating the set of windows. However, instead of sliding the segment of size WS from left to right, in step size SS, where $1<\mathrm{SS}\le \mathrm{WS}$, we now slide the m-block segment from left to right in step size of ss, where the condition on ss is now $1<\mathrm{ss}\le m$. When applied to each window, the newly formed set of m-block elements within a window of size WS, contains a number of characters given by:

$$N_m=1+\frac{\mathrm{WS}-m}{\mathrm{ss}}$$

(10)

It is important to observe that the values of m and ss are selected so that the sliding procedure produces a set of $N_m$ integer elements. To clarify this procedure, let’s observe again a few examples. Let’s assume a random set of $N=16$ single characters (this could be a window with $\mathrm{WS}=16$) and U = 2:

$$\left\{0110101101011100\right\}$$

Taking $\mathrm{ss}=1$ and $m=2$, we generate the new set of m-block characters by sliding the m-block segment of two single characters from left to right in steps of 1. This results in the following set containing $N_m=15$ elements, as dictated by (10):

$$\{01,11,10,01,10,01,11,10,01,10,01,11,11,10,00\}$$

Constructing another m-block set with $\mathrm{ss}=1$ and $m=3$, according to (10) we obtain the following set of $N_m=14$ elements:

$$\{011,110,101,010,101,011,110,101,010,101,011,111,110,100\}$$

Using this procedure, a new set of any m-block size, with $m\le \mathrm{WS}$, can be generated. For sets of single characters, the window size has to satisfy the condition $\mathrm{WS}\ge U$, in order to ensure that the IE per window can take all possible values between zero and the maximum value theoretically permitted, ${log}_b\left(U\right)$. In the case of sets of m-blocks, to ensure that the IE per set can take all possible values between zero and the maximum value theoretically permitted, ${log}_b\left(U^m\right)$, the new imposed condition is: $N_m\ge U^m$. Using (10) and solving for WS we obtain:

$$\mathrm{WS}\ge \mathrm{ss}·U^m+m-1.$$

(11)

5. Generating the Entropic Barcode of a Digital File

Using the dataset windows segmentation procedure described above, together with the m-block procedure, the Entropic Barcode of a set is obtained by computing the information entropy IE value of each window, and plotting the IE values as a function of the window index location within the new set. The IE of each window is computed identically using (1) for single character sets, or (3) for sets of m-block characters, with each window containing WS characters, $X=\{x_1,x_2,\cdots ,x_{\mathrm{WS}}\}$, and a number of U distinct characters. The Entropic Barcoding technology is the subject of a recent patent application (GB2404348.1) and allows a conversion of the information contained within any dataset into a compressed numerical set that can be used for further data processing on its own, or as a graphical optically readable barcode [8,9]. The generated Entropic Barcode is a representation of the dataset, which is irreversibly encrypted and it has a massively compressed digital footprint size. In addition to the one-way encryption and data compression, the Entropic Barcode method offers a representation of the original dataset that could be used for data integrity checking, fraud detection, data labeling and fast identification via laser barcode scanning. This technology is applicable to any dataset including genomes, but one of the main applications of the Entropic Barcoding technology is to digital files. Any digital file is composed of 0s and 1s in machine code/binary language. This can be seen as a set of N characters containing two distinct characters, 0 and 1, so $U=2$, $a_l=\{0,1\}$, and fractions distributions are $p=\{p_0,p_1\}$. Using (1), the IE of the set can be easily calculated. To demonstrate this process, let us use a random set of $N=16$ bits:

$$\left\{0110101101011100\right\}$$

If the bits within this set would occur with equal fractions ($p_j=1/2$), then the set would have $\mathrm{IE}=1$, and a total entropy of $N·\mathrm{IE}=16$ bits of information. However, counting the single bits, the above set has the following fractions:

$$p=\left\{p_0=\frac{7}{16},p_1=\frac{9}{16}\right\}$$

(12)

resulting in:

$$\mathrm{IE}=-\left(\frac{7}{16}{log}_2\left(\frac{7}{16}\right)+\frac{9}{16}{log}_2\left(\frac{9}{16}\right)\right)=0.989.$$

(13)

Hence, the IE of this binary set is 0.989 bits instead of 1 bit, and the total IE of the set is 15.84 bits instead of 16 bits. This is the basis of calculating the IE of each window within a set to generate an Entropic Barcode of the set, including a set of bits. The proposed Entropic Barcoding technology can only be implemented using fully automated computer software and we created a proof of concept software called ENtropic BARCoding (ENBARC), which is freely available by contacting the authors. For any given digital file, one can compute the Entropic Barcode representation of the original digital file, by implementing the following steps:

• The file targeted for barcoding is decomposed from its own format into a string of bytes.
• The bytes are then converted into a text file containing a long string of bits, 0s and 1s. All other characters are removed.
• The string of 1s and 0s is split into windows.
• The IE per each window is calculated.
• The Entropic Barcode of the file is generated, which is a text file containing the IE values per window versus window location index.
• The graphical representation of the Entropic Barcode of the file is obtained by plotting the IE values per window versus window location index.

Figure 2 shows diagrammatically the proposed methodology to generate the Entropic Barcode of a digital file, including an example of a graphical Entropic Barcode representation.

6. Application to Blockchain – Entropic Blockchain

Blockchains are decentralized databases. The blocks within a blockchain contain data/transactions in digital format, as well as the hash keys and time stamps. Blocks are then appended to the blockchain in chronological order, sequentially as they are created. Figure 3 shows a diagrammatic representation of a blockchain.

Since blockchains are composed of blocks of digital information, the Entropic Barcoding technology of a digital file described in the previous section can be applied to the blockchain databases, just as detailed in the previous section. This process will be explained here and the result is a superior blockchain, called Entropic Blockchain. The method involves the conversion of all the block’s content into a string of bytes. The result is another block, which is a unique expression of the initial block, but its information content is expressed as a string of digital bytes. The block of bytes is then processed into a string of bits 0s and 1s. The result is a new block represented as a binary string, which is again a unique representation of the initial block’s content. At this stage, the method allows for two possible implementation options, as shown diagrammatically in Figure 4:

• Option 1: To compress the entire block’s information into a single IE numerical value by selecting the $\mathrm{WS}=N$ and calculating the IE value of the block.
• Option 2: To compress the entire block’s information into an Entropic Barcode.

Both proposed options result in massive one-way data compression and simultaneous encryption. However, the encryption achieved in this way is irreversible, because there is no process of reconstructing the block’s information from the IE value or from the Entropic Barcode. In this sense, the IE value or the Entropic Barcode of the block are similar to a block’s Hash key, because any changes of the data inside the block, will result in changes of the IE value, or changes of its Entropic Barcode. Figure 5 shows diagrammatically the methodology proposed for the implementation of these two options to an entire blockchain database.

Repeating the procedure described in Figure 4 for the entire blockchain, i.e. for each block within the blockchain, Option 1 will produce an Entropic Barcode of IE values versus the block numbers, which is a unique 2D barcode of the entire blockchain. Deploying Option 2 will convert the entire blockchain into a chain of Entropic Barcodes, in which an Entropic Barcode represents each block and a chain on N blocks will result in a chain of N Entropic Barcodes. Both proposed options drastically reduce the size of a blockchain, while simultaneously encrypting its content. However, the new Entropic Barcode representation of the entire blockchain (Option 1) is particularly attractive because it could be used for fast peer-to-peer validation via the Entropic Differential Barcoding technique (see Section 7).

The details of mining are beyond the scope of this article and will not discussed here, but essentially each time a new block is created, a copy is sent to all nodes together with the information entropy value (Option 1) or the Entropic Barcode (Option 2) of the last block. The validation is performed on the Entropic Barcode of the blockchain Option 1), or the chain of Entropic Barcodes (Option 2), instead of the entire blockchain.

Once the majority of the nodes agree on validating the Entropic Blockchain, the newly created block is added to the blockchain’s master copy, while the last IE value of the newly created block is added to the master blockchain’s entropic barcode (Option 1), or the last entropic barcode of the newly created block is added to the chain of entropic barcodes (Option 2). The benefit is that the newly created Entropic Blockchain is small enough to be emailed or transferred rapidly from terminal to terminal and the validation process is extremely fast, saving time, energy and data storage requirements. Hence, the proposed Entropic Blockchain technology offers solutions to the scalability of blockchain data facilitating more effective download, transfer, validation and confidentiality process. However, some of the decentralized features of blockchain technology are lost, because the Entropic Blockchain requires a blockchain master copy being kept in a digital volt, while all operations are performed on its reduced Entropic Blockchain version.

7. Entropic Differential Barcode (EDB)

For any given digital asset, one can use the Entropic Barcode technology to generate an Entropic Barcode of the original asset, not only for the purpose of labeling, compressing and encrypting, but also for detecting any changes in it at later time. Detecting any changes in the digital assets is a very powerful application because it allows one to check the integrity of a digital asset without accessing any information within it, maintaining full privacy of the data. Essentially after the Entropic Barcode is generated, any changes to the original digital asset, as small as just one bit, will be easily detected regardless of its size. The method involves reconstructing the Entropic Barcode of the digital asset again and then comparing it to the original Entropic Barcode. The comparison between the two barcodes is performed via the Entropic Differential Barcode (EDB), which is obtained by subtracting the two barcodes from each other. In a previous study using a similar method and applied to genetic sequences, the Entropic Ratio has been used instead of the Entropic Differential Barcode to detect genetic mutations [10,11]. However, it is very possible that sections of the dataset might have IE = 0. This creates a problem when applying the ratio of the two spectra/barcodes as a number divided by zero is not computable. Hence, to avoid this problem, a better method based on the Entropic Differential Barcode (EDB) is introduced. If the asset is unchanged, the EDB will be 0 everywhere. If the asset suffered any changes, the EDB will show deviations from 0, at the location where the changes occurred. This method could therefore be used to check the integrity of any digital asset and to detect any possible changes to it, however small. The applications are multifold and include validation of digital data files, blockchains and crypto tokens. To demonstrate the method of Entropic Barcoding and Entropic Differential Barcoding of digital files we will show here an example.

Figure 6a shows an example of an Excel spreadsheet digital file for which we produced its unique Entropic Barcode representation (Figure 6b) of the original file. In this example, the Excel file has $N=655360$ bits. For this particular example we imposed 1000 points barcode size, i.e. the number of windows created equals 1000, $N_W=1000$, obtaining WS = 655. The SS has been taken equal to the WS, WS = SS = 655. The above example has been generated using an $m=3$ characters m-block with step size $\mathrm{ss}=m=3$. The maximum possible IE per window is therefore 3. The entire barcode contains data with maximum IE = 2.9765, minimum IE = 0 and average $〈\mathrm{IE}〉=1.5418$. For 1000 points Entropic Barcode, the resulted spectrum text file is always 15 Kilobytes (Kb), regardless of the type or size of the original file. Hence, this technique can achieve significant data compression from multiple Gigabytes (Gb) to 15Kb, while the Entropic Barcode is a true and unique representation of the original file, which does not disclose any of the file’s content.

After the barcode is generated, any changes or alterations to the original file, as small as just one bit, will be easily picked up. This method could be used to check digital fraud and the integrity of any digital file, including financial data files, banking records, image files, any type of document or software, blockchains, non-fungible tokens (NFTs), etc. Moreover, the method allows one to preserve the digital content of the file into this format while maintaining the full confidentiality of the file’s content, i.e. the Entropic Barcode does not reveal any information contained within the file. For example, one can have a digital copy of the Entropic Barcode of the spreadsheet Excel file analyzed here, and there is no way of extracting any numerical or financial data contained in the file from the barcode itself. However, after the original file has been Entropic Barcoded, any future changes to the file itself can be detected by barcoding the file again and using the Entropic Differential Barcode (EDB) method to check for the file’s integrity. To demonstrate this, we intentionally tampered the file by changing a single bit (0 into 1) and we run the EDB program to validate the file. Figure 7 shows the Entropic Barcodes of the original file, the 1-bit altered file and the EDB verification spectrum. The data clearly shows that a single bit alteration, which was a 0 changed into a 1, has been detected via the EDB validation check, showing a spectrum that contains a non-zero value at the location index 522. This means that the window index 522 contains a modification/alteration to the file. The original file’s IE value of the window index 522 was 1.831 bits. The change of a 0 into a 1 resulted in a change of the IE value from 1.831 bits to 1.847 bits. Hence the non-zero spike in the EDB spectrum that indicates the alteration has an absolute value of 0.016 bits. Any substantial changes to the file that result in a change of the total number of bits will show up as an EDB spectrum that is non-zero everywhere. This is a powerful technique that could be deployed to perform forensic verification of any form of digital file/asset including performing financial audits, and validating blockchains, crypto tokens or NFTs.

8. Conclusions

The Entropic Barcoding technology facilitates the creation of new data processing tools with unique properties. The proposed method makes use of the Shannon’s information theory to simultaneously compress, encrypt and barcode any dataset. Here we proposed using this technology to digital data files and blockchain databases, to achieve the next generation blockchain technology, called Entropic Blockchains. Using the Entropic Barcoding, the datasets are massively compressed in terms of their digital footprint, allowing savings in digital data storage and transmission of data. In the same time, while condensing a whole dataset into an Entropic Barcode, the barcode acts as a one-way encryption of the data, because there is no mechanism to reconstruct the set from its Entropic Barcode. When this method is combined with the Differential Entropic Barcoding technique also proposed here, the methodology allows fast detection of any change of the digital asset, facilitating an ultra-effective, fast and energy efficient blockchain validation method.

9. Patents

The Entropic Barcoding technology described in this study is the subject of the patent application GB2404348.1.