Applications of Entropy in Data Analysis and Machine Learning: A Review

1. Introduction

1.1. Aims and Scope

Entropy is a concept that appears in different areas of physics and mathematics with different meanings. Thus, entropy is a measure of: (i) disorder in Statistical Mechanics, (ii) uncertainty in Information and Probability Theories, (iii) (pseudo-)randomness in the Theory of Measure-preserving Dynamical Systems, and (iv) complexity in Topological Dynamics. This versatility explains why entropy has found extensive applications across various scientific disciplines since its inception in the 19th century.

Precisely, this paper aims to provide an up-to-date overview of the applications of entropy in data analysis and machine learning, where entropy stands here not only for the traditional instances but also for more recent proposals inspired by them. In data analysis, entropy is a powerful tool for detection of dynamical changes, segmentation, clustering, discrimination, etc. In machine learning, it is used for classification, feature extraction, optimization of algorithms, anomaly detection, and more. The ability of entropy to provide insights into data structure and algorithm performance has led to a widespread search for further applications and new proposals tailored to specific needs, both in data analysis and machine learning.

This being the case, the present review will be useful for researchers in the above two fields, interested in the theoretical basics and/or the current applications of entropy. Along with established applications, the authors also have taken into account innovative proposals to reflect the intense research activity on entropy that is currently underway.

At this point, the reader may be wondering what entropy is. A search for the word “entropy” on the Internet returns a large number of results, some of them also called entropy metrics, entropy-like measures or entropy-based indices in the literature. So, what is entropy actually?

1.2. Classical Entropies and Generalized Entropies

Historically, the word “entropy” was introduced by the German physicist Clausius in Thermodynamics in 1865 to designate the amount of internal energy in a system that cannot be transformed into work. In particular, entropy determines the equilibrium of a thermodynamical system, namely, the state of maximum entropy consistent with the macroscopic constraints. In the second half of the 19th century, entropy was given a microscopic interpretation in the foundational works of Boltzmann and Gibbs on Statistical Mechanics. In 1927, von Neumann generalized the Boltzmann-Gibbs entropy to the then-emerging theory of Quantum Mechanics [1]. In 1948, the word entropy appeared in a completely different context: Information Theory. If entropy is a measure of disorder in Statistical Mechanics; in the seminal paper of Shannon [2], the creator of Information Theory, entropy stands for the average uncertainty about the outcome of a random variable (or the information conveyed by knowing it). Albeit in different realms, the coincidence in names is explained because Shannon’s formula (see equation (1) below) is formally the same as Gibbs’ for the entropy of a system in thermal equilibrium with a heat bath at constant temperature [3].

This abridged history of entropy continues with Kolmogorov, who crafted Shannon’s entropy into a useful invariant in Ergodic Theory [4], and Sinai, who adapted Kolmogorov’s proposal to the theory of measure-preserving dynamical systems [5]. In turn, Adler, Konheim and McAndrew [6] generalized the Kolmogorov-Sinai (KS) entropy from measure-preserving dynamics to topological dynamics under the name of topological entropy. According to the Variational Principle, topological entropy is a tight upper bound of the KS entropies defined over certain probability measures [7].

To get down to the mathematical formulas, let $\mathcal{P}$ be the set of probability mass distributions $\{p_1,\dots ,p_W\}$ for all $W\ge 2$. Then, the Shannon entropy of the probability distribution ${\mathrm{p}}_W=\{p_1,\dots ,p_W\}$ is defined as

$$S\left({\mathrm{p}}_W\right)=S(p_1,\dots ,p_W)=-\sum _{i=1}^W{p}_{i}log{p}_i$$

(1)

where the choice of the logarithm base fixes the unit of the entropy. The usual choices being 2 (bit), e (nat) or 10 (dit). If $p_i=0$, then $0log0:={lim}_{x\to 0+}xlogx=0$. Mathematically, equation (1) is the expected value of the information function$I\left(X\right)=-logp\left(X\right)$, where X is a random variable with probability distribution ${\mathrm{p}}_W$. Since entropy is the cornerstone of Information Theory, Shannon also justified definition (1) by proving in his seminal paper [2] that it is unique (except for a positive factor) under a few, general assumptions. In their modern (equivalent) formulation, these assumptions are called the Shannon-Khinchin axioms [8], that we state below.

A positive functional H on $\mathcal{P}$, i.e., a map $H:\mathcal{P}\to {\mathbf{R}}^{+}$ (${\mathbf{R}}^{+}$ being the non-negative real numbers), is an entropy if it satisfies the following properties:

SK1 

Continuity. $H(p_1,\dots ,p_W)$ depends continuously on all variables for each W.

SK2 

Maximality. For all W,

$$H(p_1,\dots ,p_W)\le H(\frac{1}{W},\dots ,\frac{1}{W}).$$

SK3 

Expansibility. For all W and $1\le i\le W$,

$$H(0,p_1,\dots ,p_W)=H(p_1,\dots ,p_i,0,p_{i+1},\dots ,p_W)=H(p_1,\dots ,p_i,p_{i+1},\dots ,p_W).$$

SK4 

Strong additivity (or separability). For all $W,U$,

$$\begin{array}{cc}\hfill & H(p_{11},\dots ,p_{1U},p_{21},\dots p_{2U},\dots ,p_{W1},\dots ,p_{WU})\hfill \\ \hfill & =H(p_{1\ast },p_{2\ast },\dots ,p_{W\ast })+\sum _{i=1}^W{p}_{i\ast }H\left({\displaystyle \frac{p_{i1}}{p_{i\ast }}},{\displaystyle \frac{p_{i2}}{p_{i\ast }}},\dots ,{\displaystyle \frac{p_{iU}}{p_{i\ast }}}\right),\hfill \end{array}$$

(2)

where $p_{i\ast }={\sum }_{j=1}^U{p}_{ij}$.

Axiom SK4 can be formulated in a more compact way as

$$H(X,Y)=H\left(X\right)+H(Y\left|X\right),$$

where X and Y are random variables with probability distributions $\{p_{i\ast }:1\le i\le W\}$ and $\{p_{\ast j}={\sum }_{i=1}^W{p}_{ij}:1\le j\le U\}$ respectively, $H(X,Y)=H(p_{11},\dots ,p_{1U},\dots ,p_{W1},\dots ,p_{WU})$ and $H(Y\left|X\right)$ is the entropy of Y conditional on X, i.e., the expected value of the conditional distributions $p(y\left|x\right)$, averaged over the conditioning variable X[9]. In particular, if X and Y are independent (i.e., $p_{ij}=p_{i\ast }{p}_{\ast j}$), then $H(Y\left|X\right)=H\left(Y\right)$ and

$$H(X,Y)=H\left(X\right)+H\left(Y\right).$$

(3)

If H satisfies equation (3) for independent random variables X and Y, then it is called additive.

It was proved in [2] and [8] that a positive functional H on $\mathcal{P}$ that fulfills Axioms SK1-SK4 is necessarily of the form

$$H(p_1,\dots ,p_W)=-k\sum _{i=1}^W{p}_{i}log{p}_i=:S_{BGS}(p_1,\dots ,p_W)$$

(4)

for every $W\ge 2$, where k is a positive constant. For historical reasons, $S_{BGS}$ is usually called the Boltzmann-Gibbs-Shannon entropy. In Physics, k is the Boltzmann constant $k_B=1.3806504\left(24\right)\times {10}^{-23}$ J/K and log is the natural logarithm. In Information Theory, $k=1$ and log is the base 2 logarithm when dealing with digital communications. The particular case

$$S_{BGS}(1/W,\dots ,1/W)=klogW,$$

(5)

obtained for uniform distributions is sometimes referred to as the Boltzmann entropy, although the expression (5) is actually due to Planck [10]. According to Axiom SK2, the Boltzmann entropy is the maximum of $S_{BGS}$.

The same conclusion about the uniqueness of $S_{BGS}$ can be derived using other equivalent properties [11]. Since we are not interested in physical applications here, we set $k=1$ and generally refer to $S_{BGS}$ as Shannon’s entropy.

In 1961 Rényi proposed a generalization of Shannon’s entropy by using a different, more general definition of expectation value [12,13]: For any real $\alpha >0$, $\alpha \ne 1$, Rényi entropy$R_{\alpha }$ is defined as

$$R_{\alpha }(p_1,\dots ,p_W)={\displaystyle \frac{1}{1-\alpha }}log\left(\sum _{i=1}^W{p}_i^{\alpha }\right).$$

(6)

So, Rényi entropy is actually a family of entropies; in particular, $R_1:={lim}_{\alpha \to 1}{R}_{\alpha }=S_{BGS}$. Other limiting cases are $R_0:={lim}_{\alpha \to 0}{R}_{\alpha }=logW$, called Hartley or max-entropy, which coincides with the Boltzmann entropy (5) except for the value of the constant k, and $R_{\infty }:={lim}_{\alpha \to \infty }{R}_{\alpha }={min}_{1\le i\le W}(-ln{p}_i)$, called the min-entropy. These names are due to the decreasing monotonicity of Rényi’s entropy with respect to the parameter: $R_{\alpha }>R_{\beta }$ for $\alpha <\beta$.

It is easy to show that Rényi’s entropy satisfies Axioms SK1-SK3 but not SK4. Instead of strong additivity, $R_{\alpha }$ satisfies additivity:

$$R_{\alpha }({\mathrm{p}}_U\times {\mathrm{q}}_W)=R_{\alpha }\left({\mathrm{p}}_U\right)+R_{\alpha }\left({\mathrm{q}}_W\right),$$

see equation (3).

A final milestone in this short history of entropy is the introduction of non-additive entropies by Havrda and Charvát in Information Theory [14] and Tsallis in Statistical Mechanics [15], which are equivalent and usually called the Tsallis entropy:

$$T_q(p_1,\dots ,p_W)={\displaystyle \frac{1}{1-q}}\left(\sum _{i=1}^W{p}_i^q-1\right)$$

(7)

for any real $q>0$, $q\ne 1$. Again, Tsallis entropy is a family of entropies that satisfies Axioms SK1-SK3 but not SK4. Instead, $T_q$ is “q-additive”, meaning that

$$T_q({\mathrm{p}}_U\times {\mathrm{q}}_W)=T_q\left({\mathrm{p}}_U\right)+T_q\left({\mathrm{q}}_W\right)+(1-q)T_q\left({\mathrm{p}}_U\right)T_q\left({\mathrm{q}}_W\right).$$

As Rényi’s entropy, the Tsallis entropy is a generalization of Shannon’s entropy in the sense that $T_1:={lim}_{q\to 1}{T}_q=S_{BGS}$. Formally, $T_q$ can be obtained from $S_{BGS}$ by replacing the logarithm in equation (4) by the “q-logarithm” [13].

The appearance of generalizations of the Shannon entropy prompted the weaker concept of generalized entropy: a positive functional on probability distributions that satisfies Axioms SK1-SK3. Therefore, the BGS entropy, together with the Rényi and Tsallis entropies are examples of generalized entropies. Shannon’s uniqueness theorem can then be rephrased by saying that, the only generalized entropy that is strongly additive is the BGS entropy. Axioms SK1-SK3 are arguably the minimal requirements for a positive functional on probability mass distributions to be called an entropy. Most “entropies” proposed since the formulation of Rényi and Tsallis’ entropies are precisely generalized entropies in the axiomatic sense.

To wrap up this short account of the classical and generalized entropies, let us mention that Shannon’s, Rényi’s and Tsallis’ entropies (and other entropies for that matter) have counterparts for continuous-valued random variables and processes (i.e., defined on probability densities). These “differential” versions are formally obtained by replacing probability mass functions by probability densities and summations by integrations in equations (1), (6) and (7), respectively. Although also useful in applications, differential entropies may lack important properties of their discrete counterparts. For example, differential (Shannon’s) entropy lacks positivity [9].

1.3. Methodology and Organization of this Review

As said above, the primary objective of this work is to review the applications of entropy in the fields of data analysis and machine learning. In view of the many versions of entropy currently in use we had to make a selection of them based on their relevance in general, and the interest of their applications, in particular. Apart from the group of classical entropies mentioned in Section 1.2), the remaining entropies selected for this review can be classified into the following three groups.

G1 

Entropies based on Shannon’s entropy. These are entropies that use Shannon’s function (1) on probability distributions obtained from the data by a number of techniques. This group comprises: Dispersion entropy and fluctuation-based dispersion entropy, energy entropy and empirical mode decomposition energy entropy, Fourier entropy and fractional Fourier entropy, graph entropy, permutation entropy, spectral entropy, and wavelet entropy.

G2 

Entropies based on other information-theoretical concepts, such as the correlation integral, divergences, unconditional or conditional mutual information, etc. This group comprises: Approximate entropy, cross entropy and categorical cross entropy, excess entropy, kernel entropy, relative entropy or Kullback-Leibler divergence, and transfer entropy.

G3 

Entropies tailored to specific needs or inspired by other entropies. This group comprises: Bubble entropy, entanglement entropy, fuzzy entropy, intrinsic mode entropy, Kaniadakis entropy, Rao’s quadratic entropy, rank-based entropy, sample entropy, and tone entropy.

In Section 2, each of the 33 selected entropies is assigned a subsection in alphabetical order. For brevity, mathematical definitions are reminded only when convenient; otherwise, we give a qualitative account and refer the reader to the original publications or standard bibliography for the formulas. The corresponding applications to data analysis and machine learning are explained with brief but sufficient description and provided with specific references. Practical issues such as the choice of parameters or computational implementations are beyond the scope of this review. For a general bibliography on entropy, we refer the reader to the excellent review “The Entropy Universe” by M. Ribeiro et al. [16], as well as the reviews [13,17,18].

2. Applications in Data Analysis and Machine Learning

In this section, the selected entropies are sorted alphabetically. To streamline the exposition, multiscale and weighted versions [19] are included in the same section as the original entropy. The corresponding applications to data analysis and machine learning are tagged with some keywords in alphabetical order so that reverse search (i.e., searching for entropies for a given application) can also be easily performed.

2.1. Approximate Entropy

Approximate entropy was proposed by Pincus in 1991 [20] to analyze medical data. Loosely speaking, approximate entropy is a heuristic implementation of the correlation integral with time series data [18]. The approximate entropy depends on a parameter $r>0$, sometimes called tolerance, which is a cut-off that defines the concept of proximity between points via the Heaviside step function $\mathrm{\Theta }(r-∥x_i-x_j∥)$ (where $\mathrm{\Theta }\left(z\right)=1$ if $z\ge 0$, and 0 otherwise). It quantifies the change in the relative frequencies of length-k time-delay vectors with increasing k. A modified version of approximate entropy was proposed in 2000 under the name sample entropy (Section 2.25). See [21] for a tutorial.

• Applications
  • Alzheimer’s disease. Approximate entropy has been used in the non-linear analysis of EEGs and MEGs from patients with Alzheimer’s disease [22,23].
  • Anesthetic drug effects. Another field of applications is the quantification of anesthetic drug effects on the brain activity as measured by EEGs, including comparative testing of different anesthetics [24].
  • Emotion recognition. Along with other entropies, approximate entropy has been used for EEG-based human emotion recognition [25].
  • Epileptic seizure detection. Approximate entropy has also been used as biomarker in algorithms for epileptic EEG analysis, in particular, for epileptic seizure detection [26,27,28].
  • Physiological time series. See [29] for an overview of applications of approximate entropy to the analysis of physiological time series.
  • Sleep research. The applications of approximate entropy include sleep research, in particular, the separation of sleep stages based on EEG data [30].

2.2. Bubble Entropy

Bubble entropy is a metric that evaluates changes in the order of data segments in time series when a new element is added. It was proposed by Manis et al. in 2017 [31] as “an entropy almost free of parameters", inspired by permutation entropy (Section 2.20) and rank-based entropy (Section 2.22). Bubble entropy relies on the Bubble Sort algorithm [31], which compares and swaps adjacent items until ordered.

• Applications
  • Biomedical applications. Due to its minimal dependency on parameters, bubble entropy is particularly useful in biomedical applications (e.g., analysis of heart rate variability) to distinguish healthy from pathological conditions [31,32].
  • Fault bearing detection. Bubble entropy is used to reinforce the accuracy of fault bearing diagnosis through the Gorilla Troops Optimization (GTO) algorithm for classification [33]. A similar application can be found for the so-called Improved Hierarchical Refined Composite Multiscale Multichannel Bubble Entropy [34].
  • Feature extraction. Bubble entropy is compared with dispersion entropy (Section 2.6) in the extraction of single and double features in [35].

2.3. Categorical cross entropy

Categorical cross entropy (CCE) is a variant of the cross entropy (Section 2.4) that uses categorical labels instead of real-valued entries. For clarity, we consider CCE in this section and the conventional cross entropy in the next.

• Applications
  • Deep Learning. CCE is used in deep neural networks when dealing with noisy labels. An improved categorical cross entropy (ICCE) is also used in this case [36].
  • Multi-class classification. CCE is used in the analysis of multi-channel time-series multi-class classification. It is the standard loss function for tasks such as image classification, text classification, and speech recognition [37].
  • Reinforcement learning. CCE is used as an improvement of value function training (using classification instead of regression) mainly in games [38].
  • Semi-supervised learning. CCE is used in pseudo-labelling to optimize convolutional neural networks parameters [39].

2.4. Cross Entropy

The cross entropy between the probability distributions $\mathrm{p}=(p_1,\dots ,p_W)$ and $\mathrm{q}=(q_1,\dots ,q_W)$ is defined as

$$C(\mathrm{p},\mathrm{q})=-\sum _{i=1}^W{p}_{i}log{q}_i.$$

(8)

It is related to the Shannon entropy $S\left(\mathrm{p}\right)$ (equation 1) and the Kullback-Leibler divergence $D\left(\mathrm{p}\right.∥\mathrm{q})$ (Section 2.23, equation (15)) through the equation $C(\mathrm{p},\mathrm{q})=S\left(\mathrm{p}\right)+D\left(\mathrm{p}\right.∥\mathrm{q})$. As $D\left(\mathrm{p}\right.∥\mathrm{q})$, the cross entropy is used to quantify the difference between two probability distributions: the true distribution $\mathrm{q}$ (actual labels) and the predicted distribution $\mathrm{p}$ (model predictions). A typical example is determining whether an email is spam or not.

• Applications
  • Deep learning. Cross entropy is a standard loss function for training deep neural networks, particularly those involving softmax activation functions. It is very useful for applications such as object detection, language translation, and sentiment analysis [40]. In this regard, empirical evidence with limited and noisy data suggests that to measure the top-$\kappa$ error (a common measure of performance in machine learning performed with deep neural networks trained with the cross entropy loss), the loss function must be smooth, meaning that it should incorporate a smoothing parameter $ϵ$ to handle small probability events [41].
  • Feature selection. Cross entropy is used to select significant features of binary values from highly imbalanced large datasets via a framework called FMC Selector [42].
  • Image analysis. Wavelet analysis together with cross entropy are used in image segmentation, object recognition, texture analysis (e.g., fabric defect detection) and pattern classification [43].
  • Learning-to-rank methods. In [44] the author proposes a learning-to-rank loss function that is based on cross entropy. Learning-to-rank methods form a class of ranking algorithms that are widely applied in information retrieval.
  • Multiclass classification. Cross entropy is used to enhance the efficiency of solving support vector machines for multi-class classification problems [45].
  • Semi-supervised clustering. Cross entropy is employed along with the information bottleneck method in semi-supervised clustering. It is robust to noisy labels and automatically determines the optimal number of clusters under mild conditions [46].

2.5. Differential Entropy

As said in Section 1.2, differential entropy is the continuous counterpart of Shannon entropy: If X is a continuous random variable with density $\rho \left(x\right)$ and support set S, then the differential entropy of X is [9]

$$h\left(X\right)=h\left(\rho \right)=-{\int }_S\rho \left(x\right)log\rho \left(x\right)dx.$$

(9)

Therefore, differential entropy is used in the analysis of continuous random variables and processes, e.g., analog signals.

• Applications
  • Anomaly detection. Differential entropy can measure changes in the probability density function of an analog signal which reveals an anomaly in the source, whether it is a mechanical system or a patient [47].
  • Emotion recognition. Differential entropy has been used in [25] to extract features in EEG-based human emotion recognition.
  • Feature selection. A feature selection algorithm based on differential entropy to evaluate feature subsets has been proposed in [48]. This algorithm effectively represents uncertainty in the boundary region of a fuzzy rough model and demonstrates improved performance in selecting optimal feature subsets, thereby enhancing classification accuracy; see [49] for an implementation.
  • Generative models. Variational Autoencoders and other generative models leverage differential entropy to model the latent space of continuous data distributions. These models can learn better representations of the input data, thus improving the performance [50].
  • Mutual information. Differential entropy is instrumental to compute the mutual information of continuous-valued random variables and processes, e.g., autoregressive processes. Seen in speech processing (linear prediction), seismic signal processing and biological signal processing [51].
  • Probabilistic models. Differential entropy is utilized in probabilistic models such as Gaussian Mixture Models to describe the uncertainty and distribution of continuous variables. This approach is applicable to image processing and network inference as well [52].

2.6. Dispersion Entropy

Dispersion entropy characterizes time series data based on the Shannon Entropy. Introduced by Rostaghi and Azami in 2016 [53] to address limitations in other entropy measures, dispersion entropy includes a transformation that translates data into discrete symbols [16].

• Applications
  • Feature extraction. Multiscale fuzzy dispersion entropy is applied in fault diagnosis of rotating machinery to capture the dynamical variability of time series across various scales of complexity [54].
  • Image classification. A multiscale version of dispersion entropy called MDispEn_2D has been used with biomedical data to measure the impact of key parameters that may greatly influence the entropy values obtained in image classification [55].
  • Signal classification. Other generalization of dispersion entropy, namely, fractional fuzzy dispersion entropy, has been proposed as a fuzzy membership function for signal classification tasks [56].
  • Signal denoising. Dispersion entropy is also used in signal denoising via (i) adaptive techniques and group-sparse total variation [57], or (ii) empirical mode decomposition with adaptive noise [58].
  • Time series analysis. Multiscale graph-based dispersion entropy is a generalization of dispersion entropy used to analyze multivariate time series data in graph and complex network frameworks, e.g., weather and two-phase flow data; it combines temporal dynamics with topological relationships [59].

2.7. Energy Entropy and Empirical Mode Decomposition Energy Entropy

Energy entropy is the Boltzmann-Gibbs-Shannon entropy of a normalized distribution of energy levels or modes. So it is mainly used in data-driven analysis of physical systems, whether in natural sciences or technology.

In nonlinear time series analysis, one uses empirical mode decomposition (EMD) to decompose a time series into a set of intrinsic mode functions, each representing a simple oscillatory mode inherent to the data. This decomposition is adaptive and data-driven, making it suitable for analysing complex signals without requiring predefined basis functions [60]. As its name suggests, EMD energy entropy combines energy entropy with EMD to provide information about the energy distribution across different intrinsic mode functions derived from a signal.

• Applications
  • Chatter detection. This application involves detecting vibrations and noise in machining operations that can indicate chattering. In this regard, energy entropy can detect chatter in robotic milling [61].
  • Fault prediction in vibration signal analysis. EMD energy entropy has been employed to predict early fault of bearings in rotating machinery [62,63].
  • Feature extraction. Energy entropy is calculated via the empirical decomposition of the signal into intrinsic mode functions and serves as a feature for machine learning models used in chatter detection. The chatter feature extraction method is based on the largest energy entropy [64].
  • Time-series forecasting. EMD energy entropy was used in [65] to predict short-term electricity consumption by taking into account the data variability, i.e., that power consumption data is non-stationary, nonlinear, and influenced by the season, holidays, and other factors. In [66], this entropy was the tool to distinguish two kinds of financial markets.

2.8. Entanglement Entropy

Entanglement entropy originated in Quantum Mechanics as a measure of the degree of quantum entanglement between two subsystems of a quantum system. To be more precise, entanglement entropy is the von Neumann entropy (Section 2.32) of the reduced density matrix for any of the subsystems [67]; see [68] for the estination of entanglement entropy through supervised learning. In addition to its important role in quantum mechanics and quantum information theory, entanglement entropy is increasingly finding more applications in machine learning as well. This is why we have included entanglement entropy in this review.

• Applications
  • Feature extraction. In quantum machine learning, entanglement entropy is used for feature extraction by representing data in a form that highlights quantum correlations and thus, leveraging the quantum properties of the data [69].
  • Quantum models. Entanglement entropy is used in quantum models to quantify unknown entanglement by using neural networks to predict entanglement measures of unknown quantum states based on experimentally measurable data: moments or correlation data produced by local measurements [70].

2.9. Excess Entropy

The mutual information between the random variables X and Y measures the average reduction in uncertainly about one of variables that results from learning the value of the other. It is defined as [9]

$$I(X;Y)=I(Y;X)=S\left(X\right)+S\left(Y\right)-S(X,Y)\ge 0,$$

(10)

where $S(\dots )$ is Shannon’s entropy (1).

Excess entropy (also called dual total correlation) is a non-negative generalization of mutual information to more than two random variables defined as [71]

$$D(X_1,\dots ,X_N)=S(X_1,\dots ,X_N)-\sum _{i=1}^{N}S\left(X_i\right|X_1,\dots ,X_{i-1},X_{i+1},\dots ,X_N)$$

(11)

where $S\left(X_i\right|\dots )$ is the Shannon conditional entropy of $X_i$ given the other variables. Of course, $D(X_1,X_2)=I(X_1;X_2)$. An alternative definition called total excess entropy was introduced by Crutchfield and Packard in [72].

• Applications
  • Image segmentation. Excess entropy is used to measure the structural information of a 2D or 3D image and then determine the optimal threshold in a segmentation algorithm proposed in [73]. The working hypothesis of this thresholding-based segmentation algorithm is that the optimal threshold corresponds to the maximum excess entropy (i.e., to a segmentation with maximum structure).
  • Machine learning. In [74], the authors present a method called machine-learning iterative calculation of entropy, for calculating the entropy of physical systems by iteratively dividing the system into smaller subsystems and estimating the mutual information between each pair of halves.
  • Neural estimation in adversarial generative models. Mutual Information Neural Estimator is a scalable estimator used in high dimensional continuous data analysis that optimizes mutual information. The authors apply this estimator to Generative Adversarial Networks (GANs) [75].
  • Time series analysis. In this application, total excess entropy is used for classifying stationary time series into long-term and short-term memory. A stationary sequence with finite block entropy is long-term memory if its excess entropy is infinite [76].

2.10. Fluctuation-Based Dispersion Entropy

Fluctuation-based dispersion entropy (FDispEn) was introduced by Azami and Escudero in 2018 [77] to quantify the complexity and dynamical variability of time-series by incorporating information about fluctuations into the original dispersion entropy framework. This enhancement allows for a more detailed analysis of the signal’s inherent dynamics.

• Applications
  • Fault diagnosis. The so-called refined composite moving average FDispEn is used in machinery fault diagnosis by analysing vibration signals [78].
    Refined composite multiscale FDispEn and supervised manifold mapping are used in fault diagnosis for feature extraction in planetary gearboxes [79].
    Multivariate hierarchical multiscale FDispEn along with multi-cluster feature selection and Gray-Wolf Optimization-based Kernel Extreme Learning Machine helps diagnose faults in rotating machinery. It also captures the high-dimensional fault features hidden in multichannel vibration signals [80].
  • Feature extraction. FDispEn gives rise to hierarchical refined multi-scale fluctuation-based dispersion entropy, used to extract underwater target features in marine environments and weak target echo signals, thereby improving the detection performance of active sonars [81].
  • Robustness in spectrum sensing. Reference [82] proposes a machine learning implementation of spectrum sensing using an improved version of the FDispEnt as a feature vector. This improved version shows enhanced robustness to noise.
  • Signal classification. FDispEn helps distinguish various physiological states of biomedical time series and it is commonly used in biomedicine. It is also used to estimate the dynamical variability of the fluctuations of signals applied to neurological diseases [83]. Fluctuation-based reverse dispersion entropy is applied to signal classification combined with k-nearest neighbor [84].
  • Time series analysis. FDispEn is used to quantify the uncertainty of time series to account for knowledge on parameters sensitivity and studying the effects of linear and nonlinear mapping on the defined entropy in [77]. FDispEn is defined as a measure for dealing with fluctuations in time series. Then, the performance is compared to complexity measures such as permutation entropy (Section 2.20), sample entropy (Section 2.25), and Lempel-Ziv complexity [9,85].

2.11. Fourier Entropy

The Fourier entropy $h\left(f\right)$ of a Boolean function $f:{\{-1,1\}}^n\to \{-1,1\}$ is the Shannon entropy of its power spectrum $\{\widehat{f}{\left(S\right)}^2:S\subset \left[n\right]\}$, where $S\subset \left[n\right]$ stands for the $2^n$ subsets of $\{1,2,\dots n\}$ (including the empty set) and $\widehat{f}$ is the Fourier transform of f. By Parseval’s Theorem, ${\sum }_{S\subset \left[n\right]}\widehat{f}{\left(S\right)}^2=1$, so the power spectrum of f is a probability distribution.

• Applications
  • Decision trees: The Fourier Entropy-Influence Conjecture, made by Friedgut and Kalai [86], says that the Fourier entropy of any Boolean function f is upper bounded, up to a constant factor, by the total influence (or average sensitivity) of f. This conjecture, applied to decision trees, gives interesting results that boil down to $h\left(f\right)=O\left(\log \right(L\left(f\right)\left)\right)$, meaning that $\left|h\left(f\right)\right|<Clog\left(L\left(f\right)\right)$, where $L\left(f\right)$ denotes the minimum number of leaves in a decision tree that computes f and $C>0$ is independent of f and $L\left(f\right)$ [87]. Another similar application to decision trees can be found in [88].
  • Learning theory. The Fourier Entropy-Influence Conjecture is closely related to the problem of learning functions in the membership model. It is said that if a function has small Fourier entropy it means that its Fourier transform is concentrated on a few characters, i.e. the function can be approximated by a sparse polynomial, a class that is very important in the context of learning theory [89]. Learning theory provides the mathematical foundation for understanding how algorithms learn from data, guiding the development of machine learning models.

2.12. Fractional Fourier Entropy

Fractional Fourier entropy contains two ingredients: the fractional Fourier transforms of the data [90] and the Shannon entropy of probability distributions related to the resulting frequency spectrum, e.g., the power spectrum.

• Applications
  • Anomaly detection in remote sensing. Fractional Fourier entropy is used in hyperspectral remote sensing to distinguish signals from background and noise [91,92].
  • Artificial intelligence. Two-dimensional fractional Fourier entropy helps to diagnose COVID-19 by extracting features from chest CT images [93].
  • Biomedical image classification. Fractional Fourier entropy has proven helpful in detecting pathological brain conditions. By using it as a new feature in Magnetic Resonance Imaging, the classification of images is improved in time and cost [94].
  • Deep learning. Fractional Fourier entropy is used in the detection of gingivitis via feed-forward neural networks. It reduces the complexity of image extraction before classification and can obtain better image eigenvalues [95].
  • Emotion recognition. Fractional Fourier entropy, along with two binary support vector machines, helps improve the accuracy of emotion recognition from physiological signals in electrocardiogram and galvanic skin responses [96].
  • Multilabel classification. Fractional Fourier entropy has been used in a tea-category identification system, which can automatically determine tea category from images captured by a 3 charge-coupled device digital camera [97].

2.13. Fuzzy Entropy

Fuzzy entropy, introduced by Ishikawa and Mieno in 1979 [98], quantifies the uncertainty or fuzziness within a system, thus extending entropy concepts to fuzzy sets to better represent uncertainty. In 2014, Zheng proposed multiscale fuzzy entropy [99]. Fuzzy entropy was introduced into fuzzy dynamical systems in [100].

• Applications
  • Clustering and time-series analysis. Fuzzy entropy is used in problems of robustness against outliers in clustering techniques in [101].
  • Data analysis. Fuzzy entropy is proposed in [102] to assess the strength of fuzzy rules with respect to a dataset, based on the greatest energy and smallest entropy of a fuzzy relation.
  • Fault detection. Fuzzy entropy (along with dispersion entropy, Section 2.6) was the best performer in a comparative study of entropy-based methods for detecting motor faults [103]. Multiscale fuzzy entropy is used to measure complexity in time series in rolling bearing fault diagnosis [104].
  • Feature selection and mathematical modelling. Fuzzy entropy is used in feature selection to evaluate the relevance and contribution of each feature in Picture Fuzzy Sets [105].
  • Image classification. Fuzzy entropy, in the form of multivariate multiscale fuzzy entropy, is proposed and tested in [106] for the study of texture in color images and their classification.
  • Image segmentation. Fuzzy entropy is the objective function of a colour image segmentation technique based on an improved cuckoo search algorithm [107].

2.14. Graph Entropy

Graph entropy was introduced by Korner in 1971 [108] to quantify the complexity or information content of a graph. It is usually defined as the Shannon entropy (although any other entropy would do) of a probability distribution over the graph’s vertex set. In addition to applications in data analysis and machine learning, graph entropy is also applied in combinatorics; see [109] for a survey of graph entropy.

A particular case of graph entropy is the horizontal visibility (HV) graph entropy of a time series, which is the graph entropy of the so-called HV graph of the time series [110]. In particular, this method is useful for distinguishing between different types of dynamical behaviours in nonlinear time series, such as chaotic versus regular dynamics [111].

• Applications
  • Dimension reduction and feature selection. Graph entropy gives rise to the Conditional Graph Entropy that helps in the alternating minimization problem [112].
  • Graph structure. Graph entropy is used to measure the information content of graphs, as well as to evaluate the complexity of the hierarchical structure of a graph [113].
  • Graph-based time series analysis. Graph entropy can be used in time series analysis in conjunction with any method that transforms time series into graphs. An example is the HV graph entropy presented above; see [114] and references therein.
  • Node embedding dimension selection. Graph entropy is applied in Graph Neural Networks through the Minimum Graph Entropy algorithm. It calculates the ideal node embedding dimension of any graph [115].
  • Time series analysis. HV graph entropy along with sample entropy has been used in [116] to identify abnormalities in the EEGs of alcoholic subjects. HV transfer entropy was proposed in [117] to estimate the direction of the information flow between pairs of coupled time series.

2.15. Havrda–Charvát Entropy

Havrda–Charvát (HC) entropy, also known as the Havrda–Charvát α-entropy, was introduced by Havrda and Charvát [14] in 1967 in Information Theory. It is formally identical to the Tsallis entropy (Section 2.31, introduced by Tsallis in 1988 in Statistical Mechanics [15]. Similar to Rényi entropy (Section 2.24), HC entropy is a family of entropies parameterized by $\alpha >0$ and generalizes the Shannon entropy in the sense that the HC entropy coincides with the Shannon entropy in the limit $\alpha \to 1$.

Although nowadays the most popular name for this entropy is Tsallis entropy, we present in this section applications published in articles that refer to this entropy as the Havrda–Charvát entropy.

• Applications
  • Computer vision. An HC entropy-based technique for group-wise registration of point sets with unknown correspondence is used in graphics, medical imaging and pattern recognition. By defining the HC entropy for cumulative distribution functions (CDFs), the corresponding CDF-HC divergence quantifies the dissimilarity between CDFs estimated from each point-set in the given population of point sets [118].
  • Financial time series. Weighted HC entropy outperforms regular HC entropy when used as a complexity measure in financial time series. The weights turn out to be useful for showing amplitude differences between series with the same order mode (i.e. similarities in patterns or specific states) and robust to noise [119].
  • Image segmentation and classification. HC entropy is applied as loss function in image segmentation and classification tasks using convolutional neural networks in [120].
  • Loss functions in deep learning. HC entropy can be used to design loss functions in deep learning models. These loss functions are particularly useful in scenarios with small datasets, common in medical applications [121].

2.16. Intrinsic Mode Entropy

Intrinsic mode entropy (IME) computes the sample entropy (Section 2.25) over different scales of intrinsic mode functions extracted by the empirical mode decomposition method (Section 2.7). It was introduced by Amoud et al. in 2007 [122].

• Applications
  • Language gesture recognition. IME is used in [123] to analyse data from a 3-dimensional accelerometer and a five-channel surface electromyogram of the user’s dominant forearm for automated recognition of Greek sign language gestures.
  • Neural data analysis. An IME version with improved discriminatory capacity in the analysis of neural data is proposed in [124].
  • Time series analysis. IME is used in nonlinear time series analysis to efficiently characterize the underlying dynamics [122]. As any multiscale entropy, IME is particularly useful for the analysis of physiological time series [125]. See also [126] for an application to the analysis of postural steadiness.

2.17. Kaniadakis Entropy

Kaniadakis entropy, also known as $\kappa$-entropy due to its dependence on a parameter $0<\kappa <1$, was introduced by the physicist G. Kaniadakis in 2002 [127] to address the limitations of classical entropy in systems exhibiting relativistic effects. It is defined as

$$S_{\kappa }(p_1,\dots ,p_W)=\sum _{i=1}^W{\displaystyle \frac{p_i^{1-\kappa }-p_i^{1+\kappa }}{2\kappa }}.$$

(12)

Kaniadakis entropy is a relativistic generalization of the BGS entropy (4) in the sense that the latter is recovered in the $\kappa \to 0$ limit [128]. Like other physical entropies such as Tsallis’ and von Neumann’s, Kaniadakis entropy has also found interesting applications in applied mathematics.

• Applications
  • Image segmentation. Kaniadakis entropy is used in image thresholding to segment images with long-tailed distribution histograms; the parameter $\kappa$ is selected via a swarm optimization search algorithm [129].
  • Images threshold selection. Kaniadakis entropy can be used to construct an objective function for image thresholding. By using the energy curve and the Black Widow optimization algorithm with Gaussian mutation, this approach can be performed on both grayscale and colour images of different modalities and dimensions [130].
  • Seismic imaging. Application of the Maximum Entropy Principle with $S_{\kappa }$ leads to the Kaniadakis distribution, a deformation of the Gaussian distribution that has application, e.g., in seismic imaging [131].

2.18. Kernel Entropy

Kernel (or kernel-based) entropy is an evolution of the approximate entropy (Section 2.1) that consists of replacing the Heaviside step function in the definition of proximity by other functions (or “kernels”) to give more weight to the nearest neighbors. Thus, in the Gaussian kernel entropy, the Gaussian kernel

$$ker(i,j;r)=exp\left(-{\displaystyle \frac{{∥x_i-x_j∥}^2}{10r^2}}\right)$$

(13)

is used. Here $x_i$, $x_j$ are entries of a time series and r is the parameter of the approximate entropy. Other popular kernels include the spherical, Laplacian and Cauchy functions [132].

Of course, the same refinement can be done in the computation of the sample entropy (Section 2.25), an improvement of the approximate entropy. To distinguish between the two resulting kernel entropies, one speaks of kernel-based approximate or sample entropy.

• Applications
  • Complexity of time-series. The authors of [133] present experimental evidence that Gaussian kernel entropy outperforms approximate entropy when it comes to analyze the complexity of time series.
  • Fetal heart rate discrimination. In [134] the authors compare the performance of several kernel entropies on fetal heart rates discrimination, with the result that the circular and Cauchy kernels outperform other, more popular kernels, such as the Gaussian or the spherical ones.
  • Parkinson’s disease. Gaussian kernel entropy, along with other nonlinear features, is used in [135] in the task of automatic classification of speech signals from subjects with Parkinson’s disease and a control set.
  • Pathological speech signal analysis. Reference [132] is a study of several approaches in the field of pathological speech signal analysis. Among the new pathological voice measures, the authors include different kernel-based approximate and sample entropies.

2.19. Kolmogorov-Sinai Entropy

As mentioned in Section 1.2, Kolmogorov-Sinai entropy is a classical entropy introduced by Kolmogorov in ergodic theory in 1958 [136] and extended by Sinai to the theory of measure-preserving dynamical systems in 1959 [5]. Although it is defined using refinements of dynamical partitions of the state space, it is usually estimated via Pesin’s formula [137], which involves the strictly positive Lyapunov exponents of the system; see, e.g., [138] and [139] for other numerical methods. Kolmogorov-Sinai entropy is a fundamental invariant in the theory of metric dynamical systems.

• Applications
  • Time series analysis. The perhaps main practical application of the Kolmogorov-Sinai entropy is the analysis of nonlinear, real-valued time series, where it is used to characterize the underlying dynamical system, in particular, its chaotic behavior. Recent practical examples include short-term heart rate variability [140], physical models of the vocal membranes [139], autonomous driving [141], and EEG-based human emotion recognition [25,142].

2.20. Permutation Entropy

The conventional (or Shannon) permutation entropy of a time series was introduced by Bandt and Pompe in 2002 [143]. It is the Shannon entropy (Section 2.26) of the probability distribution obtained from the ordinal patterns of length $L\ge 2$ in the time series, i.e., the rankings (or permutations) of the series entries in sliding windows of size L. Therefore, permutation entropy depends on the parameter L. In the particular case that the time series is an orbit generated by a selfmap f of a one-dimensional interval, then the permutation entropy rate converges to the Kolmogorov entropy of f (Section 2.19) when $L\to \infty$[144]. This means that the permutation entropy calculated with a sufficiently large L (and divided by $L-1$) is a good estimator of the Kolmogorov entropy of one-dimensional dynamics. Furthermore, permutation entropy is easy to program, relatively robust to noise, and can be computed practically in real time since knowledge of the data range is not needed [145]. See, e.g., [146,147,148,149] for references on theoretical and practical aspects of permutation entropy.

If, instead of the Shannon entropy of the ordinal patterns distribution, we use other entropy, e.g., Rényi entropy (Section 2.24 or Tsallis entropy (Section 2.31, then we obtain the corresponding "permutational version": permutation Rényi entropy, permutation Tsallis entropy, and more [150,151]. There are also "weighted versions" that take into account not only the rank order of the entries in a window but also their amplitudes; see, e.g., [152]. In turn, "multiscale versions" (including multiscale permutation Rényi and Tsallis entropy) account for multiple time scales in time series by using different time delays [153,154,155].

• Applications
  • Analysis of EEGs. One of the first applications of permutation entropy was the analysis of EEGs of subjects with epilepsy because normal and abnormal signals (during epileptic seizures) have different complexities [156]. Furthermore, since permutation entropy can be computed in virtually real time, it has been used to predict seizures in epilepsy patients by tracking dynamical changes in EEGs [147]. Further examples can be found in the reviews [18,147]. Results can be improved using permutation Rényi and Tsallis entropy due to their additional, fine-tunable parameter [150,157].
  • Determinism detection. Time series generated by one-dimensional maps have necessarily forbidden ordinal patterns of all sufficiently large lengths L[146]. Theoretical results under some provisos and numerical results in other cases show that the same happens with higher dimensional maps [158,159]. Therefore, the scaling of permutation entropy with L can distinguish noisy deterministic signals from random signals [146,160].
  • Emotion recognition. Permutation entropy is used to help in tasks of feature extraction in EEGs [25].
  • Nonlinear time series analysis. Permutation entropy has been extensively used in the analysis of continuous-valued time series for its straightforward discretization of the data and ease of calculation. Numerous applications can be found, e.g., in [147,148,149] and the references therein.
  • Obstructive sleep apnea. A combination of permutation entropy-based indices and other entropic metrics was used in [161] to distinguish subjets with obstructive sleep apnea from a control group. The data consisted of heart rate and beat-to-beat blood pressure recordings.
  • Prediction. Permutation entropy has been used along with variational modal decomposition to predict wind power [162].
  • Speech signals. In their seminal paper [143], Bandt and Pompe used precisely permutation entropy to analyze speech signals and showed that it is robust with respect to the window length, sampling frequency and observational noise.
  • The causality-complexity plane. Permutation entropy together with the so-called statistical complexity builds the causality-complexity plane, that has proven to be a powerful tool to discriminate and classify time series [163]. By using variants of the permutation entropy and the statistical complexity, the corresponding variants of the causality-complexity plane are obtained, possibly with enhanced discriminatory abilities for the data at hand [149].
  • Unstructured data. Nearest-neighbor permutation entropy is an innovative extension of permutation entropy tailored for unstructured data, irrespective of their spatial or temporal configuration and dimensionality, including, e.g., liquid crystal textures [164].

2.21. Rao’s Quadratic Entropy

Rao’s quadratic entropy (RQE, not to be confused with Rényi’s collision entropy $R_2$, Section 2.24, also called quadratic entropy) was proposed in 1982 [165] as a measure of diversity in biological populations. Given W species, RQE is defined as

$$\mathrm{RQE}(p_1,\dots ,p_W)=\sum _{i,j=1}^W{\delta }_{i,j}{p}_i{p}_j$$

(14)

where ${\delta }_{i,j}$ is the difference between the i-th and the j-th specie and $\{p_1,...,p_W\}$ is the probability distribution of the W species in the multinomial model.

• Applications
  • Environmental monitoring. RQE helps calculate the environmental heterogeneity index and assist prioritization schemes [166].
  • Genetic diversity metrics. RQE is used to measure diversity for a whole collection of alleles to accommodate different genetic distance coding schemes and computational tractability in case of large datasets [167].
  • Unsupervised classification. RQE is used as a framework in the support vector data description algorithm for risk management, enhancing knowledge in terms of interpretation, optimization, among others [168].

2.22. Rank-Based Entropy

Rank-based entropy (RbE), introduced by Citi et al. in 2014 [169] within the framework of multiscale entropy, measures the disorder or uncertainty in a dataset by analysing the ranks of the data rather than their absolute values, it is useful for ordinal data so that a proper rank can be created. This measure captures the complexity and variability in the data by examining how the ranks of the data points are distributed.

• Applications
  • Anomaly detection. RbE is applied in mixed data analysis to check the influence of categorical features, using Jaccard index for anomaly ranking and classification [170].
  • Feature selection. RbE is used in the Entropy-and-Rank-based-Correlation framework to select features, e.g., in the detection of fruit diseases [171].
  • Mutual information. RbE is used to rank mutual information in decision trees for monotonic classification [172].
  • Node importance. RbE is employed in the analysis of graphs to rank nodes taking into account the local and global structure of the information [173].
  • QSAR models. RbE is employed in Quantitative Structure-Activity Relationship models (QSAR) to analyse their stability via “rank order entropy”, suggesting that certain models typically used should be discarded [174].
  • Time series analysis. RbE is used in terms of correlation entropy to test serial independence in [175]. A multiscale version was used in [169] to study data of heart rate variability.
  • Time series classification. RbE helps classify order of earliness in time series to generate probability distributions in different stages [176].

2.23. Relative Entropy

Relative entropy, also known as Kullback-Leibler (KL) divergence [9], is an information-theoretical measure that quantifies the difference or “distance” between two probability distributions. Specifically, if $\mathrm{p}=(p_1,\dots ,p_W)$ and $\mathrm{q}=(q_1,\dots ,q_W)$ are two probability distributions, then the relative entropy or KL divergence from $\mathrm{p}$ to $\mathrm{q}$, $D\left(\mathrm{p}\right.∥\mathrm{q})$, is defined as

$$D\left(\mathrm{p}\right.∥\mathrm{q})=\sum _{i=1}^W{p}_{i}log{\displaystyle \frac{p_i}{q_i}}.$$

(15)

It follows that $D\left(\mathrm{p}\right.∥\mathrm{q})\ge 0$ (Gibb’s inequality) and $D\left(\mathrm{p}\right.∥\mathrm{q})=0$ if and only if $\mathrm{p}=\mathrm{q}$. Note that $D\left(p\right.∥q)$ is not a distance in the strict sense because $D\left(\mathrm{p}\right.∥\mathrm{q})\ne D\left(\mathrm{q}\right.∥\mathrm{p})$ in general, although it can be easily symmetrized by taking any mean (arithmetic, geometric, harmonic,...) of $D\left(p\right.∥q)$ and $D\left(q\right.∥p)$. Therefore, if $\mathrm{p}$ is a true probability distribution approximated with $\mathrm{q}$, then $D\left(\mathrm{p}\right.∥\mathrm{q})$ is a measure of the approximation error. As another useful example, if $\mathrm{p}$ is a bivariate joint distribution and q is the product distribution of the two marginals, then $D\left(p\right.∥q)$ is the mutual information between the random variables defined by the marginal distributions, equation (10).

See [11] for a generalizacion of divergence (or relative entropy), where the $log{p}_i/q_i$ in equation (15) is replaced by $f(p_i/q_i)$ where f is a convex function on $(0,\infty )$ with $f\left(1\right)=0$.

• Applications
  • Anomaly detection. KL divergence has been used for plane control in Software-Defined Networking as a method to detect Denial of Service attacks in [177].
  • Bayesian networks. The efficient computation of the KL divergence of two probability distributions, each one coming from a different Bayesian network (with possibly different structures), has been considered in [178].
  • Feature selection. The authors of [179] show that the KL divergence is useful in information-theoretic feature selection due to the fact that maximising conditional likelihood corresponds to minimising KL-divergence between the true and predicted class posterior probabilities.
  • Multiscale errors. KL divergence is a useful metric for multiscale errors [48]. A recent application to the study of the behavior of various nonpolar liquids via the Relative Resolution algorithm can be found in [180].
  • Parameter minimization in ML. Parameters that minimize the KL divergence minimize also the cross entropy and the negative log likelihood. So, the KL divergence is useful in optimization problems where the loss function is a cross-entropy [181].

2.24. Rényi Entropy

Rényi entropy $R_{\alpha }$, where $\alpha >0$ and $\alpha \ne 1$, was introduced by Alfréd Rényi in 1961 [12] as a generalization of Shannon entropy in the sense that $R_1$ is set equal to the latter by continuity; see Section 1.2 and the review [13] for detail. The parameter $\alpha$ allows for different emphasis on the probabilities of events, making it a versatile measure in information theory and its applications. Thus, for $\alpha <1$ the central part of the distribution is flattened, i.e., high probability events are suppressed, and low-probability events are enhanced. The opposite happens when $\alpha <1$. As a function of the parameter $\alpha$, the Rényi entropy is non-increasing. Particular cases include the Hartley entropy or max-entropy $R_0={lim}_{\alpha \to 0}{R}_{\alpha }$, the collision or quadratic entropy $R_2$, and the min-entropy $R_{\infty }={lim}_{\alpha \to \infty }{R}_{\alpha }$.

• Applications
  • Anomaly Detection. Lower values of $\alpha$ highlight rare events, making this measure useful for identifying anomalies [182,183]. In particular, Rényi entropy is used in network intrusion detection for detecting botnet-like malware based on anomalous patterns [184].
  • Automated identification. Average Renyi entropy, along with other entropic measures, have been used as inputs for SVM algorithms to classify focal or non-focal EEGs of subjects affected by partial epilepsy [185].
  • Clustering. Rényi entropy can provide robust similarity measures that are less sensitive to outliers [182].
  • Extreme entropy machines. Rényi’s quadratic entropy $R_2$ is used in the construction of extreme entropy machines to improve classification problems [186].
  • Feature selection and character recognition. Adjustment of the parameter $\alpha$ can help to emphasize different parts of the underlying probability distribution and hence the selection of the most informative features. Rényi entropy is used for feature selection in [182,187]. Max-entropy is used in [188] for convolutional feature extraction and improvement of image perception.
  • Medical time series analysis Applications of the Rényi entropy in time series analysis reach from epilepsy detection in EEG (see, e.g., [26]) over artifact rejection in multichannel scalp EEG (see [189] and references therein) to early diagnosis of Alzheimer’s disease in MEG data (see, e.g., [190].

2.25. Sample Entropy

Sample Entropy was introduced by Richman and Moorman in 2000 [191] as an improvement over approximate entropy (Section 2.1), namely, its calculation is easier and independent of the time series length. Sample entropy is the negative natural logarithm of the conditional probability that close sequences of m points remain close when one more point is added, within a tolerance $r>0$.

• Applications
  • Automated identification. Average sample entropy and other entropy measures are used as input for an SVM algorithm to classify focal and non-focal EEG signals of subjects with epilepsy [185].
  • Fault diagnosis. Sample entropy has been used for multi-fault diagnosis in lithium batteries [192].
  • Image classification. Sample entropy, in the form of multivariate multiscale sample entropy, is used for classifying RGB colour images to compare textures, based on a threshold to measure similarity [106].
  • Image texture analysis. Two-dimensional sample entropy has shown to be a useful texture feature quantifier for the analysis of biomedical images [193].
  • Mutual information. Modified sample entropy has been used in skin blood flow signals to analyse mutual information and, hence, study the association of microvascular dysfunction in different age groups [194].
  • Neurodegenerative disease classification. Sample entropy is used to classify neurodegenerative diseases. Gait signals, support vector machines and nearest neighbours are employed to process the features extracted using sample entropy [195].
  • Short signal analysis. Quadratic sample entropy is an evolution of sample entropy that has been proposed for the analysis of short length biomedical signals [196,197].
  • Time-series analysis. Sample entropy, often in the form of multiscale sample entropy, is a popular tool in time series analysis, in particular with biomedical data [198]. For example, it is used for the fast diagnosis and monitoring of Parkinson’s disease [199] and human emotion recognition [25] using EEGs. A modified version of multiscale sample entropy has recently been used for diagnosing epilepsy [200]. See [29] for an overview of applications of sample entropy to the analysis of physiological time series.
  • Weather forecasting. Sample entropy is applied in weather forecasting by using transductive feature selection methods based on clustering-based sample entropy [201].

2.26. Shannon Entropy

Shannon entropy was introduced by Claude Shannon in his foundational 1948 paper "A Mathematical Theory of Communication" [2] as the cornerstone of digital and analog Information Theory. Indeed, Shannon entropy informs the core theorems of Information Theory [9]. In the case of discrete probability distributions, the Shannon entropy measures the uncertainty about the outcome of a random variable with the given distribution or, alternatively, the expected information conveyed by that outcome, maximum uncertainty (or minimum information) being achieved by uniform distributions. It also quantifies the rate of information growth produced by a data source modelled as a stationary random process. In the case of continuous probability distributions, the Shannon entropy is called differential entropy and its applications to data analysis and machine learning were the subject of Section 2.5.

In this section we only consider discrete probability distributions, i.e., finite state random variables and processes (possibly after a discretization or symbolization of the data). A typical example are real-valued time series, where the Shannon entropy is used to measure their complexity and, hence, distinguish between different dynamics. The Shannon entropy of certain probability mass distributions (e.g., ordinal patterns of a time series, power spectrum of a signal, eigenvalues of a matrix) may have particular names (permutation, spectral, von Neumann entropies); in this case, the applications of such entropies are presented in the corresponding sections.

• Applications
  • Accurate prediction. Shannon entropy is employed in machine learning models to improve the accuracy of predictions of molecular properties in the screening and development of drug molecules and other functional materials [202].
  • Anomaly detection. Shannon entropy is employed in sensors (Internet of Things) to identify anomalies using the CorrAUC algorithm [203].
  • Artificial intelligence. Shannon entropy contributes to the creation of the Kolmogorov Learning Cycle, which acts as a framework to optimize "Entropy Economy", helped by the intersection of Algorithmic Information Theory (AIT) and Machine Learning (ML). This framework enhances the performance of the Kolmogorov Structure Function, leading to the development of "Additive AI". By integrating principles from both AIT and ML, this approach aims to improve algorithmic efficiency and effectiveness, driving innovation in AI by balancing information theory with practical machine learning applications [204].
  • Automated identification.Average Shannon entropy and other entropy measures are used as inputs of an SVM algorithm to classify focal or non-focal EEG signals of subjects with epilepsy [185].
  • Fault bearing diagnosis. Multi-scale stationary wavelet packet analysis and the Fourier amplitude spectrum are combined to obtain a new discriminative Shannon entropy feature that is called stationary wavelet packet Fourier entropy in [205]. Features extracted by this method are then used to diagnose bearing failure.
  • Feature selection. Shannon’s mutual information entropy is used to design an entire information-theoretic framework that improves the selection of features in [179]. Shannon entropy is employed in biological science to improve classification of data in clustering of genes using microarray data [206].
  • Hard clustering. Shannon entropy is used as a criterion to measure the confidence in unsupervised clustering tasks [207].
  • Natural language processing.Shannon entropy quantifies the predictability (or redundancy) of a text. Therefore, it is instrumental in language modelling, text compression, information retrieval, among others [2,9]. For example, it is used in [208] for keyword extraction, i.e., to rank the relevance of words.
  • Policy learning.Shannon entropy acts as a regularization inside of an iterative policy optimization method for certain quadratic linear control scenarios [209].
  • Signal analysis. Shannon entropy is used as the cost functional of compression algorithms in sound and image processing [210].
  • Statistical inference. According to the Maximum Entropy Principle of Jaynes [211], "in making inferences on the basis of partial information we must use the probability distribution which has maximum entropy subject to whatever is known". This principle has been traditionally applied with the Shannon entropy and several moment constraints of a probability distribution to infer the actual distribution [9,18].

2.27. Spectral Entropy

Spectral entropy, proposed by Kapur and Kesavan in 1992 [212], is an entropy based on the Shannon entropy. Here, the probability distribution is the (continuous or discrete) power spectrum in a representative frequency band, obtained from a signal or time series via the Fourier transform, and conveniently normalized. Hence, the spectral entropy characterizes a signal by the distribution of power among its frequency components.

• Applications
  • Audio analysis. Spectral entropy has been applied for robust audio content classification in noisy signals [213]. Specifically, spectral entropy is used to segment input signals into noisy audio and noise. Also, spectral entropy (in the form of Multiband Spectral Entropy Signature) has been shown to outperform other approaches in the task of sound recognition [214].
  • Damage event detection. Spectral entropy detects damage in vibration recordings from a wind turbine gearbox [215].
  • Data time compression. Spectral entropy has been successfully applied to identify important segments in speech, enabling time-compression of speech for skimming [216].
  • Deep learning synchronization. Spectral entropy evaluates synchronization in neuronal networks, providing analysis of possibly noisy recordings collected with microelectrode arrays [217].
  • Feature extraction. Spectral entropy is used to extract features from EEG signals in [25] (emotion recognition) and [218] (assessment of the depth of anaesthesia).
  • Hyperspectral anomaly detection. Hyperspectral Conditional Entropy enters into the Entropy Rate Superpixel Algorithm, which is used in hyper spectral-spatial data to recognize unusual patterns [219].
  • Signal detection. Spectral entropy has been used to detect cetacean vocalization in marine audio data [220]. The time frequency decomposition was done with the short time Fourier transform and the continuous wavelet transform.

2.28. Tone Entropy

Tone entropy was proposed by Oida et al. in 1997 [221] to study heart period fluctuations in electrocardiograms. Specifically, tone entropy is the Shannon entropy of a probability distribution derived from the percentage index of heart period variation. Therefore, its applications are mainly in cardiology.

• Applications
  • Biomedical analysis. Tone entropy has been employed to study the autonomic nervous system in age groups at high-risk of cardiovascular diseases [222]. In [223], tone entropy was used to study the influence of gestational ages on the development of the foetal autonomic nervous system by analyzing the foetal heart rate variability.
  • Time series. Tone entropy has been used in time series analysis to differentiate between physiologic and synthetic interbeat time series [224].

2.29. Topological and Topology-Based Entropies

Topological entropy was introduced in [6] to measure the complexity of continuous dynamics on topological spaces. On metric spaces, topological entropy measures the exponential growth rate of the number of distinguishable orbits with finite precision. See, e.g., [225,226] for exact formulas and fast algorithms to compute the topological entropy of piecewise monotone maps and multimodal maps.

In time series analysis and digital communication technology, one is mostly interested in spaces with a finite number of states. Such time series can be the result symbolizing a continuous-valued time series. In this case, the states are usually called letters (or symbols), the state space is called alphabet and the blocks of letters are called words (which correspond to the "admissible" or "allowed" strings of letters). If $A\left(n\right)$ is the number of words of length n, then the topological entropy of the time series is defined as

$$h_{top}=\underset{n\to \infty }{lim}log(A\left(n\right)/n),$$

(16)

where the base of the logarithm is usually 2 or e.

Along with the above "conventional" topological entropies in dynamical systems and time series analysis, there are a number of adhoc entropies in time series analysis, sometimes also called topological entropies. This name is due to the fact that those entropies are based on topological properties extracted from the data, for example, via graphs or persistent homology. For clarity, here we refer to them as topology-based entropies. Examples include graph entropy and horizontal visibility graph entropy (Section 2.14). See [227] for an account of topological methods in data analysis.

• Applications
  • Cardiac dynamics. Given a (finite) time series, the out-link entropy is derived from the adjacency matrix of its ordinal network. This entropy has been used to classify cardiac dynamics in [228].
  • Convolutional neural networks. The authors of [229] propose a method for quantitatively clarifying the status of single unit in convolutional neural networks using algebraic topological tools. Unit status is indicated via the calculation of a topology-based entropy, called feature entropy.
  • Damage detection. Persistent entropy can also address the damage detection problem in civil engineering structures. In particular, to solve the supervised classification damage detection problem [230].
  • Detection of determinism. Permutation topological entropy (i.e., the topological entropy of the distribution of ordinal patterns of length $L\ge 2$ obtained from a time series) can be used to detect determinism in continuous-valued time series. Actually, it suffices to check the growth of ordinal patterns with increasing L’s, since this growth is exponential for deterministic signals ($h_{top}$ converges to a finite number) and factorial for random ones ($h_{top}$ diverges) [146,158].
  • Financial time series. Topological entropy has been applied to horizontal visibility graphs of financial time series in [231].
  • Similarity of piecewise linear functions. Piecewise linear functions are a useful mathematical tool in different areas of applied mathematics, including signal processing and machine learning methods. In this regard, persistent entropy (a topological entropy based on persistent homology) can be used to measure their similarity [232].

2.30. Transfer Entropy

A relevant question in time series analysis of coupled random or deterministic processes is the causality relation, i.e., which process is driving and which is responding. Transfer entropy, introduced by Schreiber in 2000 [233], measures the information exchanged between two processes in both directions separately. It can be considered as an information-theoretical (or nonlinear) implementation of Granger causality [234].

Given two stationary random processes $\mathbf{X}={\left(X_t\right)}_{t\ge 0}$ and $\mathbf{Y}={\left(Y_t\right)}_{t\ge 0}$, the transfer entropy from $\mathbf{Y}$ to $\mathbf{X}$, $T_{\mathbf{Y}\to \mathbf{X}}$, is the reduction of uncertainty in future values of $\mathbf{X}$, given past values of $\mathbf{X}$, due to the additional knowledge of past values of $\mathbf{Y}$. For simplicity, we consider here the simplest (lowest dimensional) case:

$$T_{\mathbf{Y}\to \mathbf{X}}=S\left(X_{t+1}\right|X_t)-S\left(X_{t+1}\right|X_t,Y_t)$$

(17)

where $S\left(X_{t+1}\right|\dots )$ is the conditional Shannon entropy of the variable $X_{t+1}$ on the other variable(s) [9]. If the process $\mathbf{Y}$ is not causal to $\mathbf{X}$ (i.e., they are independent), then $S\left(X_{t+1}\right|X_t,Y_t)=S\left(X_{t+1}\right|X_t)$ and $T_{\mathbf{Y}\to \mathbf{X}}=0$; otherwise, $S\left(X_{t+1}\right|X_t,Y_t)<S\left(X_{t+1}\right|X_t)$ and $T_{\mathbf{Y}\to \mathbf{X}}>0$. Observe that $T_{\mathbf{Y}\to \mathbf{X}}$ is not an entropy proper but a conditional mutual information, namely: $T_{\mathbf{Y}\to \mathbf{X}}=I(X_{t+1};Y_t\mid X_t)$ [9,18].

• Applications
  • Accelerated training in Convolutional Neural Networks (CNN). The authors of [235] propose a training mechanism for CNN architectures that integrates transfer entropy feedback connections. In this way, the training process is accelerated as fewer epochs are needed. Furthermore, it generates stability, hence, it can be considered a smoothing factor.
  • Improving accuracy in Graph Convolutional Neural Networks (GCN). The accuracy of a GCN can be improved by using node relational characteristics (such as heterophily), degree information, and feature-based transfer entropy calculations. However, depending on the number of grah nodes, the computation of the transfer entropy can significantly increase the computational load [236].
  • Improving neural network performance. A small, few-layer artificial neural network that employs feedback can reach top level performance on standard benchmark tasks, otherwise only obtained by large feed-forward structures. To show this, the authors of [237] use feed-forward transfer entropy between neurons to structure feedback connectivity.
  • Multivariate time series forecasting. Transfer entropy is used to establish causal relationships in multivariate time series converted into graph neural networks, each node corresponding to a variable and edges representing the casual relationships between the variables. Such neural networks are then used for prediction [238].
  • Time series analysis. The main application of transfer entropy since its formulation has been the analysis of multivariate time series (whether biomedical, physical, economical, financial, ...) for revealing causal relationships via information directionality. See [239] and the references therein for the conceptual underpinnings and practical applications.

2.31. Tsallis Entropy

Tsallis entropy $T_q$, defined in Section 1.2, equation (7), was introduced by Tsallis in 1988 in Statistical Mechanics [15]. The parameter q takes the real values $q>0$, $q\ne 1$; $T_q$ converges to the Shannon entropy when $q\to 1$ [13]. Tsallis entropy is identical in form to the Havrda–Charvát entropy (Section 2.15).

Tsallis entropy is particularly useful for describing systems with non-extensive properties, such as long-range interactions, non-Markovian processes, and fractal structures. In machine learning, Tsallis entropy is used to improve algorithms in areas such as clustering, image segmentation, and anomaly detection by changing and fine tuning the parameter q. See [240] for the general properties of the Tsallis entropy.

• Applications
  • Anomaly detection. Tsallis entropy is used in network intrusion detection by detecting botnet-like malware based on anomalous patterns in the network [184].
  • Clustering. A Tsallis entropy based categorical data clustering algorithm is proposed in [241]. It is shown there, that when the attributes have a power law behavior the proposed algorithm outperforms existing Shannon entropy-based clustering algorithms.
  • Feature selection. Tsallis-entropy-based feature selection is used in [242] to identify significant features, which boosts the classification performance in machine learning. The authors propose an algorithm to optimize both the classifier (a Support Vector Machine) and Tsallis entropy parameters, so improving the classification accuracy.
  • Image segmentation. Tsallis Entropy is maximized to use it for segmenting images by maximizing the entropy within different regions of the image [243].
  • Pre-seismic signals. Tsallis entropy has been used in [244] to analyze pre-seismic electromagnetic signals.

2.32. Von Neumann Entropy

Von Neumann entropy [1] is the equivalent of Shannon entropy in Quantum Statistical Mechanics and Quantum Information Theory. It is defined via the density matrix of an ensemble of quantum states, which (i) is Hermitian, (ii) has unit trace, and (iii) is positive semidefinite. Therefore, the eigenvalues of the density matrix build a probability distribution and, precisely, the von Neumann entropy of the ensemble is the Shannon entropy of the probability distribution defined by the eigenvalues of the corresponding density matrix.

The same approach can be used with any matrix with the properties (i)-(iii), for example, the Pearson correlation matrix of a Markov chain (divided by the dimension of the matrix) or the Laplacian matrix of a graph. This explains the use of von Neumann entropy in classical data analysis as well.

• Applications
  • Feature selection and dimensionality reduction. Von Neumann entropy is employed in the case of kernelized relevance vector machines to asses dimensionality reduction for better model performances [245].
  • Graph-based learning. In [246] the authors propose a method to identify vital nodes in hypergraphs that is based on von Neumann entropy. More precisely, this method is based on the high-order line graph structure of hypergraphs and measures changes in network complexity using von Neumann entropy.
  • Graph similarity and anomaly detection. The von Neumann graph entropy (VNGE) is used to measure the information divergence and distance between graphs in a sequence. This is used for various learning tasks involving network-based data. The Fast Incremental von Neumann Graph Entropy algorithm reduces the computation time of the VNGE, making it feasible for real-time applications and large datasets [247].
  • Network analysis. Von Neumann entropy is used in [248] to build visualization histograms from the edges of networks and then component analysis is performed on a sample for different networks.
  • Pattern recognition in neurological time series. Von Neumann entropy has been used (together with other entropies) in [249] for automated pattern recognition in neurological conditions, a crucial task in patient monitoring and medical diagnosis.

2.33. Wavelet Entropy

Wavelet entropy was introduced by Rosso et al. in 2001 [250]. It combines the wavelet transform with the concept of entropy to analyse the complexity and information content of multi-frequency signals. More precisely, the wavelet transform decomposes a signal into components at various scales, capturing both time and frequency information, while wavelet entropy quantifies the degree of disorder or unpredictability in those components, thus providing insights into the signal structure and complexity. Wavelet entropy is defined by the Shannon formula, but here the probability distribution corresponds to different resolution levels.

• Applications
  • Emotion recognition. Wavelet entropy can detect little variations in signals and it has been used in [25] to develop an automatic EEG classifier.
  • Fault detection. Wavelet entropy is applied to monitor the condition of machinery and detect faults by analysing vibration signals [251].
  • Feature extraction. Wavelet entropy is used to extract features from biomedical signals such as EEG and ECG to identify different physiological states or detect abnormalities [250].

3. Discussion

In Section 1 of this review, we provided a brief historical account of the concept of entropy. Due to its seemingly unconnected appearances in Thermodynamics, Statistical Mechanics, Information Theory, Dynamical Systems, etc., this was necessary to clarify its role in data analysis and machine learning, the subject of the present review. To this end, we gave an axiomatic characterization of entropy and generalized entropies, and acknowledged the inclusion in our review of several entropy-based probability functionals that also go by the name of entropy in the literature and are useful for the mentioned applications. In the case of univariate arguments, entropy quantifies the uncertainty, complexity, and information content of the data. In the case of bivariate and multivariate arguments, entropy quantifies similarity, distance, and information flow.

In Section 2 we collected a representative sample of 33 entropies (including the classical entropies visited in Section 1.2) to show the versatility and potential of the general concept of entropy in practical issues. Indeed, applications such as biomedical signal analysis, fault diagnosis, feature extraction, anomaly detection, optimization cost, and more highlight the diversity of the applications of entropy in data analysis and machine learning.

Section 2 also showed that, more than 150 years after its formulation, entropy and its applications remain the subject of intense research. In fact, new concepts of entropy, evolutions and generalizations are constantly being proposed in the literature to address new challenges. As a result, entropy is being applied to current topics of applied mathematics, in particular, data analysis and machine learning.

4. Conclusions

We conclude this review with some final remarks.

• The choice of a particular entropy-based method depends in general on the application. Some methods may be more popular than others because they are designed for the purpose or dataset at hand, or simply because they have some computational advantage. In this regard, Section 2 presented possible candidates for different applications but no performance comparison between them was discussed. In fact, such a comparison would require a case-by-case test as, for example, in Reference [103], where the authors study motor fault detection with the approximate, dispersion, energy, fuzzy, permutation, sample and Shannon entropies (see Section 2.13 for the best performer). Along with the selection of the “right” entropy, a common concern among practitioners is the choice of parameters and hyperparameters. A combination of methods and parameter settings may be also a good approach in practice [161] [249].
• We have not included applications of entropy to cryptography in this review because they belong to the general field of Data Science (through data security) rather than to Data Analysis. Entropy (mainly Shannon’s and Rényi’s entropies) have been applied to measure the “randomness” of encrypted messages in the so-called chaotic cryptography, which applies ideas from Chaos Theory and chaos synchronization to the masking of analog signals [252]. To deal with digital signals, new tools such as discrete entropy [253] and discrete Lyapunov exponents [254] have also been developed for application to chaotic cryptography, inspired by their conventional counterparts.
• We have not delved into the numerical methods to compute entropies from the data. Being functionals of probability distributions, most methods for computing entropy are based on the estimation of data probabilities. In the case of discrete-valued data, the probabilities are usually estimated via relative frequencies (the maximum likelihood estimator) and possibly extrapolation methods in case of undersampling [146]. In the case of continuous-valued data, the probability densities are usually estimated via kernel density estimation (also called Parzen-Rosenblatt windowing) [255]. Furthermore, there are methods that do not rely on probability estimation like, e.g., Lempel-Ziv complexity, which resorts to pattern matching [85]. Also, in some particular cases, the entropy can be estimated via spectral information. For example, the quadratic Rényi entropy $R_2$ can be directly estimated via inner product matrices and principal component analysis [256]. See [257] for a general review on the estimation of entropy.