Characterizing Spatiotemporal Complex Patterns from Entropy Measures

1. Introduction

The intricate relationship between probability and entropy is a cornerstone in information theory and statistical thermodynamics, providing a robust framework for analyzing a multitude of phenomena ranging from data transmission processes to the behavior of many physical systems. Entropy, derived from the probability distribution of the states of a process or system, can be interpreted as a quantitative measure of randomness or disorder, offering deep insights into the underlying dynamics of several complex systems (see, for instance, Refs. [2,7,12,23,24,25]).

From a thermodynamic perspective, the entropy concept is intimately tied to the statistical mechanics of microstates. The entropy, S, is defined by Boltzmann’s entropy equation, $S=k_{B}ln\mathrm{\Omega }$, where $k_B$ is the Boltzmann constant and $\mathrm{\Omega }$ represents the number of microstates. This relationship can be interpreted as the degree of disorder or randomness in a system’s microscopic configurations, drawing a direct connection between the macroscopic observable properties and the statistical behavior of microstates. Complementary, in the realm of information theory, entropy is fundamentally concerned with quantifying the expected level of “information,” “surprise,” or “uncertainty” in the potential outcomes of a system [3]. This quantification is intricately linked to the probability distribution of these outcomes. It essentially measures the average unpredictability or the requisite amount of information needed to describe a random event, thereby providing a metric for the efficiency of data transmission and encoding strategies. Therefore, the duality of the entropy interpretation works as a bridge between the abstract realm of information and the tangible world of the statistics of physical systems. It encapsulates the essence of entropy as a fundamental measure, providing a unifying lens through which the behavior of complex systems, whether in the context of information processing or thermodynamics, can be coherently understood and analyzed. This interdisciplinary approach not only deepens our understanding of individual phenomena but also reveals the underlying universality of the concepts of randomness and information across diverse scientific domains.

In the scenario described above, it is necessary to identify entropy measures that are effective in characterizing spatiotemporal patterns of complex processes typically observed or simulated in $2D+1$ and $3D+1$: following the notation of the amplitude equation theory, where D corresponds to the spatial dimension in which the amplitude of a variable fluctuates over time. This need is justified by the great advances in the generation of Big Data in computational physics, with emphasis on Direct Numerical Simulation (DNS) of turbulence [17,18], ionized fluids [19,21,22,26,27], reactive-diffusive processes [14], to highlight a few.

Our main objective in this work is to present and evaluate the performance of a set of information entropy measurements, conjugated two by two, to characterize different classes of 3D structural patterns arising from nonlinear spatiotemporal processes. To this end, the article is organized as follows: the analytical methodology is presented in Section 2, and the data in Section 3. The results, in the context of a benchmark based on the generalization of the Silhouette Score, are presented and interpreted in Section 4. Our concluding remarks, with emphasis on pointing out the usability of the method in the context of data-driven science, are presented in the last section.

2. Methods

Various entropy metrics have been proposed in the literature, including spectral entropy, permutation entropy, and statistical complexity.

The process of defining a new metric typically involves two fundamental steps: (i) establishing the probability definition, and (ii) determining the entropic form. This framework allows for the generalization of any new metric by specifying these two steps1. In the following subSection 2.1 and Section 2.2, we present respectively the key techniques for defining probabilities and entropic forms. Subsequently, in subSection 2.3, we introduce a methodology to assess these metrics using criteria commonly applied to clustering techniques.

2.1. Probabilities

Probability is a concept that quantifies the likelihood of an event occurring. It is expressed as a numerical value between 0 and 1. Here, 0 signifies the complete impossibility of an event, while 1 denotes absolute certainty. Mathematically, if we consider a process with a finite number of possible outcomes, the probability $Pr\left(E\right)$ of an event E is defined by the ratio:

$$Pr\left(E\right)=\frac{\mathrm{Number}\mathrm{of}\mathrm{favorable}\mathrm{outcomes}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{possible}\mathrm{outcomes}}.$$

(1)

This definition is useful to gain insight of systems that produce discrete real-valued outcomes. In such a case, the histogram of proportions of observed events is the usual tool to estimate the underlying probability distribution of such outcomes.

Many systems produce continue-valued multidimensional outcomes, and the observer needs to define how to estimate a useful probability able to characterize their behavior. Approaches such as permutation and spectral analysis incorporate spatial locality and scale considerations to elucidate the occurrence of specific patterns.

In the permutation approach, local spatial differences (increase, decrease, or constancy) represent the states. New states can be generated by permuting the array elements. Thus, the probabilities account for the occurrences of those states. To extend this definition to multiple dimensions, a given array is flattened. Further details of this technique have been explored by Pessa ([5]).

Another methodology involves spectral analysis, wherein the probability is computed as the Power Spectrum Density (PSD) of the signal $P\left(\omega \right)$, normalized accordingly. Since this approach considers the probability associated with a given frequency $\omega$, it explores the scaling relation of the signal. For instance, white noise, characterized by equal power across all frequencies, represents a type of signal exhibiting maximum entropy. Whereas the red noise presents a higher PSD for lower frequencies, which represents a lower entropy value. This approach has been popularized in the literature to study time series [7,8,33]. The probabilities presented in this section describe the possible spatial states, while the subsequent subsection elaborates on the entropic characterization of this system.

2.2. Entropic Forms

Several entropy equations and generalizations have been proposed, such as the Boltzmann-Gibbs entropy (also known as Shannon entropy), Tsallis entropy, and Rényi entropy. The most general form is the Shannon entropy, expressed as:

$$S_H=-\sum _{i=1}^W{p}_{i}log{p}_i.$$

(2)

Here, $p_i$ is the probability of state i, which can also comprise complex numbers [10], and W is the size of the set of possible events. The value of $S_H$ depends on the distribution. Notably, $S_H$ is maximum when all the probabilities are equal, i.e., under the uniform distribution; in this case, $S_H=-logW$, and minimum when $p_i$ is Dirac’s delta. To account for this maximum value, the normalized Shannon entropy is given by:

$$S_H=-\frac{{\sum }_{i=1}^W{p}_{i}log{p}_i}{logW}.$$

(3)

Another significant entropic form is the Tsallis entropy, proposed as a generalization of Boltzmann-Gibbs entropy [1], with the equation:

$$S_q=\frac{1-{\sum }_{i=1}^W{p}_i^q}{q-1},$$

(4)

where q is the entropic index or nonextensivity parameter, and plays a crucial role in determining the degree of nonextensivity in the Tsallis entropy.

It is important to explore a range of values for the parameter q to derive a metric distinct from Shannon entropy since ${lim}_{q\to 1}{S}_q=S_H$. Therefore, we suggest exploring values for q in the range $1<q<5$ and seek a relationship denoted by $\alpha$, where $log{S}_q=\alpha logq$. This approach enables the examination of this generalization of $S_H$.

A unique strategy to characterize complex non-linear systems has been the Gradient Pattern Analysis (GPA). This technique involves computing a set of metrics derived from the gradient lattice representation and the gradient moments, see Appendix A. Specifically, we highlight $G_4$, which is determined as the Shannon entropy from the complex representation of the gradient lattice:

$$G_4=\left|\sum _{j=0}^{V_A}\frac{z_j}{z}ln\frac{z_j}{z}\right|.$$

(5)

In the lattice context, the gradient signifies the local variation of amplitudes, computed as the spatial derivative at every embedding dimension. From these spatial derivatives, the complex representation:

$$z_j=\left|v_j\right|e^{i{\theta }_j},$$

(6)

is formed, comprising both the modulus ($|v_j|$) and phases (${\theta }_j$). To obtain a probability, the complex notation is normalized by $z=\sum z_j$. For a in-depth review of this metric, please refer to [9,10]. Table 1 provides a summary of all combinations of entropic forms with associated probabilities, along with the GPA metric, that were examined in this study.

To assess the efficacy of each metric and explore the impact of various combinations of probability definitions with entropic forms, we introduce a criterion outlined in the subsequent section. This criterion is formulated with a focus on clustering the entropy measures of the dataset.

2.3. Silhouette Score and Generalized Silhouette Score

Non-supervised algorithms face unique challenges, a remarkable one is defining their efficiency. The silhouette score is a criterion for defining if a set has been well-clusterized [34]. Given an element $x_i$ in a cluster ${\pi }_k$, this metric is computed as follows [11,12]:

$$s\left(x_i\right)=\frac{b\left(x_i\right)-a\left(x_i\right)}{max\left\{b\left(x_i\right),a\left(x_i\right)\right\}},$$

(7)

where $a\left(x_i\right)$ is the average dissimilarity, that is the average distance of $x_i$ to all other elements in the cluster ${\pi }_k$, and $b\left(x_i\right)$ is the average distance to elements of other clusters. The greater the $s\left(x_i\right)$ value, the better performance of the clustering algorithm because it has produced groups with low dissimilarity and large distances between clusters. This technique can be extended to feature extraction if one considers the individual datasets as the clusters ${\pi }_k$. However, it is equally essential to account for the potential correlation between metrics, as metrics may inadvertently capture the same data aspects, which is undesirable. To mitigate this, we use the modulus of the Pearson correlation $\left|r\right|$ to form the penalty term $1-\left|r\right|$ as follows:

$$s^{\prime }\left(x_i\right)=\left(1-\left|r\right|\right)\frac{b\left(x_i\right)-a\left(x_i\right)}{max\left\{b\left(x_i\right),a\left(x_i\right)\right\}},$$

(8)

which we call the Generalized Silhouette Score (GSS).

After defining a group of entropy measurements and the tool (GSS), which allows determining the best pair of measurements to compose a 2D parameter space, we selected the data set to test and validate our methodological approach.

3. Data

Our main objective is to test the performance of a space composed of two entropy measures in which it is possible to distinguish different classes of complex spatiotemporal processes. For this first study, we chose turbulence-related processes and simulated dynamic colored noises.

We employ simulated data related to the following processes: (i) white noise; (ii) colored noise; (iii) weak turbulence; (iv) hydrodynamic turbulence; and (v) magnetohydrodynamic turbulence (MHD). The main reason for choosing these processes, except colored noise, is that they all present random-type patterns with underlying dynamic characteristics based on physical processes described by partial differential equations (diffusion, reaction and advection). Each was obtained from simulations identified in Table 2.

Based on the power law scaling algorithm technique [13], we created our noise simulator [15]. The data representing weak turbulence (also called chemical or reactive-diffusive turbulence) were obtained from the solution of the Ginzburg-Landau Complex Equation [14,16]. The hydrodynamic turbulence patterns were selected from the John Hopkins database (JHTDB) [20], and the MHD turbulence simulation was simulated using the PENCIL code [19]. Details regarding the simulations are provided in the supplementary material.

To test the approach based on entropy measurements, we selected a total of 25 snapshots representing the evolution of each chosen process. The snapshots were extracted from 3D simulations, taking as a criterion the analysis of the central slice of each hypercube, as the measurement techniques used to act on matrices within a two-dimensional approach.

Figure 1 shows representative snapshots of the respective spatiotemporal processes. These visualizations provide a compelling narrative of the dynamic behavior of each system, highlighting the wide variety of patterns that emerge through temporal dynamics in the phase space.

The numerical procedures and/or technical acquisition details related to the data shown in Figure 1 are available as supplementary material in the repository3, in the section entitled Data Simulations.

4. Results and Interpretation

The analyses in this study were conducted within 2D metric spaces, encompassing all possible entropy measure combinations. Based on the minimum information principle, this configuration offers advantages in terms of interpretability, considering the minimum set of parameters that can be addressed as labels in a possible machine learning interpretation. Our approach to measuring entropies from the data follows the following steps:

• Input of a snapshot;
• Pre-processing whose output is a $64\times 64$ matrix with amplitude ranging from 0 to 255;
• Generation of 3 matrix data outputs: 2D-histogram, 2D-Permutation and 2D-FFT Spectra
• For each of the three domains, the entropy measures are calculated.

Given the definition of the three types of domains interpreted as probability (from histogram, permutation and spectrum), we have six entropy variations, as detailed in Section 2. To distinguish these metrics, we introduced superscripts denoted by h for histogram probability, p for permutation probability, and s for spectral probability. The GPA analysis yields another metric, resulting in 21 scores, as illustrated in Figure 2.

As a result, the most effective combination is the pair Spectral Tsallis Entropy ($S_q^s$) and Permutation Shannon Entropy ($S_H^p$). A visual representation of this space, accompanied by some snapshots, is presented in Figure 3. In this space, the metrics reveal a constant Permutation Shannon Entropy dynamical noise systems, that are solely distinguished by the Spectral Tsalllis Entropy, indicating the differences of scaling effects in pattern formation. Conversely, the distinct complex nonlinear characteristics and reaction terms observed in MHD simulations are more pronounced in Permutation Shannon Entropy, accentuating the diversity of localized patterns alongside the larger-scale ones.

The analysis of the entropy distribution is essential in a classification context, as it offers insights into the similarity between a new dataset and various models. However, carefully analysing the entropy metrics over time can highlight important aspects of the underlying physical processes. For instance, the transition from initial conditions to an oscillatory relaxation state is evident in Figure 4. This outcome aligns with expectations in the context of the CGL system, owing to the periodic nature of the reaction term. However, it is essential to highlight that in this introductory study, we avoided simulations with more complex regimes (such as relaxations) as the primary purpose here is to present a new method, and the objective here is not to use it to deepen the physical interpretation of each process.

5. Outlook

Based on the study and approach presented here, we defined a methodological pipeline for spatiotemporal characterization of simulated and/or observed complex processes (Figure 5). The method can be applied to identify and segregate different classes of processes, as well as to classify isolated patterns when necessary. In a context where measured and simulated data may exist, it also serves to validate models. Likewise, the pair of entropy measurements can also serve as a binomial label for training deep learning architectures for automatic classification.

6. Concluding Remarks

This work undertook a comprehensive analysis of entropy metrics and their application to complex extended non-linear systems. The study explored new approaches, including different entropy measures and a new generalized silhouette score for measure evaluation.

Through meticulous consideration of canonical datasets, distinct patterns have been characterized in terms of entropy metrics. The pivotal finding was the identification of the optimal pair: Spectral Tsallis Entropy ($S_q^s$) and Permutation Shannon Entropy ($S_H^p$), yielding superior outcomes in the generalized silhouette score. This combination showcased efficacy in distinguishing spatiotemporal dynamics coming from different classes of turbulent-like processes, including pure stochastic 2D $1/f^{-k}$ (colored) noise.

The new method contributes valuable insights into applying entropy probabilistic measures, providing a foundation for future studies in terms of extended complex system pattern formation characterization.

Initial work considering entropy measurements for training machine learning models is underway. In this context, it also includes a study of the computational complexity of the method for a benchmark with other measures and approaches that may emerge. This strategy is fundamental when we think about the presented method being applied in a data science context.

Finally, we carefully organize all the mathematical content and data used in this work in a GitHub repository4 to guarantee the reproducibility of this experiment.