Rényi Entropy Estimation via Affine Transformations of Generalized Means

1. Introduction

1.1. From Probability Theory to Information Theory

Probability theory plays a central role in modeling complex systems that operate under uncertainty. From Cardano’s early reflections on dice games of chance (De Ludo Aleae, ~1564) [1] and the correspondence between Fermat and Pascal on the problem posed by Pacioli (1654) [2], to Kolmogorov’s axiomatic formalization (1933) [3], the discipline of probability theory has gradually evolved from its heuristic origins into a mathematically rigorous and essential framework. Subsequently, information theory enabled a deeper exploration of the structure of probability distributions by using the concept of information entropy. Drawing inspiration from Boltzmann’s thermodynamic formulation, Shannon in 1948 [4] defined the entropy of a discrete probability distribution $\mathit{p}$ as

$$H_S\left(\mathit{p}\right)=-\sum _{k=1}^n{p}_{k}log{p}_k$$

(1)

where $p_k$ denotes the probability of the $k^{\mathrm{th}}$ event, and n is the number of possible outcomes. However, this fundamental quantity does not always distinguish between distributions with different internal structures. To address this limitation, Rényi in 1961 [5] introduced a parametric family of entropy measures, generalizing Shannon’s definition:

$$H_{\alpha }\left(\mathit{p}\right)={\displaystyle \frac{1}{1-\alpha }}log\sum _{k=1}^n{p}_k^{\alpha }\alpha >0,\alpha \ne 1.$$

(2)

As $\alpha \to 1$, the Rényi entropy converges to the Shannon entropy, preserving theoretical consistency. Varying $\alpha$, the measure emphasizes different parts of the distribution, enabling finer structural analysis of probability distributions.

1.2. The Challenge of Accurate Entropy Estimation

Rényi’s axiomatic approach offered a flexible generalization with theoretical applications in communication theory [6,7], statistical mechanics [8,9], cryptography [10,11] and artificial intelligence [12,13]. However, estimating Rényi entropies from empirical data is difficult due to structural and statistical factors that compromise result reliability [14,15,16], including:

(a) Zero-frequency problem: rare events may be entirely absent from finite samples, leading to underestimated entropy—especially in heavy-tailed or sparse distributions;

(b) Paradox of data sparsity: as data become sparser in the sample space, the chance of observing all outcomes in correct relative proportions declines, paradoxically shrinking the empirical entropy even as statistical unreliability increases;

(c) Amplification of errors: Entropy’s nonlinear dependence on probabilities means that small empirical errors can be magnified, yielding high variance and instability;

(d) Curse of dimensionality: as the dimensionality increases, the event space grows exponentially, making data sparse and many outcomes rare or unobserved;

(e) Sensitivity to outliers and noise: anomalous events or random fluctuations can distort frequencies and cause erratic entropy values.

These challenges are particularly acute in small-sample regimes, high-dimensional settings, and data derived from high-entropy processes—all common in modern data analysis. As a result, even well-established estimation techniques often produce biased or unstable results.

1.3. Related Works and Previous Contributions

The estimation of members of the family of Rényi entropies—including Shannon entropy ($\alpha \to 1$), collision entropy ($\alpha =2$), and min-entropy ($\alpha \to \infty$)—has been widely investigated and has garnered increasing attention due to its relevance in various applicative fields. Immediately after the invention of these indices, it was soon recognized that their naive plug-in estimators suffer from high bias and variance in small-sample or high-dimensional regimes. Later works proposed corrections and non-parametric models [17,18], though these remained limited in effectiveness. In response to these challenges, more efforts have progressively been directed toward developing increasingly sophisticated estimation methods: [10,15,16,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89]. Despite these improvements, an universally recognized method capable of expressing optimal performance in any situation has not yet been achieved. Thus, in the continued absence of a definitive and unique procedure, the pursuit of a robust general-purpose estimator remains an open and central challenge in the study of Rényi entropies. It is also worth noting that, due to these difficulties, many researchers have proposed alternative indices, equipped with their customized estimators, such as Onicescu’s informational energy [90], approximate entropy [91], sample entropy [92], multiscale entropy [93], permutation entropy [94] and bubble entropy [95]. Beyond these isolated alternatives, one of the few comprehensive frameworks comparable in scope to the Rényi entropy family is the parametric family of Tsallis entropies [96], whose applicability is hindered by the lack of key properties such as additivity, scale invariance, and ease of normalization.

1.4. Motivation and Contribution of This Paper

This work is motivated by the persistent lack of a unified and widely accepted method for estimating Rényi entropies—a gap that has remained open for more than sixty years since their theoretical inception. We propose a novel method for the accurate estimation of Rényi entropies that requires minimal parameter tuning and can be used under diverse data conditions. The method effectively addresses the challenges mentioned in the previous paragraph. Empirical evidence confirms its performance in low-sample, high-dimensional, and high-entropy settings. At the core of the procedure lies an affine transformation1 applied to the empirical frequency distribution, introducing a structured geometric regularization whose overall effect is a global linear stretching. This transformation regularizes the geometry of the input space prior to the entropy computation, leading to an improved approximation of the true Rényi entropy via the Rényi mean2. This approach stems from the idea that the empirical entropies reside in a context that is different from the context of theoretical entropies and a transformation is necessary for the “translation” of these measures from one context to the other. The proposed transformation reorients and rescales the data in a principled manner, considerably reducing the estimation error for any entropy index and directly reaching the optimality for the case of $\alpha =2$ (collision entropy).

2. Overview on Materials and Methods of Information Theory

2.1. Transforming a Discrete-State Stochastic Process into a Probability Distribution

Consider a discrete-state stochastic process ($DSP$) whose sequence of infinite samples $s_{-\infty },\dots ,s_{i-1},s_i,s_{i+1},\dots ,s_{\infty }$ presents values belonging to an alphabet ${\mathit{X}}_q$ containing q ordered symbols. Let $\Omega (q,d)$ be a d-dimensional discrete sample space resulting from the Cartesian product d times of ${\mathit{X}}_q$.

$$\Omega (q,d)=\underset{dtimes}{\underbrace{{\mathit{X}}_q\times {\mathit{X}}_q\times ...\times {\mathit{X}}_q}},$$

(3)

and let $n=q^d$ be the cardinality of $\Omega (q,d)$. Each elementary event ${\mathit{e}}_k$, with $k\in \{1,2,...,n\}$, is uniquely identified by a vector with d coordinates ($x{1}_k,x{2}_k,...,x{d}_k$), with $x{1}_k,x{2}_k,...,x{d}_k\in X_q$. According to the procedure indicated by Shannon in [4] at pages 5 and 6, the infinite sequence of samples constituting the $DSP$ can be transformed into occurrences $\#\left({\mathit{e}}_k\right)$ of the elementary events of $\Omega (q,d)$ by progressively considering all the d-grams taken from the samples as if they were the vector coordinates of the events and counting the number of times each vector appears in the sequence. Then, according to the frequentist definition of probability, the final resulting probability distribution is expressible in set theory notation as

$$\mathit{p}{(\Omega (q,d))}_{DSP}=\left\{p{\left({\mathit{e}}_k\right)}_{DSP}={\displaystyle \frac{\#{\left({\mathit{e}}_k\right)}_{DSP}}{{\sum }_{k=1}^n\#{\left({\mathit{e}}_k\right)}_{DSP}}}|{\mathit{e}}_k\in \Omega (q,d)\right\}.$$

(4)

In the following, in the absence of ambiguity, $\mathit{p}{(\Omega (q,d))}_{DSP}$—that is, a $DPD$ obtained by elaborating the data of a $DSP$— will be indicated with the bold symbol $\mathit{p}$ and the probability of the $k^{th}$ event with $p_k$.

2.2. Integer-Order Rényi $\alpha$-Entropies as Synthetic Indices for the Characterization of $DPD$s

The family of integer-order Rényi $\alpha$-entropies offers synthetic indices that characterize $DPD$s in terms of their informative content [5]. They are defined as:

$$\begin{array}{c}\hfill \alpha =1H_1\left(\mathit{p}\right)\triangleq -\sum _{k=1}^n{p}_k\log p_k\\ \hfill \alpha \in {\mathbb{N}}^{+}\alpha \ne 1H_{\alpha }\left(\mathit{p}\right)\triangleq {\displaystyle \frac{1}{1-\alpha }}\log \left(\sum _{k=1}^n{p}_k^{\alpha }\right)0\le H_{\alpha }\left(\mathit{p}\right)\le \log n\\ \hfill \overrightarrow{\alpha }\infty H_{\infty }\left(\mathit{p}\right)\triangleq -\log max\left\{\mathit{p}\right\}.\end{array}$$

(5)

The corresponding specific entropies are then defined as

$$\begin{array}{c}\hfill \alpha =1{\eta }_1\left(\mathit{p}\right)\triangleq {\displaystyle \frac{H_1\left(\mathit{p}\right)}{\log n}}=-\sum _{k=1}^n{p}_k{log}_n{p}_k\\ \hfill \alpha \in {\mathbb{N}}^{+}\alpha \ne 1{\eta }_{\alpha }\left(\mathit{p}\right)\triangleq {\displaystyle \frac{H_{\alpha }\left(\mathit{p}\right)}{\log n}}={\displaystyle \frac{1}{1-\alpha }}{log}_n\left(\sum _{k=1}^n{p}_k^{\alpha }\right)0\le {\eta }_{\alpha }\left(\mathit{p}\right)\le 1\\ \hfill \overrightarrow{\alpha }\infty {\eta }_{\infty }\left(\mathit{p}\right)\triangleq {\displaystyle \frac{H_{\infty }\left(\mathit{p}\right)}{\log n}}=-{log}_{n}max\left\{\mathit{p}\right\}.\end{array}$$

(6)

Specific entropies are preferred over plain ones because:

(a) They range in $[0,1]$ via min-max normalization, which provides a unique bounded scale that greatly facilitates the interpretation of entropy values at any scale;

(b) They minimize the dependence on the sample space size n;

(c) They resolve doubts about the choice of the logarithm base, highlighting scale invariance;

(d) Entropies in base b are then given by $H_{\alpha }(\mathit{p},b,n)={\eta }_{\alpha }\left(\mathit{p}\right)·{log}_{b}n$.

2.3. Rényi Entropy Rates

Unlike Rényi entropies, whose utility is mainly related to the classification of $DPD$s, Rényi entropy rates are important theoretical quantities useful for the characterization of $DSP$s [61,72]; they are defined as

$$H_{\alpha }^{\prime }\left(DSP\right)\triangleq \underset{d\to \infty }{lim}{\displaystyle \frac{1}{d}}{H}_{\alpha }\left(\mathit{p}{(\Omega (q,d))}_{DSP}\right)0\le H_{\alpha }^{\prime }\left(DSP\right)\le logq.$$

(7)

Moreover, it is known that, for strongly stationary $DSP$, any Rényi entropy rate converges to the same limit of a sequence of Cesaro means of conditional entropies:

$$H_{\alpha }^{\prime }\left(DSP\right)=\underset{d\to \infty }{lim}{H}_{\alpha }\left(\mathit{p}\left(A_d\right)\right|\mathit{p}(A_1\times A_2\times \dots \times A_{d-1})).$$

(8)

and, as conditional Rényi entropies preserve the chain rule [97,98,99], they can also be calculated as

$$H_{\alpha }^{\prime }\left(DSP\right)=\underset{d\to \infty }{lim}\left[H_{\alpha }\left(\mathit{p}{(\Omega (q,d))}_{DSP}\right)-H_{\alpha }\left(\mathit{p}{(\Omega (q,d-1))}_{DSP}\right)\right].$$

(9)

Specific Rényi entropy rates are defined by the following min-max normalization:

$$\begin{array}{c}\hfill {\eta }_{\alpha }^{\prime }\left(DSP\right)={\displaystyle \frac{H_{\alpha }^{\prime }\left(DSP\right)}{\log q}}=\underset{d\to \infty }{lim}{\displaystyle \frac{\left[H_{\alpha }\left(\mathit{p}{(\Omega (q,d))}_{DSP}\right)-H_{\alpha }\left(\mathit{p}{(\Omega (q,d-1))}_{DSP}\right)\right]}{\log q}}=\\ \hfill =\underset{d\to \infty }{lim}\left[d{\eta }_{\alpha }\left(\mathit{p}{(\Omega (q,d))}_{DSP}\right)-(d-1){\eta }_{\alpha }\left(\mathit{p}{(\Omega (q,d-1))}_{DSP}\right)\right]\end{array}$$

(10)

with $0\le {\eta }_{\alpha }^{\prime }\left(DSP\right)\le 1$.

2.4. Relationship Between Specific Rényi Entropy Rate and Specific Rényi Entropy

The following relationship holds:

$${\eta }_{\alpha }^{\prime }\left(DSP\right)=\underset{d\to \infty }{lim}{\eta }_{\alpha }\left(\mathit{p}{(\Omega (q,d))}_{DSP}\right)$$

(11)

Accordingly, the specific Rényi entropy rate ${\eta }_{\alpha }^{\prime }\left(DSP\right)$ can be computed via either (10) or (11), with the former being preferable due to faster convergence.

2.5. Rényi Generalized Dimensions

Rényi entropies and their associated rates provide a unified framework for analyzing probability distributions and classifying stochastic processes. A closely related concept in multifractal analysis is the Rényi generalized dimension [100], defined as

$$D_{\alpha }(\Theta (d,ϵ))={\displaystyle \frac{1}{\alpha -1}}·{\displaystyle \frac{log\left({\sum }_i{p}_i{(\Theta (d,ϵ))}^{\alpha }\right)}{logϵ}},$$

(12)

where $\Theta (d,ϵ)$ denotes a partition of the state space into d-dimensional cells of linear size $ϵ$, and $p_i(\Theta (d,ϵ))$ is the probability that a trajectory of the dynamical system ($DS$) crosses the i-th cell:

$$p_i(\Theta (d,ϵ))={\displaystyle \frac{\mu \left({\mathrm{cell}}_i\right)}{{\sum }_{k=1}^n\mu \left({\mathrm{cell}}_k\right)}},\sum _{i=1}^n{p}_i(\Theta (d,ϵ))=1,$$

with $\mu \left({\mathrm{cell}}_i\right)$ denoting the number of visits in the i-th d-cell by the trajectory.

2.6. Rényi Generalized Dimension Rates

In analogy with entropy rates, we define the Rényi generalized dimension rate as

$$D_{\alpha }^{\prime }\left(DS\right)=\underset{ϵ\to 0}{lim}{\displaystyle \frac{1}{\alpha -1}}·{\displaystyle \frac{log\left({\sum }_i{p}_i{(\Theta (d,ϵ))}^{\alpha }\right)}{logϵ}}=\underset{ϵ\to 0}{lim}{\displaystyle \frac{H_{\alpha }\left(\mathit{p}\right)}{log(1/ϵ)}},$$

(13)

where $H_{\alpha }\left(\mathit{p}\right)$ is the Rényi entropy of order $\alpha$ associated with the distribution $\mathit{p}=\left\{p_i\right\}$ induced by the partition.

Letting $q=1/ϵ$, the number of partition elements scales as $n=q^d$. Since the specific Rényi entropy is defined as ${\eta }_{\alpha }\left(\mathit{p}\right)=H_{\alpha }\left(\mathit{p}\right)/logn$, the dimension rate becomes

$$D_{\alpha }^{\prime }\left(DS\right)=\underset{q\to \infty }{lim}{\displaystyle \frac{{\eta }_{\alpha }\left(\mathit{p}\right)·logn}{logq}}=d·\underset{q\to \infty }{lim}{\eta }_{\alpha }\left(\mathit{p}\right),0\le D_{\alpha }^{\prime }\left(DS\right)\le d.$$

(14)

Here, d is fixed, while q increases. To improve convergence and enable stable numerical estimation, we employ a differential formulation based on logarithmic increments:

$$D_{\alpha }^{\prime }\left(DS\right)=d·\underset{q\to \infty }{lim}{\displaystyle \frac{{\eta }_{\alpha }\left(\mathit{p}(\Omega (q,d))\right)·logq-{\eta }_{\alpha }\left(\mathit{p}(\Omega (q-1,d))\right)·log(q-1)}{logq-log(q-1)}}$$

(15)

2.7. Rényi Mutual Information

Another key quantity derived from $\alpha$-Rényi entropies is the $\alpha$-Rényi mutual information [48], defined for three random variables X, Y, and $(X\times Y)$, with respective events $\left\{x_k\right\}$, $\left\{y_i\right\}$, and $\left\{(x_k,y_i)\right\}$, and corresponding probability distributions $\mathit{p}$, $\mathit{t}$, and $\mathit{u}$:

$$I_{\alpha }(X;Y)\triangleq \sum _{k=1}^n\sum _{i=1}^m{u}_{ki}log{\displaystyle \frac{u_{ki}}{p_k{t}_i}}=H_{\alpha }\left(\mathit{p}\left(X\right)\right)+H_{\alpha }\left(\mathit{t}\left(Y\right)\right)-H_{\alpha }\left(\mathit{u}(X\times Y)\right)$$

(16)

The corresponding expression in terms of specific entropy is:

$${\iota }_{\alpha }(X;Y)\triangleq \sum _{k=1}^n\sum _{i=1}^m{u}_{ki}{log}_n{\displaystyle \frac{u_{ki}}{p_k{t}_i}}={\eta }_{\alpha }\left(\mathit{p}\left(X\right)\right)+{\eta }_{\alpha }\left(\mathit{t}\left(Y\right)\right)-{\eta }_{\alpha }\left(\mathit{u}(X\times Y)\right).$$

(17)

2.8. Measures of Entropic Distance Between Distributions

Dissimilarity between two distributions A and B is often measured using Rényi divergences [101], of which the Kullback–Leibler divergence [102] (also known as relative entropy) is a special case. Since these divergences are not directly derived from Rényi entropies, we instead propose to measure their geometric distance in entropy space using the following formula:

$$D_{\eta }\left(\overline{\mathit{AB}}\right)=\sqrt{{\displaystyle \frac{{\sum }_{\alpha }{({\eta }_{\alpha }\left(\mathit{A}\right)-{\eta }_{\alpha }\left(\mathit{B}\right))}^2}{\#{\eta }_{\alpha }}}},0\le D_{\eta }\left(\overline{\mathit{AB}}\right)\le 1.$$

(18)

where $\#{\eta }_{\alpha }$ represents the number of entropy indices used to determine the state. This method provides a symmetric and broader generalization with respect to Rényi divergences, also allowing the comparison of distributions with different event-space cardinalities.

2.9. Two Fundamental Families of Structured Probability Distributions

To support controlled and interpretable experimentation, we define two fundamental families of structured probability distributions that form the basis of our experimental analysis. Designed to span the entire range of distributions over a finite alphabet, these families exhibit distinct structural properties that represent entropy extremes within the distribution space (see Figure 1 and Figure 2). They are:

1.

Mono Dominant with Uniform Residues (MD_UR): This family generates distributions in which, starting from the uniform case, a single symbol progressively absorbs more probability mass by subtracting it uniformly from the other symbols. The family can be defined as:

$$p_1={\displaystyle \frac{1}{q}}+\theta \left(1-{\displaystyle \frac{1}{q}}\right)$$

$$p_i={\displaystyle \frac{1-p_1}{q-1}}\mathrm{for}i=2,\dots ,q$$

where the parameter $\theta \in [0,1]$ controls the strength of the dominant symbol.

2.

Uniform Block Dominant with Mono Residue (UBD_MR): This family defines structured, biased distributions starting from a uniform base. Probability mass is progressively removed from the last non-zero event and equally redistributed among all preceding ones. The resulting distribution exhibits a dominant block of equal, high-probability values at the front, followed by a single lower residual value, with all subsequent entries set to zero. This construction induces a non-trivial truncation of the support and does not admit a simple closed-form expression.

For both families, explicit generation algorithms are provided to support reproducibility and controlled experimentation. These distributions define the entropic bounds that encompass all admissible shapes, from highly concentrated to nearly uniform. They are well suited to reveal how entropy estimators behave under structural constraints and will be used to benchmark performance, assess sensitivity, and identify systematic biases.

2.10. Ambiguity Reduction in Entropy-Based Measures

While a single entropy index provides valuable information, it may be insufficient to distinguish between probability distributions that share the same entropy value but differ in structural characteristics. This limitation can be addressed by combining multiple independent and complementary entropy indices—such as different Rényi entropies (e.g., Shannon entropy and collision entropy)—which enhance the ability to differentiate among distributions. For instance, for two distributions with identical Shannon entropy, the further observation of their collision entropy reveals structural differences that would otherwise remain hidden (see Figure 3).

2.11. Interpreting Rényi Entropies as the Negative Logarithm of Generalized Means

The generalized α-Hölder mean[103] of a discrete probability distribution $\mathit{p}$ over n elementary events is defined as:

$${}^H\overline{\mathit{p}}_{\alpha }\triangleq {\left({\displaystyle \frac{1}{n}}\sum _{i=1}^n{p}_i^{\alpha }\right)}^{{\displaystyle \frac{1}{\alpha }}},{\displaystyle \frac{1}{n}}\le {}^H\overline{\mathit{p}}_{\alpha }\le 1$$

(19)

This mean interpolates between classical means as $\alpha$ varies: in fact ${}^H\overline{\mathit{p}}_{-1}$ is the harmonic mean, ${}^H\overline{\mathit{p}}_0$ is the geometric mean, ${}^H\overline{\mathit{p}}_1$ is the arithmetic mean, ${}^H\overline{\mathit{p}}_2$ is the root mean square, ${}^H\overline{\mathit{p}}_{\infty }$ is the mean considering the maximum value. Building on this idea, we define the generalized α-Rényi mean of a discrete probability distribution $\mathit{p}$ as:

$${}^R\overline{\mathit{p}}_{\alpha }\triangleq {\left(\sum _{k=1}^n{p}_k^{\alpha }\right)}^{{\displaystyle \frac{1}{\alpha -1}}},{\displaystyle \frac{1}{n}}\le {}^R\overline{\mathit{p}}_{\alpha }\le 1$$

(20)

Accordingly, the Rényi entropy of order $\alpha$ can be expressed as:

$${\eta }_{\alpha }\left(\mathit{p}\right)=-{log}_n\left({}^R\overline{\mathit{p}}_{\alpha }\right)⟺{}^R\overline{\mathit{p}}_{\alpha }=n^{-{\eta }_{\alpha }\left(\mathit{p}\right)}$$

(21)

Each order $\alpha$ corresponds to a distinct generalized averaging scheme, providing insight into the distribution’s internal structure from complementary informational perspectives. This framework also recovers several well-known entropy measures as special cases:

Shannon entropy ($\alpha \to 1$) from the Shannon mean:

$${\eta }_1\left(\mathit{p}\right)=-{log}_n\left({}^R\overline{\mathit{p}}_1\right),{}^R\overline{\mathit{p}}_1\triangleq \prod _{k=1}^n{p}_k^{p_k}\left(\mathrm{if}{p}_k=0⟹p_k^{p_k}=1\right)$$

(22)

Collision entropy ($\alpha =2$) from the collision mean:

$${\eta }_2\left(\mathit{p}\right)=-{log}_n\left({}^R\overline{\mathit{p}}_2\right),{}^R\overline{\mathit{p}}_2\triangleq \sum _{k=1}^n{p}_k^2$$

(23)

Min-entropy ($\alpha \to \infty$) from the max-mean, equal to the maximum probability:

$${\eta }_{\infty }\left(\mathit{p}\right)=-{log}_n\left({}^R\overline{\mathit{p}}_{\infty }\right),{}^R\overline{\mathit{p}}_{\infty }\triangleq p_1\mathrm{with}{p}_1\ge \dots \ge p_n$$

(24)

From this perspective, each specific Rényi entropy can be interpreted as the negative logarithm (base n) applied to the corresponding generalized Rényi mean, resulting in a normalized, scale-invariant measure of an intrinsic property of the probability distribution.

3. Practical Implementation of Methods of Information Theory

This section addresses the transition from idealized models—based on exact knowledge of discrete probability distributions—to practical scenarios involving finite samples, where the true underlying distributions are unknown. By reformulating the theoretical framework in terms of observed relative frequencies, it enables inference and estimation of properties of the underlying generative process. This procedure is composed of three steps: (a) Determining relative frequencies from samples;

(b) Calculating apparent Rényi means/entropies of the relative frequencies;

(c) Inferring the actual Rényi means/entropies of the underlying probability distribution.

3.1. Converting a Realization into a Relative Frequency Distribution

In practical applications, the theoretical procedure described in § Section 2.1 can be adapted as follows: consider the L samples $s_1,s_2,\dots ,s_d,s_{d+1},\dots ,s_L$ of a realization r, extracted from a $DSP$ represented using q ordered symbols. Each d-gram, consisting of d consecutive samples from r, is treated as the occurrence of an elementary event in the d-dimensional sample space $\Omega (q,d)$, where each d-gram defines a unique vector in this space. For example, the first two d-grams taken from $r_q$, $(x_1,x_2,...,x_d)$ and $(x_2,x_3,...,x_{d+1})$ identify the first occurrences of two elementary events. The count of the occurrences of events is performed for all the d-grams progressively identified in the sequence of the samples of r. The absolute frequency of each elementary event $\#\left({\mathit{e}}_k\right)$ is normalized by the total number of occurrences, ($N={\sum }_{k=1}^n\#{\left(e_k\right)}_r=L-d+1$), yielding the corresponding relative frequency $f{\left({\mathit{e}}_k\right)}_r$. The final resulting $RFD$ is expressible in set theory notation as

$$\mathit{f}{(\Omega (q,d))}_r=\left\{f{\left({\mathit{e}}_k\right)}_r={\displaystyle \frac{\#{\left({\mathit{e}}_k\right)}_r}{{\sum }_{k=1}^n\#{\left({\mathit{e}}_k\right)}_r}}|{\mathit{e}}_k\in \Omega (q,d)\right\}$$

(25)

where ${\sum }_{k=1}^n\#{\left({\mathit{e}}_k\right)}_r$ represents the total number of observed d-grams. For brevity, when there is no risk of ambiguity, an RFD $\mathit{f}{(\Omega (q,d))}_r$ derived from a realization within a given sample space will be denoted simply as $\mathit{f}$, and $f_k$ represents the relative frequency attributed to the $k^{th}$ event.

3.2. Rényi Mean and Specific Rényi Entropy of a Relative Frequency Distribution

For a relative frequency distribution, the empirical Rényi mean is defined as:

$${}^R\overline{\mathit{f}}_{\alpha }={\left(\sum _{k=1}^n{f}_k^{\alpha }\right)}^{{\displaystyle \frac{1}{\alpha -1}}}\left({\displaystyle \frac{1}{N}}\le {}^R\overline{\mathit{f}}_{\alpha }\le 1\right)$$

(26)

Consequently, the empirical specific Rényi entropy results

$${\eta }_{\alpha }\left(\mathit{f}\right)=-{log}_n\left({}^R\overline{\mathit{f}}_{\alpha }\right)⟺{}^R\overline{\mathit{f}}_{\alpha }=n^{-{\eta }_{\alpha }\left(\mathit{f}\right)}$$

(27)

3.3. Apparent vs. Actual Specific Entropies in Controlled Probabilistic Processes

Figure 4 and Figure 5 show the result of the comparison of theoretical and empirical specific entropies for the two previously introduced limit families of discrete probability distributions, MD_UR and UBD_MR. Three theoretical indices, ${\eta }_1\left(\mathit{p}\right)$, ${\eta }_2\left(\mathit{p}\right)$, and ${\eta }_{\infty }\left(\mathit{p}\right)$, are placed in relationship each with its empirical counterpart, respectively $\overline{{\eta }_1}\left(\mathit{f}\right)$, $\overline{{\eta }_2}\left(\mathit{f}\right)$, and $\overline{{\eta }_{\infty }}\left(\mathit{f}\right)$, calculated from relative frequencies. Each family includes 100 synthetic processes, generated by progressively deforming a fair die distribution ($q=6,p_1=..=p_6=1/6$) into a degenerate one ($p_1=1$), with a distortion percentage increasing from $0\%$ to $100\%$ along its entropic path. For each theoretical distribution, $r=20,000$ realizations (each $L=254$ samples long) are produced, and $N=250$ event samples are drawn in a $d=5$ dimensional space $\Omega$ ($n=6^5=7776$), resulting in a sparse data regime (data density $\delta =N/n=3.21\%$). As distortion grows, both true and empirical entropies decrease, but empirical estimates systematically underestimate true values, especially near uniformity. Saturation sets in beyond the threshold $\eta {\left(\mathit{f}\right)}_{\mathrm{Lim}}={log}_{n}N\approx 0.616$, highlighted in the figures by a vertical dashed line, where the empirical entropy is restricted by the limited number of observed data. Despite construction differences, both families exhibit similar transitions from uniformity to concentration, highlighting common limitations of entropy estimation under finite sampling. Thus, these models provide a controlled framework for studying the interaction between distribution structure, sparsity, and estimation reliability, providing insights for high-dimensional inference.

3.4. Determination of the Relationship Between Actual and Apparent Rényi Mean

To enable a meaningful comparison between actual Rényi mean (based on the true distribution $\mathit{p}$) and apparent Rényi mean (computed from empirical frequencies $\mathit{f}$), it is convenient to apply a min-max normalization to both quantities using their respective minimal and maximal attainable values. These transformations give two comparable quantities, both ranging in $[0,1]$ interval:

$$0\le {\displaystyle \frac{{}^R\overline{\mathit{p}}_{\alpha }-\frac{1}{n}}{1-\frac{1}{n}}}\le 10\le {\displaystyle \frac{{}^R\overline{\mathit{f}}_{\alpha }-\frac{1}{N}}{1-\frac{1}{N}}}\le 1$$

(28)

The second normalization assumes that when the true entropy is maximal—that is, when ${\eta }_{\alpha }\left(\mathit{p}\right)=1$—the apparent Rényi mean ${}^R\overline{\mathit{f}}_{\alpha }$ reaches its minimum value of exactly ${\displaystyle \frac{1}{N}}$. However, empirical evidence—as shown in Figure 4, Figure 5—suggests otherwise: the minimum value of the apparent Rényi mean ${}^R\overline{\mathit{f}}_{\alpha }$ under maximum entropy conditions can deviate from ${\displaystyle \frac{1}{N}}$ and this deviation is maximal for $\alpha \to \infty$.

To account for this discrepancy, we introduce the term ${\overline{m}}_{\alpha }(N,q,d)$, which represents the empirical lower bound of the $\alpha$-Rényi mean. It is computed as the average value of the Rényi mean over multiple realizations drawn from a perfectly uniform process, governed by a perfectly uniform distribution (U)—i.e., one with maximal specific entropy—under the same sampling conditions $(N,q,d)$.

This empirically refined mean minimum, specific to each triplet $(N,q,d)$, is defined as:

$${\overline{m}}_{\alpha }(N,q,d)\triangleq {〈{}^R\overline{\mathit{f}}_{\alpha }〉}_{r\left(U\right)}$$

(29)

Then, as in general ${\overline{m}}_{\alpha }\le {}^R\overline{\mathit{f}}_{\alpha }\le 1$, the relationship between the empirical Rényi mean ${}^R\overline{f}_{\alpha }$ and the true Rényi mean ${}^R\overline{p}_{\alpha }$ can be described as:

$${\displaystyle \frac{\widehat{{}^R\overline{\mathit{p}}_{\alpha }}-\frac{1}{n}}{1-\frac{1}{n}}}={\displaystyle \frac{{〈{}^R\overline{\mathit{f}}_{\alpha }〉}_{\mathit{r}}-{\overline{m}}_{\alpha }}{1-{\overline{m}}_{\alpha }}}$$

(30)

Solving for $\widehat{{}^R\overline{\mathit{p}}_{\alpha }}$, we obtain:

$$\widehat{{}^R\overline{\mathit{p}}_{\alpha }}=\left({\displaystyle \frac{1-\frac{1}{n}}{1-{\overline{m}}_{\alpha }}}\right)·{〈{}^R\overline{\mathit{f}}_{\alpha }〉}_{\mathit{r}}+\left({\displaystyle \frac{1}{n}}-{\displaystyle \frac{1-\frac{1}{n}}{1-{\overline{m}}_{\alpha }}}·{\overline{m}}_{\alpha }\right)$$

(31)

This expression reveals that the first-order approximation of any Rényi mean can be obtained by applying a linear scaling and an offset to the empirical Rényi mean. Explicitly, this is a deterministic affine transformation of the form:

$$\widehat{{}^R\overline{\mathit{p}}_{\alpha }}=A·{〈{}^R\overline{\mathit{f}}_{\alpha }〉}_{\mathit{r}}+B$$

(32)

where:

• $A={\displaystyle \frac{1-\frac{1}{n}}{1-{\overline{m}}_{\alpha }}}$ is the scaling factor;
• $B={\displaystyle \frac{1}{n}}-A·{\overline{m}}_{\alpha }$ is the translation offset;

3.5. Toward Accurate Estimation of Rényi Entropies via Affine Transformations

The affine structure of (32) suggests a possible interpretation of the estimator: it maps the apparent internal state of the distribution (observed empirically from relative frequencies) into its actual internal state (in the theoretical context of probabiity distributions). The Rényi entropy itself (${\eta }_{\alpha }\left(\mathit{p}\right)=-{log}_n\left({}^R\overline{\mathit{p}}_{\alpha }\right)$) can thus be interpreted as an observable readout of this internal state. Based on equation (31), the estimated specific Rényi entropy is:

$$\begin{array}{c}\hfill \widehat{{\eta }_{\alpha }}\left(\mathit{p}\right)=-{log}_n\left[\widehat{{}^R\overline{\mathit{p}}_{\alpha }}\right]=-{log}_n\left[{\displaystyle \frac{1-\frac{1}{n}}{1-{\overline{m}}_{\alpha }}}\left({〈{}^R\overline{\mathit{f}}_{\alpha }〉}_{\mathit{r}}-{\overline{m}}_{\alpha }\right)+{\displaystyle \frac{1}{n}}\right]=\\ \hfill =-{log}_n\left[{\displaystyle \frac{1-\frac{1}{n}}{1-{\overline{m}}_{\alpha }}}\left(n^{-\widehat{{\eta }_{\alpha }}\left(\mathit{f}\right)}-{\overline{m}}_{\alpha }\right)+{\displaystyle \frac{1}{n}}\right]\end{array}$$

(33)

This expression holds for all $\alpha \in {\mathbb{R}}^{+}$, including limiting cases such as Shannon entropy ($\alpha \to 1$) and min-entropy ($\alpha \to \infty$). This approach yields a non-iterative, scalable, and theoretically grounded estimator. Its derivation is based on:

• Min-max normalization of actual and empirical Rényi means, to ensure comparability;
• A linear transformation between the two domains;
• The logarithmic relationship between Rényi entropy and Rényi mean.

By integrating these elements, the estimator provides a principled way to recover the entropy of the underlying process from observed frequencies, based on the following properties:

1.

Change of paradigm: data do not need correction, but they have to be rescaled from an empirical to a theoretical context.

2.

Consistency: as $N\to \infty$, we have ${\overline{m}}_{\alpha }\to 1/n$ and $\mathit{f}\to \mathit{p}$, ensuring that $\widehat{{\eta }_{\alpha }}\left(\mathit{p}\right)$ converges to the true entropy ${\eta }_{\alpha }\left(\mathit{p}\right)$.

3.

Bias and variance control: the estimator compensates for the known bias in empirical frequencies by incorporating a prior empirical mean ${\overline{m}}_{\alpha }$, resulting in a lower mean squared error compared to classical estimators.

4.

Computational efficiency: the formula requires only a few arithmetic operations and, optionally, a numerically precomputed lookup table for ${\overline{m}}_{\alpha }(q,d,N)$, making it suitable for large-scale inference.

The left column of Figure 6 consolidates the results previously shown in the left panels of Figure 4 and Figure 5, reorganizing the same data into separate plots by entropy type: specific Shannon entropy ${\eta }_1$ at the top, specific collision entropy ${\eta }_2$ in the middle, and specific min-entropy ${\eta }_{\infty }$ at the bottom. For each entropy type, the results obtained from the MD_UR and UBD_MR distribution families define a region that encloses all points achievable by applying the same procedures to any probability distribution under identical sampling conditions ($N=L-d+1$) and sample space structure ($n=q^d$). The right column of Figure 6 illustrates the effect of applying formula  (33) to the data, which induces a geometric transformation that diagonally stretches the region representing the relationship between the entropy of theoretical and empirical distributions. In the specific case of collision entropy, this transformation causes the curves associated with all distribution families—from MD_UR to UBD_MR—to nearly coincide with the diagonal line (see central panel in Figure 6). As a result, this transformation alone completes the estimation procedure for specific collision entropy. In contrast, the estimation of other Rényi entropies, such as Shannon and min-entropy, requires an additional adjustment step to reach comparable accuracy.

As a result, the specific collision entropy emerges as the most coherent index. Moreover, its optimal estimation can be performed with a single transformation. For this reason, it will serve as the horizontal reference axis in subsequent diagrams, facilitating comparative assessments of the estimators of other entropies. Using this approach, the data from Figure 6 can be recomposed by plotting the specific collision entropy ${\eta }_2$ on the X-axis and mapping ${\eta }_1$ on the left Y-axis and ${\eta }_{\infty }$ on the right Y-axis of the diagram, respectively. The outcome of this new composition is shown in Figure 7.

3.6. A Second Transformation for More Accurate Estimation of Entropies: Orthogonal Stretch

While the diagonal stretch is sufficient for accurately estimating the specific collision entropy, it proves inadequate for other Rényi entropies. As illustrated in Figure 7, this affine transformation alone fails to fully map the region derived from empirical data onto the corresponding region defined by theoretical probability distributions. To overcome this limitation, a second transformation—orthogonal to the direction associated with the collision mean—is required to complete the geometric alignment. A detailed analysis of this additional step, which represents a further advancement toward accurate Rényi entropy estimation, lies beyond the scope of the present work, which primarily serves as an introduction to this novel approach.

3.7. A Practical Example of Applying the First Order Affine Transformation

In this example, we demonstrate how the entropic state of a $DPD$ can be recovered from a set of short realizations sampled from the process defined by the original $DPD$. The recovery is carried out by applying the previously described first-order affine transformation to the generalized means of empirical data distributions. Its procedure consists of the following steps:

1.

Choice of the stochastic process: In this case, without loss of generality, and following the foundational approach adopted by early pioneers of probability theory (e.g., Cardano, Galileo, Pascal, Fermat, Huygens, Bernoulli, de Moivre, Newton), we consider a memoryless stochastic process generated by repeated rolls of a six-sided die subject to a specific probability distribution. The key advantage of using memoryless processes is that their entropic state remains invariant under changes in the dimensionality of the sample space used for statistical analysis.

2.

Choice of the theoretical $DPD$: We select a distribution whose entropic characteristics lie far from the regions typically covered by empirical distributions derived from small samples, thereby making the recovery task more challenging, for instance:

$$T=\{0.36,0.2,0.11,0.11,0.11,0.11\}$$

3.

Choice of the composition of the entropic space: We consider a three dimensional entropic space. The entropies are: specific Shannon entropy (${\eta }_1$), specific collision entropy (${\eta }_2$), and specific min-entropy (${\eta }_{\infty }$). A generic three-dimensional point $A({\eta }_1,{\eta }_2,{\eta }_{\infty })$ is projected onto two two-dimensional points: $A({\eta }_2,{\eta }_1)$, located above the diagonal in the collision entropy-Shannon entropy plane (blue zone), and $A({\eta }_2,{\eta }_{\infty })$, located beneath the diagonal in the collision entropy-min-entropy plane (red zone). The value of ${\eta }_2$ is shared between the two projections.

4.

Visualization of theoretical, empirical, and stretched contexts: To clearly distinguish the regions concerning the states of theoretical probability distributions, of their derived empirical distributions, and of the states resulting after the application of the first-order affine transformation, the diagram includes the curves generated by the elaborations over MD_UR and UBD_MR families, along with the curves relative to the corresponding realizations and their stretched versions.

5.

Determination of the entropic state points of the theoretical distribution T: These points describe the entropic state of the initial distribution and represent the target of the estimation process:

$$T({\eta }_2\left(\mathit{p}\right),{\eta }_1\left(\mathit{p}\right))=(0.850,0.927)$$

$$T({\eta }_2\left(\mathit{p}\right),{\eta }_{\infty }\left(\mathit{p}\right))=(0.850,0.570)$$

6.

Generation of realizations: A set of $r=1,000$ realizations is generated from the process by applying a random number generator to the previously defined probability distribution T. A large number of realizations helps reduce the deviation of the final estimation from the target value. Each realization consists of a sequence of $L=254$ samples.

7.

Choice of the the parameters of the sample space Ω: To significantly reduce the density of data, we select a sample space $\Omega (q,d)$ with the same alphabet of the process $q=6$ and dimension $d=5$. The total number of elementary events is $n=7776$.

8.

Embedding data into the sample space to derive relative frequencies: Consequently, the number of elementary events that occur in $\Omega (q,d)$ is $N=L-d+1=254-5+1=250$. This leads to a very sparse data regime, with a density $\delta =N/n=250/7776=3.21\%$.

9.

Calculation of their center of gravity S: For each realization, we compute the Rényi means and their logarithmic mapping in entropy plane; the results are then averaged:

$$S(\overline{{\eta }_2}\left(\mathit{f}\right),\overline{{\eta }_1}\left(\mathit{f}\right))=(0.603,0.601)$$

$$S(\overline{{\eta }_2}\left(\mathit{f}\right),\overline{{\eta }_{\infty }}\left(\mathit{f}\right))=(0.603,0.494)$$

10.

Calculation of their center of gravity E: The affine transformation is applied to the Rényi means of the empirical data and the entropic state points of the translated Rényi means are derived; the results are then averaged:

$$E(\widehat{{\eta }_2}\left(\mathit{p}\right),\widehat{{\eta }_1}\left(\mathit{p}\right))=(0.845,0.880)$$

$$E(\widehat{{\eta }_2}\left(\mathit{p}\right),\widehat{{\eta }_{\infty }}\left(\mathit{p}\right))=(0.845,0.615)$$

11.

Evaluation of the distance between E and T: The estimation method is satisfying when point E results coincident or very near to point T

Figure 8 illustrates the effect of the previous steps.

4. Discussion

4.1. Considerations on the Example Presented in § 3.7

The example in § 3.7 illustrates a typical scenario where the entropic state of a theoretical distribution T significantly diverges from the empirical state S, derived from relative frequencies. Directly using values from S to estimate T would thus lead to substantial errors. Applying the affine transformation (31) to the Rényi generalized means computed from data yields entropy estimates defining a new state E, much closer to T. The collision entropy of E closely matches T, whereas the Shannon and min-entropy estimates show larger discrepancies, highlighting the need for further adjustments. Moreover, empirical points appearing clustered become more dispersed after transformation, indicating an increase in variance. Given that the value of $m_{\alpha }$ in this example is computed from a great number fo realizations, applying it to a smaller set (e.g., $1,000$ realizations) might produce state points exceeding $1.0$, particularly for min-entropy.

4.2. Considerations Concerning Algorithms for Generating MD_UR and UBD_MR Families of $DPD$s

To explore entropy trends in controlled distributions, we developed two algorithms: one for Mono Dominant with Uniform Residues (MD_UR) and another for Uniform Block Dominant with Mono Residue (UBD_MR), both derived from a uniform baseline by applying increasing distortion percentages. Algorithm 1 (MD_UR) is straightforward, with linear complexity $O\left(q\right)$, involving a single pass over events. Algorithm 2 (UBD_MR), however, redistributes probability from the tail to the head using nested loops and conditional logic, resulting in quadratic complexity $O\left(q^2\right)$. Thus, steering distributions toward Algorithm 1’s path is computationally simpler than compressing them via Algorithm 2, reflecting intrinsic asymmetries in entropy landscapes that could impact the modeling of natural probabilistic processes.

4.3. Data Contextualization vs. Data Correction in Entropy Estimation Methods

Another insight is a relativistic reinterpretation of relative frequencies in entropy estimation. Instead of viewing these frequencies as biased or erroneous, we suggest considering them as geometrically constrained observations. Limited dataset size naturally compresses empirical frequencies within the entropy state space. The affine transformation of Rényi generalized means thus acts not as traditional correction but as geometric re-contextualization, stretching empirical observations to align them with the theoretical entropy structure. This interpretation parallels general relativity’s insight that mass shapes spacetime: data mass similarly shapes observable entropy space. Small datasets produce compressed entropy states, which expand toward theoretical configurations as data mass increases. Figure 7 clearly visualizes this phenomenon, showing initial empirical compression and subsequent geometric expansion via affine transformation and logarithmic mapping.

5. Conclusion

In a broad sense, this work aims to contribute to answer to Hilbert’s Sixth Problem by addressing the longstanding challenge of rigorously bridging axiomatic foundations and empirical methodologies. It proposes a novel entropy estimator, grounded in a structured affine transformation of empirical frequency distributions, effectively translating formal definitions of Rényi entropy into practical, statistically robust and computationally efficient estimation methods. The paper introduces three core innovations: (1) the use of specific entropies ${\eta }_{\alpha }$, enabling clear geometric interpretations; (2) the definition of two structured probability distribution families—Mono Dominant with Uniform Residues (MD_UR) and Uniform Block Dominant with Mono Residue (UBD_MR); and (3) a general estimation framework applicable across varying datasets, data types, and dimensions, capable of accurately estimating Shannon, collision, and min-entropy. Among these, collision entropy stands out because of its simplicity of estimation via affine stretching. However, relying solely on collision entropy can obscure critical nuances. Thus, the proposed approach naturally encourages the complementary use of multiple entropy indices (collision, Shannon, and min-entropy), enriching the characterization of uncertainty. Furthermore, this framework offers rigorous epistemological insights by precisely quantifying how empirical data contribute to our understanding of underlying realities.

6. Future Work

The findings of this study suggest several future research directions:

• Improved transformation models: Develop secondary orthogonal transformations to refine entropy estimation, especially min-entropy, through adaptive corrections.
• Analytical characterization and simplification: Investigating analytical approximations or tight bounds for empirical Rényi means $m_{\alpha }$, reducing computational complexity.
• Variance control and confidence estimation: Enhancing robustness by exploring variance control methods, including smoothing kernels, shrinkage techniques, or deriving confidence intervals analytically.
• Extension to real-world data: Validation of the estimator on real datasets, particularly in cybersecurity, neuroscience, and computational biology, to test practical effectiveness.
• Generalization to other information functionals: Extending the affine estimation framework to broader information-theoretic measures, expanding its theoretical and practical scope.
• Integration into statistical and machine learning workflows: Exploring applications of entropy estimation within machine learning as feature transformations, loss functions, or regularizers to innovate data-driven modeling.