An Intrinsic Characterization of Shannon’s and Rényi’s Entropy

1. Results

We denote by $\{.\}$ an ordered set und let

$$\begin{array}{ccc}\hfill a\{p_1,p_2,\dots \}& =& \{a{p}_1,a{p}_2,\dots \},a\ge 0,\hfill \\ \hfill \{p_1,p_2,\dots \}\cup \{q_1,q_2,\dots \}& =& \{p_1,p_2,\dots ,q_1,q_2,\dots \}.\hfill \end{array}$$

(7)

As we will address only the very first elements of an ordered set, the intuitive, but sloppy notation on the right hand side of Equation (7) is sufficient.

Let denote by P the set of all discrete probability distributions and by $P_n\subset P$ the set of distributions with at most n atomic events that have a probability greater than zero. We denote such a probability distribution by $\{p_1,\dots ,p_n\}\in P_n$. The uniform distribution is denoted by $U_n=\{1/n,\dots ,1/n\}$.

Reordering the elements in Equation (5) will turn out to be very useful in the proofs. We define for $p\in P$

$$\begin{array}{ccc}\hfill p_{\widehat{k}}& =& \{p_i:i\ne k\},\hfill \\ \hfill {\partial }_{k}p& =& p_{k}p\cup p_{\widehat{k}}.\hfill \end{array}$$

(8)

For instance,

$$\begin{array}{ccc}\hfill {\partial }_3\{p_1,p_2,p_3\}& =& \{p_3{p}_1,p_3{p}_2,p_3{p}_3,p_1,p_2\}.\hfill \end{array}$$

We use the convention $0log0=0$.

Theorem 1.

Let $H:P\to [0,\infty ]$ be a function. Then the following two assertions are equivalent:

(1)

H is the Shannon entropy,

$$H\left(p\right)=-\sum _{p_i\in p}{p}_{i}log{p}_i,$$

up to a positive multiplicative constant.

(2)

H has the following properties:

(a)

H is a continuous function in the topology of convergence in distribution;

(b)

$H\left(\{p_1,\dots ,p_n\}\right)=H\left(\{p_{\pi \left(1\right)},\dots ,p_{\pi \left(n\right)}\}\right)$ for any permutation π and $\{p_1,\dots ,p_n\}\in P_n$, $n\in \mathbb{N}$;

(c)

$H\left(\{p_1,\dots ,p_n\}\right)=H\left(\{0,p_1,\dots ,p_n\}\right)$ for all $\{p_1,\dots ,p_n\}\in P_n$, $n\in \mathbb{N}$;

(d)

$H\left(p\right)\le H\left(U_n\right)<\infty$ for all $p\in P_n$, $n\in \mathbb{N}$;

(e)

$H\left(p\right)\in (0,\infty )$ if p is a non-degenerate geometric distribution;

(f)

$H(p\times q)=H\left(p\right)+H\left(q\right)$ for all $p,q\in P$;

(g)

some function $f:[0,1]\to [0,\infty )$ exists such that for all $p\in P$ we have

$$\begin{array}{c}\hfill H\left({\partial }_{1}p\right)=(1+f\left(p_1\right))H\left(p\right).\end{array}$$

(9)

Conds. 2a-2g can be regarded as interpretable, hence intrinsic properties, cf. [Section 1.2] [5]. For this reason we prefer Cond. 2d over a monotonicity assumption on H. Note that Conds. 2a-2d are precisely Khinchin ’s ([1]) conditions as given in [2] except for replacing the chain rule by Conds. 2e-2g.

Remark 1.

Note that f is not unique since it is undetermined at 1. A convenient choice is to take $f\left(1\right)={lim}_{a\to 1}f\left(a\right)$.

Remark 2.

Our approach has a simple, but nice implication. Let $p,q\in P_n$, $n\in \mathbb{N}$. Then,

$$\begin{array}{ccc}\hfill \mathrm{KL}({\partial }_{1}p,{\partial }_{1}q)& =& (1+p_1)\mathrm{KL}(p,q),\hfill \end{array}$$

where $\mathrm{KL}$ denotes the Kullback-Leibler divergence.

In contrast to our characterization of the Shannon entropy, our analogous characterization of the Rényi entropy needs stronger assumptions on the unknown function f.

Definition 1

(Def 2.3.2 in [12]). A function $f:U\to \mathbb{R}$, $U\subset \mathbb{R}$, is called real analytic if for each $y\in U$ a neighborhood $U_y$ of y exists, such that for all $x\in U_y$ there is an absolutely and uniformly convergent series representation of the form $f\left(x\right)={\sum }_k{a}_k{(x-y)}^k$.

In the following, we call a function $f:[0,1]\to [0,\infty )$ smooth if ${\prod }_{k=0}^{\infty }(1+f\left(a^{2^k}\right))<\infty$ for all $a\in (0,1)$ and if it can be extended to a real analytic function on a domain including $(0,1]$.

Theorem 2.

Let $H:P\to [0,\infty ]$ be a function. Then, the following two assertions are equivalent:

(1)

H is the Rényi entropy,

$$H\left(p\right)={\displaystyle \frac{1}{1-\alpha }}log\sum _{p_i\in p}{p}_i^{\alpha },$$

for some $\alpha \in (0,\infty )\setminus \left\{1\right\}$, up to a positive multiplicative constant.

(2)

H satisfies Conds. 2a-2f of Theorem 1. A value $\gamma \in \mathbb{R}\setminus \left\{0\right\}$ and a smooth function $f:[0,1]\to [0,\infty )$ exist such that, for $S=e^{\gamma H}$, we have

$$\begin{array}{c}\hfill S\left({\partial }_{1}p\right)=(1+f\left(p_1\right))S\left(p\right)-f\left(p_1\right)\forall p\in P.\end{array}$$

(10)

Since Theorem 2 gives a characterization only up to a multiplicative constant, the missing constant is typically determined by the requirement that the Shannon entropy is canonically obtained as $\alpha \to 1$. The limit of the Rényi entropy as $\alpha \to \infty$ is called the min-entropy and will be considered next. Two major differences to the two previous theorems exist: First, the function f appears on the left hand side of the proportionality relation. Second, two characterizing equations are necessary. Unfortunately, the nice interpretability of Equation (9) reduces in Equation (10) and even further in Equations (11) and ().

Theorem 3.

Let $H:P\to [0,\infty ]$ be a function. Then, the following two assertions are equivalent:

(1)

H is the min-entropy,

$$H\left(p\right)=-log\underset{p_i\in P}{max}{p}_i.$$

(2)

H satisfies Conds. 2a-2c, 2e and 2f of Theorem 1. Furthermore, $H\left(p\right)<\infty$ for $p\in P_n$, $n\in \mathbb{N}$. A smooth, monotone function $f:[0,1]\to [0,\infty )$ exists, such that for $S=e^{-H}$ and all $p\in P\setminus \left\{\right\{1\left\}\right\}$, we have

$$\begin{array}{ccc}\hfill f\left(p_1\right)\vee S\left({\partial }_{1}p\right)& =& S\left(p\right),\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill (1+f\left(p_1\right))S\left({\displaystyle \frac{{\left({\partial }_{1}p\right)}_{\widehat{1}}}{1-p_1^2}}\right)& =& S\left({\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}\right).\hfill \end{array}$$

(12)

2. Proofs

2.1. Auxiliary Results

For the reader’s convenience, we first repeat several known functional equations and properties.

Lemma 1.

Let H be the Shannon entropy or the Rényi entropy. Then, $H\left(U_n\right)$ is maximal on $P_n$.

See, for instance, Lemma 2.2.4 and Remark 4.4.4 in [4] for a proof.

Lemma 2.

Let $H:P\to [0,\infty ]$ be a function fullfilling Conds. 2d and 2f of Theorem 1. Then, $H\left(U_n\right)=clogn$ for all $n\in \mathbb{N}$ and some $c>0$.

Proof. 

The inclusion $P_n\subset P_{n+1}$ and Cond. 2d immediately imply monotonicity, i.e.

$$H\left(U_n\right)\le H\left(U_{n+1}\right)\forall n\in \mathbb{N}.$$

By Cond. 2f and Implication 1 of Theorem V in [13] follows the claim.

□

Lemma 3

(Identity theorem, Corollary 1.2.7 in [14]). If f and g are real analytic functions on an open interval $U\subset \mathbb{R}$ and if there is a sequence of distinct points $x_1,x_2,\dots$ in U with $x_{\infty }={lim}_{n\to \infty }{x}_n\in U$ such that $f\left(x_n\right)=g\left(x_n\right)$ for all $n\in \mathbb{N}$, then

$$f\left(x\right)=g\left(x\right)\mathrm{for}\mathrm{all}x\in U.$$

The proof for the Shannon entropy needs the uniqueness of the solution of the functional Equation (13) below. We state two lemmas for the same purpose. The first needs the strong additional assumption that the solution is real analytic. The lemma itself remains unused, but the ideas behind the proof are used several times. The second lemma states the version used in Theorem 1; see Theorem 6.4.8 in [6] for a condition similar to Equation (16).

Lemma 4.

Let f be a real-valued function on a domain that includes $[0,1)$. The following two assertions are equivalent:

(1)

$f\left(a_0\right)=a_0$ for some $a_0\in (0,1)$, f is real analytic and

$$\begin{array}{ccc}\hfill (1-f\left(a\right))\prod _{k=0}^{\infty }(1+f\left(a^{2^k}\right))& =& 1\forall a\in (0,1);\hfill \end{array}$$

(13)

(2)

f is the identity.

Proof. 

By Equation 0.266 in [15], the identity satisfies Equation (13). Assume now that Equation (13) holds. Then,

$$\begin{array}{c}\hfill f\left(a^2\right)=f^2\left(a\right),\end{array}$$

since

$$\begin{array}{ccc}\hfill 1& =& (1-f\left(a\right))(1+f\left(a\right))\prod _{k=1}^{\infty }(1+f\left(a^{2^k}\right))\hfill \\ & =& (1-f\left(a\right))(1+f\left(a\right))/(1-f\left(a^2\right)).\hfill \end{array}$$

(14)

It follows that

$$\begin{array}{c}\hfill f\left(a^{2^k}\right)={\left(f\left(a\right)\right)}^{2^k}\forall a\in (0,1),k\in \mathbb{Z}.\end{array}$$

(15)

Since $a^{2^k}\to 0$ as $k\to \infty$, Lemma 3 yields that f is uniquely determined by the knowledge of a single value $f\left(a_0\right)$ with $a_0\in (0,1)$. □

Lemma 5.

Let f be a real-valued function on $(0,1)$. The following two assertions are equivalent:

(1)

Equation (13) holds, f is continuous and for all $a\in (0,1)$ we have

$$\begin{array}{ccc}\hfill f\left(a\right)+f(1-a)& =& 1;\hfill \end{array}$$

(16)

(2)

f is the identity.

Proof. 

Equation (16) yields $f(1/2)=1/2$. Equations (15) and (16) imply that $f\left({(1-a^{2^k})}^{2^r}\right)={(1-f^{2^k}\left(a\right))}^{2^r}$ for all $a\in (0,1)$ and $k,r\in \mathbb{Z}$. Let $T\left(A\right)=\{{(1-a^{2^k})}^{2^r}:a\in A,r,k\in \mathbb{Z}\}$ for $A\subset (0,1)$, $D_1=T\left(\{1/2\}\right)$, $D_{n+1}:=D_n\cup T\left(D_n\right)$ and $D={\bigcup }_n{D}_n$. It suffices to show that D is dense in $(0,1)$. Assume that it is not i.e., an intervall $(a,b)\subset (0,1)$ exists with $(a,b)\cap D=\varnothing$. Since $1/2\in D$, we may assume without loss of generality, that $(a,b)\subset (1/2,1)$. Then,

$$(a^2,b^2)\cap D=(1-b^2,1-a^2)\cap D=\varnothing .$$

Consider the map f on the open intervalls of $(0,1)$,

$$f\left((a,b)\right)\mapsto \left\{\begin{array}{cc}(a^2,b^2),\hfill & ifa\ge 1/2,b^2>1/2\hfill \\ (1-b^2,1-a^2),\hfill & ifa\ge 1/2,b^2\le 1/2,\hfill \\ (a,b)\hfill & a<1/2\hfill \end{array}\right.$$

Let $f^{\left(n\right)}$ be the n-times iterative application of f, i.e.,

$$f^{\left(n\right)}=f\circ \dots \circ f$$

and $(c,d)=f^{(n-1)}\left((a,b)\right)$. Per assumption f maps $(a,b)\subset (1/2,1)$ to either $(a^2,b^2)$ or $(1-b^2,1-a^2)$. Per construction, if $f^{\left(n\right)}\left((a,b)\right)=(1-c^2,1-d^2)$, then $f^{(n+1)}\left((a,b)\right)\ne (1-c^2,1-d^2)$. If eventually $f^{\left(n\right)}\left((a,b)\right)=f^{(n+1)}\left((a,b)\right)=(c^2,d^2)$, then it always holds $1/2\in (c^2,d^2)$. Let $\left|\right(a,b\left)\right|$ be the length of an interval $(a,b)$, i.e., $b-a$. If $a\ge 1/2$, then

$$\left|f(a,b)\right|=b^2-a^2=(b+a)(b-a)>b-a.$$

Hence, $f^{(n+1)}\left((a,b)\right)$ finally contains $1/2$ or its length is strictly monotoneously increasing. If the latter were true, the limit would exist as $\left|f\right|\le 1/2$. In the limit we would have $b^2-a^2=b-a$, a contradiction to $b>a\ge 1/2$. □

2.2. Properties of the Operator ${\partial }_k$

We will use the operator ${\partial }_k$, defined in Equation (8), iteratively. For instance,

$$\begin{array}{ccc}\hfill {\partial }_2{\partial }_{k}p& =& \left(p_2{p}_k\right)(p_{k}p\cup p_{\widehat{k}})\cup {(p_{k}p\cup p_{\widehat{k}})}_{\widehat{2}},1\le k\le n,n\ge 2,\hfill \\ \hfill {\partial }_2{\partial }_3\{p_1,p_2,p_3\}& =& {\partial }_2\{p_3{p}_1,p_3{p}_2,p_3{p}_3,p_1,p_2\}\hfill \\ & =& \begin{array}{ccccc}\hfill \{p_3{p}_2{p}_3{p}_1,& \hfill p_3{p}_2{p}_3{p}_2,& \hfill p_3{p}_2{p}_3{p}_3,& \hfill p_3{p}_2{p}_1,& \hfill p_3{p}_2{p}_2,\\ \hfill p_3{p}_1,& & \hfill p_3{p}_3,& \hfill p_1,& \hfill p_2\}\end{array}\hfill \\ & =& \{p_1{p}_2{p}_3^2,p_2^2{p}_3^2,p_2{p}_3^3,p_1{p}_2{p}_3,p_2^2{p}_3,p_1{p}_3,p_3^2,p_1,p_2\}.\hfill \end{array}$$

Note that the terms in general do not commute, e.g., ${(p_{k}p\cup p_{\widehat{k}})}_{\widehat{k}}\ne {(p_{\widehat{k}}\cup p_{k}p)}_{\widehat{k}}$ and ${\partial }_i{\partial }_{k}p\ne {\partial }_k{\partial }_{i}p$. The proofs of the theorems in Section 1 rely on the properties of various probability distributions. We denote by $b_a$ the Bernoulli distribution, i.e.,

$$b_a=\{a,1-a\},a\in (0,1),$$

and by $g_a$ the geometric distribution, i.e.,

$$g_a=\{1-a,(1-a)a,(1-a)a^2,\dots \},a\in (0,1).$$

For $n\in \mathbb{N}$, let

$$\begin{array}{ccc}\hfill g_{n,a}& =& {\displaystyle \frac{1-a}{1-a^n}}\{1,a,a^2,\dots ,a^{n-1}\},a\in [0,1),\hfill \end{array}$$

$$\begin{array}{ccc}\hfill G_{n,a}& =& \{a^n,1-a,(1-a)a,\dots ,(1-a)a^{n-1}\},a\in [0,1],\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill U_{k,n}& =& \{1-k/n,\underset{k\mathrm{times}}{\underbrace{1/n,\dots ,1/n}}\},0\le k<n.\hfill \end{array}$$

(18)

The geometric distribution plays a dominant role in all of the subsequent proofs. Its important role has been known in information theory: it maximizes the Shannon entropy among discrete probability distributions given the mean [Theorem 5.8] [16] and minimizes the min-entropy under fixed variance, among all discrete log-concave random variables in $\mathbb{Z}$ [Theorem 1] [17]. Let denote by $\dot{=}$ equality up to inserting zeros and reordering.

Lemma 6.

Let $p\in P$ and ${\partial }_1^{\infty }p:={lim}_{n\to \infty }{\partial }_1^{n}p$. Then,

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }p& \dot{=}& g_{p_1}\times {\displaystyle \frac{p_{\widehat{1}}}{1-p_1}},p_1<1,\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill \left\{1\right\}& \dot{=}& \left\{1\right\}\times \left\{1\right\}.\hfill \end{array}$$

(20)

In particular, for $a\in (0,1)$, $n\in \mathbb{N}$ and $0\le k<n$, we have

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{b}_a& \dot{=}& g_a,\hfill \end{array}$$

(21)

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{g}_a& \dot{=}& g_{1-a}\times g_a,\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{G}_{n,a}& \dot{=}& g_{a^n}\times g_{n,a}\dot{=}{g}_a\dot{=}{G}_{\infty ,a},\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill {\partial }_2^{\infty }{G}_{n,a}& \dot{=}& g_{1-a}\times G_{n-1,a},\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{U}_{k,n}& \dot{=}& g_{1-k/n}\times U_k,\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill {\partial }_2^{\infty }{U}_{k,n}& \dot{=}& g_{1/n}\times U_{k-1,n-1},0\le k<n,\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill {\partial }_3^{\infty }{\partial }_2{b}_a& \dot{=}& g_a\times b_a.\hfill \end{array}$$

(27)

Furthermore, for $a\in (0,1)$ and $n\in \mathbb{N}$,

$$\begin{array}{ccc}\hfill {\partial }_1{G}_{n,a}& \dot{=}& G_{2n,a},\hfill \end{array}$$

(28)

$$\begin{array}{ccc}\hfill {\left(G_{n,a}\right)}_{\widehat{2}}& =& a{G}_{n-1,a},\hfill \\ \hfill (1-a^n)g_{n,a}& =& {\left(G_{n,a}\right)}_{\widehat{1}},\hfill \\ \hfill U_{n-1,n}& =& U_n.\hfill \end{array}$$

(29)

Proof.  We have

$$\begin{array}{c}\hfill {\partial }_1^{n}p=\left\{p_1^{2^n}\right\}\cup \bigcup _{k\in \{2^n-1,\dots ,0\}}{p}_1^k{p}_{\widehat{1}},\end{array}$$

since this is true for $n=1$ and

$$\begin{array}{ccc}\hfill {\partial }_1^{n}p& =& {\partial }_1\left({\partial }_1^{n-1}p\right)={\partial }_1\left(\left\{p_1^{2^{n-1}}\right\}\cup \bigcup _{k\in \{2^{n-1}-1,\dots ,0\}}{p}_1^k{p}_{\widehat{1}}\right)\hfill \\ & =& p_1^{2^{n-1}}\left(\left\{p_1^{2^{n-1}}\right\}\cup \bigcup _{k\in \{2^{n-1}-1,\dots ,0\}}{p}_1^k{p}_{\widehat{1}}\right)\cup \bigcup _{k\in \{2^{n-1}-1,\dots ,0\}}{p}_1^k{p}_{\widehat{1}}\hfill \\ & =& \left\{p_1^{2^n}\right\}\cup \bigcup _{k\in \{2^n-1,\dots ,2^{n-1}\}}{p}_1^k{p}_{\widehat{1}}\cup \bigcup _{k\in \{2^{n-1}-1,\dots ,0\}}{p}_1^k{p}_{\widehat{1}}\hfill \end{array}$$

by induction. Hence,

$$\begin{array}{c}\hfill {\partial }_1^{\infty }p\dot{=}\bigcup _{k\in \{\dots ,1,0\}}{p}_1^k{p}_{\widehat{1}}.\end{array}$$

Now, $p_1^k{p}_{\widehat{1}}=\left((1-p_1)p_1^k\right)·{\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}$ for $p_1<1$, implying Equation (19). Further, we have

$$\begin{array}{ccc}\hfill {\partial }_2{b}_a& =& (1-a)b_a\cup \left\{a\right\},\hfill \end{array}$$

so that Equation () holds by Equation (19). The other equalities are immediate. □

2.3. Proof of Theorem 1

Assume that H is the Shannon entropy. Then,

$$H\left(g_a\right)=-\sum _{k=0}^{\infty }(1-a)a^{k}log\left((1-a)a^k\right)=-(aloga+(1-a)log(1-a))/(1-a)\in (0,\infty )$$

for $a\in (0,1)$. All the other assertions in the second condition hold obviously, when choosing f to be the identity. Now, assume that the second condition holds. By Cond. 2a, Cond. 2b holds for all $p\in P$. Cond. 2f and Equation () yield $H\left(\right\{1\left\}\right)=0$. It is easy to check that

$$H\left({\partial }_1^{n}p\right)=H\left(p\right)\prod _{k=0}^{n-1}\left(1+f\left(p_1^{2^k}\right)\right).$$

Thus,

$$\begin{array}{c}\hfill H\left({\partial }_1^{\infty }p\right)=H\left(\underset{n\to \infty }{lim}{\partial }_1^{n}p\right)=\underset{n\to \infty }{lim}H\left({\partial }_1^{n}p\right)=H\left(p\right)F\left(p_1\right)\end{array}$$

(30)

with

$$\begin{array}{c}\hfill F\left(a\right)=\prod _{k=0}^{\infty }\left(1+f\left(a^{2^k}\right)\right),a\in (0,1).\end{array}$$

The function F is finite on $(0,1)$ due to Cond. 2e, Cond. 2f, and Equations () and (30), i.e.,

$$\begin{array}{c}\hfill H\left(g_a\right)F(1-a)=H\left(g_{1-a}\right)+H\left(g_a\right).\end{array}$$

It follows

$$F(1-a)-1={\displaystyle \frac{H\left(g_{1-a}\right)}{H\left(g_a\right)}},a\in (0,1).$$

Replacing a by $1-a$ in the equation above yields

$$\begin{array}{c}\hfill \left(F\right(1-a)-1)\left(F\right(a)-1)=1,a\in (0,1).\end{array}$$

(31)

Now,

$$\begin{array}{c}\hfill H\left(b_a\right)F\left(a\right)=H\left(g_a\right),\end{array}$$

(32)

by Equations (21) and (30). Equations (), (9) and (30) yield

$$\begin{array}{c}\hfill H\left(b_a\right)(1+f(1-a))F\left(a\right)=H\left(g_a\right)+H\left(b_a\right).\end{array}$$

(33)

Since $H\left(b_a\right)>0$ by Equation (32) and Cond. 2e, we immediately get that

$$\begin{array}{c}\hfill f(1-a)F\left(a\right)=1,a\in (0,1).\end{array}$$

(34)

Equation (31) now writes $f\left(a\right)+f(1-a)=1$. Plugging in $p=b_a$ into Equation (9) shows that f is continuous on $(0,1)$. Equation (34) and Lemma 5 yield that $f\left(a\right)=a$ and $F\left(a\right)=1/(1-a)$ for $a\in (0,1)$. Equation (9) implies that $f\left(0\right)=0$ so that f is the identity on $[0,1)$. Lemma 2 implies that $H\left(U_n\right)=clogn$ for some $c>0$. Henceforth, we may assume that $c=1$. Cond. 2f and Equations (19) and (30) deliver

$$\begin{array}{c}\hfill H\left(p\right)F\left(p_1\right)=H\left(g_{p_1}\right)+H\left({\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}\right),p_1<1.\end{array}$$

(35)

This implies

$$\begin{array}{c}\hfill logn·F(1/n)=H\left(U_n\right)F(1/n)=H\left(g_{1/n}\right)+H\left(U_{n-1}\right)=H\left(g_{1/n}\right)+log(n-1),n\ge 2,\end{array}$$

so that, by Equation (32),

$$\begin{array}{c}\hfill H\left(b_{1/n}\right)=logn-{\displaystyle \frac{log(n-1)}{F(1/n)}}=-{\displaystyle \frac{1}{n}}log{\displaystyle \frac{1}{n}}-\left(1-{\displaystyle \frac{1}{n}}\right)log\left(1-{\displaystyle \frac{1}{n}}\right),n\ge 2.\end{array}$$

Due to the recursion Formula (35) and Equation (32), it suffices to show that the above formula extends to $H\left(b_a\right)=-aloga-(1-a)log(1-a)$, $a\in (0,1)$, to finish the proof. By Equations () and (30), we get

$$H\left(U_{k,n}\right)F(1/n)=H\left(g_{1/n}\right)+H\left(U_{k-1,n-1}\right),$$

so that

$$H\left(U_{k,n}\right)=\sum _{i=0}^{k-1}{\displaystyle \frac{H\left(g_{1/(n-i)}\right)}{{\prod }_{j=n-i}^{n}F(1/j)}}={\displaystyle \frac{1}{n}}\sum _{i=0}^{k-1}(n-i)H\left(b_{1/(n-i)}\right)$$

by iteration and by Equation (32), respectively. On the other hand, by Equations () and (30),

$$H\left(U_{k,n}\right)F(1-k/n)=H\left(g_{1-k/n}\right)+log\left(k\right),$$

so that

$$\begin{array}{c}\hfill {\displaystyle H\left(g_{1-k/n}\right)+log\left(k\right)}\\ & =& {\displaystyle \frac{1}{k}}\sum _{i=0}^{k-1}(n-i)\left(-{\displaystyle \frac{1}{n-i}}log{\displaystyle \frac{1}{n-i}}-\left(1-{\displaystyle \frac{1}{n-i}}\right)log\left(1-{\displaystyle \frac{1}{n-i}}\right)\right)\hfill \\ & =& {\displaystyle \frac{1}{k}}\sum _{i=0}^{k-1}\left((n-i)log(n-i)-\left(n-i-1\right)log\left(n-i-1\right)\right)\hfill \\ & =& {\displaystyle \frac{1}{k}}\left(nlogn-(n-k)log(n-k)\right)\hfill \end{array}$$

and by Equation (32)

$$H\left(b_{k/n}\right)=-{\displaystyle \frac{k}{n}}log{\displaystyle \frac{k}{n}}-\left(1-{\displaystyle \frac{k}{n}}\right)log\left(1-{\displaystyle \frac{k}{n}}\right),n\ge 2,k<n.$$

Cond. 2a finalizes the proof.

2.4. Proof of Theorem 2

Assume that H is the Rényi entropy. Let

$$\begin{array}{c}\hfill S\left(p\right)=e^{(1-\alpha )H\left(p\right)},\end{array}$$

i.e., $S\left(p\right)={\parallel p\parallel }_{\alpha }^{\alpha }$. Then,

$$S\left({\partial }_{1}p\right)=p_1^{\alpha }\sum _i{p}_i^{\alpha }+\sum _i{p}_i^{\alpha }-p_1^{\alpha }$$

so that $f\left(r\right)=r^{\alpha }$. Reversely, assume that the second condition holds. Since $H\ge 0$ and S is strictly monotone in H we can choose the sign of $\gamma$ such that S is increasing in H and $S\ge 1$. Furthermore, S is continuous. If $S\left(p\right)=1$, then $1=S\left({\partial }_{1}p\right)=S\left({\partial }_{2}p\right)=\dots =S\left({\partial }_1^{\infty }p\right)$. Hence, $S\left(b_a\right)>1$ for $a\in (0,1)$ as otherwise, by Equation (21), we would get a contradiction to Cond. 2e. It follows that $f\left(a\right)=(S\left({\partial }_1{b}_a\right)-S\left(b_a\right))/(S\left(b_a\right)-1)$ is continuous on $(0,1)$. Equation (10) implies $f\left(0\right)\left(S\right(\left\{0\right\}\cup p)-1)=0$ for all $p\in P$, so that $f\left(0\right)=0$. Cond. 2f and Equation () yield $H\left(\right\{1\left\}\right)=0$ and thus $S\left(\right\{1\left\}\right)=1$. Cond. 2f in Theorem 1 reads

$$\begin{array}{c}\hfill S(p\times q)=S\left(p\right)S\left(q\right).\end{array}$$

(36)

By Cond. 2a , Cond. 2b holds for all $p\in P$. It is easy to check by induction that Equation (10) implies

$$\begin{array}{c}\hfill S\left({\partial }_1^{n}p\right)=S\left(p\right)\prod _{k=0}^{n-1}\left(1+f\left(p_1^{2^k}\right)\right)-\sum _{\ell =0}^{n-1}f\left(p_1^{2^{\ell }}\right)\prod _{k=\ell +1}^{n-1}\left(1+f\left(p_1^{2^k}\right)\right).\end{array}$$

(37)

Henceforth, we assume that $a\in (0,1)$. Cond. 2e and Equation () yield

$$\begin{array}{c}\hfill \infty >S\left(g_a\right)S\left(g_{1-a}\right)\leftarrow S\left(g_{1-a}\right)\prod _{k=0}^{n-1}\left(1+f\left(a^{2^k}\right)\right)-\sum _{\ell =0}^{n-1}f\left(a^{2^{\ell }}\right)\prod _{k=\ell +1}^{n-1}\left(1+f\left(a^{2^k}\right)\right)\end{array}$$

(38)

as $n\to \infty$. Since $F\left(a\right)={\prod }_{k=0}^{\infty }\left(1+f\left(a^{2^k}\right)\right)$ is assumed to be finite, the function G given by

$$\begin{array}{ccc}\hfill G\left(a\right)& =& \sum _{\ell =0}^{\infty }f\left(a^{2^{\ell }}\right)F\left(a^{2^{\ell +1}}\right)\hfill \end{array}$$

is finite by Equation (38). Hence, by Equations (19), (36) and (37),

$$\begin{array}{c}\hfill S\left(p\right)F\left(p_1\right)-G\left(p_1\right)=S\left(g_{p_1}\right)S\left({\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}\right),p_1<1.\end{array}$$

(39)

Equation (39) implies that

$$\begin{array}{c}\hfill S\left(g_a\right)F(1-a)-G(1-a)=S\left(g_{1-a}\right)S\left(g_a\right),\end{array}$$

so that $G\left(a\right)=(F\left(a\right)-S\left(g_a\right))S\left(g_{1-a}\right)$ and Equation (39) reads

$$\begin{array}{c}\hfill S\left(p\right)F\left(p_1\right)-(F\left(p_1\right)-S\left(g_{p_1}\right))S\left(g_{1-p_1}\right)=S\left(g_{p_1}\right)S\left({\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}\right),p_1<1.\end{array}$$

(40)

In particular,

$$\begin{array}{c}\hfill S\left(b_a\right)F\left(a\right)-(F\left(a\right)-S\left(g_a\right))S\left(g_{1-a}\right)=S\left(g_a\right).\end{array}$$

(41)

In analogy to Equation (33) we get from Equation () that

$$\left[S\left(b_a\right)(1+f(1-a))-f(1-a)\right]F\left(a\right)-(F\left(a\right)-S\left(g_a\right))S\left(g_{1-a}\right)=S\left(g_a\right)S\left(b_a\right),$$

so that, by Equation (41),

$$\left[S\left(b_a\right)(1+f(1-a))-f(1-a)\right]F\left(a\right)-S\left(b_a\right)F\left(a\right)+S\left(g_a\right)=S\left(g_a\right)S\left(b_a\right),$$

i.e., as $S\left(b_a\right)>1$,

$$\begin{array}{c}\hfill f(1-a)F\left(a\right)=S\left(g_a\right).\end{array}$$

(42)

Since $S\left(g_a\right)>1$ for all $a\in (0,1)$ it follows that $f\left(a\right),F\left(a\right)\ne 0$ for all $a\in (0,1)$. Now, Equation (40) reads

$$\begin{array}{c}\hfill S\left(p\right)-(1-f(1-p_1))S\left(g_{1-p_1}\right)=f(1-p_1)S\left({\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}\right).\end{array}$$

(43)

Since

$$\begin{array}{c}\hfill 1<S\left(b_a\right)=(1-f\left(a\right))S\left(g_a\right)+f\left(a\right),\end{array}$$

(44)

the function f cannot take the value 1 on $(0,1)$. Due to the continuity of f we may summarize that

$$\begin{array}{c}\hfill f\left(0\right)=0,f\left(a\right)\in (0,1)\mathrm{for}a\in (0,1).\end{array}$$

Lemma 2 implies that $S\left(U_n\right)=n^c$ for some $c>0$. Then, Equation (43) gives

$$\begin{array}{c}\hfill (1-f(1-1/n))S\left(g_{1-1/n}\right)=n^c-f(1-1/n){(n-1)}^c.\end{array}$$

Furthermore, taking the idea from Equation (),

$$S\left(U_{k,n}\right)=(1-f(k/n))S\left(g_{k/n}\right)+f(k/n)k^c$$

and, cf. Equation (),

$$\begin{array}{ccc}\hfill S\left(U_{k,n}\right)& =& (1-f(1-1/n))S\left(g_{1-1/n}\right)+f(1-1/n)S\left(U_{k-1,n-1}\right)\hfill \\ & =& n^c+f(1-1/n)\left[-{(n-1)}^c+S\left(U_{k-1,n-1}\right)\right]\hfill \\ & =& n^c+f(1-1/n)f(1-1/(n-1))\left[-{(n-2)}^c+S\left(U_{k-2,n-2}\right)\right]\hfill \\ & =& n^c+(1-{(n-k)}^c)\prod _{j=0}^{k-1}f(1-1/(n-j)),\hfill \end{array}$$

so that, for $0\le k<n$, we have

$$\begin{array}{c}\hfill (1-f(k/n))S\left(g_{k/n}\right)=n^c+(1-{(n-k)}^c)\prod _{j=0}^{k-1}f(1-1/(n-j))-f(k/n)k^c.\end{array}$$

(45)

The continuity of S and Equations (43) and (42) imply that H is uniquely determined if f and c are unique. So, it remains to determine them. In line with the idea from Equation (), Equations (43) and () yield

$$\begin{array}{ccc}\hfill S\left(G_{n,a}\right)& =& (1-f\left(a\right))S\left(g_a\right)+f\left(a\right)S\left(G_{n-1,a}\right)\hfill \\ & =& (1-f\left(a\right))S\left(g_a\right)\sum _{k=0}^{n-1}{f}^k\left(a\right)+f^n\left(a\right)\hfill \\ & =& f^n\left(a\right)+(1-f^n\left(a\right))S\left(g_a\right).\hfill \end{array}$$

(46)

By Equations (28) and (10) we have

$$\begin{array}{ccc}\hfill S\left(G_{2n,a}\right)& =& (1+f\left(a^n\right))S\left(G_{n,a}\right)-f\left(a^n\right).\hfill \end{array}$$

Equation (46) yields

$$\begin{array}{ccc}\hfill f^{2n}\left(a\right)+(1-f^{2n}\left(a\right))S\left(g_a\right)& =& (1+f\left(a^n\right))(f^n\left(a\right)+(1-f^n\left(a\right))S\left(g_a\right))-f\left(a^n\right),\hfill \end{array}$$

i.e.,

$$\begin{array}{c}\hfill (f\left(a^n\right)-f^n\left(a\right))(1-f^n\left(a\right))(S\left(g_a\right)-1)=0.\end{array}$$

Since $S\left(g_a\right)>1$ and $f<1$ for $a\in (0,1)$ we get for $n=2$ that

$$\begin{array}{c}\hfill f^2\left(a\right)=f\left(a^2\right)\forall a\in (0,1),\end{array}$$

or, $f\left(a^{2^k}\right)=f^{2^k}\left(a\right),k\in \mathbb{Z}$. Since ${\left(a^{2^{-k}}\right)}_{k\in \mathbb{N}}$ converges to 1 and f is assumed to be smooth, Lemma 3 and the proof of Lemma 4 yield the uniqueness of f, and $f\left(a\right)=a^{\alpha }$ for $a\in [0,1]$ and some $\alpha >0$. Letting $n=2k$, Equations (44) and (45) yield

$$\begin{array}{ccc}\hfill S\left(b_{1/2}\right)& =& {\left(2k\right)}^c+(1-k^c)\prod _{j=0}^{k-1}f\left(1-1/(n-j)\right)-f(1/2)k^c+f(1/2)\hfill \\ & =& {\left(2k\right)}^c+(1-k^c)2^{-\alpha }-2^{-\alpha }{k}^c+2^{-\alpha }.\hfill \end{array}$$

Hence,

$$2^{\alpha }S\left(b_{1/2}\right)=(2^{\alpha +c}-2)k^c+2.$$

Since the left hand side is independent of k, the equation $c=1-\alpha$ must hold.

2.5. Proof of Theorem 3

Assume H is the min-entropy and let f be the identity. Then, Equations (11) and () obviously hold. Reversely, let the second condition of the theorem hold. Cond. (2f) and Equation () imply $H\left(\right\{1\left\}\right)=0$ and thus $S\left(\right\{1\left\}\right)=1$. Let

$${\widehat{\partial }}_i^{n}p={\displaystyle \frac{1}{1-p_i^{2^n}}}{\left({\partial }_i^{n}p\right)}_{\widehat{i}},$$

${\widehat{\partial }}_i^{0}p=p_{\widehat{i}}/(1-p_i)$ and $S=e^{-H}$. Equation () yields

$$\begin{array}{c}\hfill S\left({\widehat{\partial }}_1^{0}p\right)=S\left({\widehat{\partial }}_1^{n}p\right)\prod _{k=0}^{n-1}(1+f\left(p_1^{2^k}\right))=S\left(g_{2^n,p_1}\right)S\left({\widehat{\partial }}_1^{0}p\right)\prod _{k=0}^{n-1}(1+f\left(p_1^{2^k}\right)),p_1<1,\end{array}$$

(47)

by iteration. By Equation () follows $f\left(0\right)=0$ and f has to be monotonously increasing. Assume $f\left(a\right)=0$ for some $a\in (0,1]$. Then, $f\left(p_1\right)=0$ for all $0\le p_1\le a$ and, by Equation (11),

$$S\left(p\right)=S\left({\widehat{\partial }}_{1}p\right)=S\left({\widehat{\partial }}_1^{2}p\right)=\dots =S\left({\widehat{\partial }}_1^{\infty }p\right)=S\left(g_{p_1}\right)S\left({\widehat{\partial }}_1^{0}p\right),p_1\le a,p_1<1.$$

Letting $p=g_{1-a}$ we get $S\left(g_a\right)=1$, which contradicts Cond. 2e. Now, Equation (47) yields

$$\begin{array}{c}\hfill S\left(g_{2^n,a}\right)\prod _{k=0}^{n-1}(1+f\left(a^{2^k}\right))=1,\end{array}$$

so that

$$\begin{array}{ccc}\hfill S\left(g_a\right)& =& 1/\prod _{k=0}^{\infty }(1+f\left(a^{2^k}\right))\in (0,1),a\in (0,1).\hfill \end{array}$$

(48)

Henceforth, we assume that $a\in (0,1)$. Since ${\widehat{\partial }}_1{b}_a=b_{1/(1+a)}$, we have, by Equation ()

$$\begin{array}{c}\hfill (1+f\left(a\right))S\left(b_{1/(1+a)}\right)=1,\end{array}$$

and

$$\begin{array}{ccc}\hfill S\left(b_a\right)& =& \left\{\begin{array}{cc}{\left(1+f\left(\right(1-a)/a)\right)}^{-1},\hfill & a\ge 1/2\hfill \\ {\left(1+f(a/(1-a\left)\right)\right)}^{-1},\hfill & a<1/2\hfill \end{array}\right..\hfill \end{array}$$

(49)

Equation (11) yields

$$\begin{array}{c}\hfill \Phi \left(p_1\right)\vee S\left({\partial }_1^{\infty }p\right)=S\left(p\right),\end{array}$$

where the limit $\Phi \left(a\right):={\bigvee }_{k=0}^{\infty }f\left(a^{2^k}\right)$ exists since f is monotonously increasing and thus implies $\Phi =f$. Hence, by Equation (19),

$$\begin{array}{ccc}\hfill S\left(p\right)& =& S\left({\partial }_1^{\infty }p\right)\vee f\left(p_1\right)=S\left(g_{p_1}\right)S\left({\widehat{\partial }}_1^{0}p\right)\vee f\left(p_1\right),\hfill \end{array}$$

(50)

in particular,

$$\begin{array}{ccc}\hfill S\left(g_{1-a}\right)& =& S\left(g_a\right)S\left(g_{1-a}\right)\vee f\left(a\right).\hfill \end{array}$$

(51)

Since $S\left(g_a\right)<1$, it follows from Equation (51) that

$$\begin{array}{c}\hfill S\left(g_a\right)=f(1-a).\end{array}$$

(52)

Thus, the recursion formula (50) determines H uniquely as soon as f is unique. Equations (52) and (48) yield

$$f(1-a)\prod _{k=0}^{\infty }(1+f\left(a^{2^k}\right))=1,$$

so that in analogy to Equation (14)

$$\begin{array}{ccc}\hfill (1+f(a\left)\right)f(1-a)& =& (1+f\left(a\right))f(1-a)f(1-a^2)\prod _{k=1}^{\infty }(1+f\left(a^{2^k}\right))\hfill \\ & =& f(1-a^2).\hfill \end{array}$$

(53)

Now, Equations (50), () and (52) yield

$$\begin{array}{c}\hfill S\left(G_{n,a}\right)=S\left({\partial }_1^{\infty }{G}_{n,a}\right)\vee f\left(a^n\right)=S\left(g_a\right)\vee f\left(a^n\right)=f(1-a)\vee f\left(a^n\right).\end{array}$$

On the other hand, by Equation () and iterating Equation (50) follows

$$\begin{array}{ccc}\hfill S\left(G_{n,a}\right)& =& f(1-a)\vee S\left(g_{1-a}\right)S\left(G_{n-1,a}\right)\hfill \\ & =& f(1-a)\vee S^n\left(g_{1-a}\right)\hfill \\ & =& f(1-a)\vee f^n\left(a\right).\hfill \end{array}$$

Hence,

$$f\left(a^n\right)=f^n\left(a\right),\mathrm{if}1-a<a^n.$$

In particular,

$$f\left(a^2\right)=f^2\left(a\right)\mathrm{for}a>(\sqrt{5}-1)/2,$$

and ${lim}_{a\to 1}f\left(a\right)$ is necessarily 1. We have

$$\begin{array}{c}\hfill S\left(b_a\right)=S\left({\partial }_2{b}_a\right)\vee f(1-a)=(S\left(g_a\right)S\left(b_a\right)\vee f\left(a\right))\vee f(1-a)=f\left(a\right)\vee f(1-a).\end{array}$$

(54)

Equations (49) and (54) and the monotonicity of f entail

$$\begin{array}{cc}\hfill f\left(a\right)(1+f(-1+1/a\left)\right)=1,& a\ge 1/2.\end{array}$$

In particular $f(1/2)=1/2$. Equation (53) delivers $f(3/4)=3/4>(\sqrt{5}-1)/2$. Hence, for $a=3/4$, we have $f\left(a^{2^{-k}}\right)=a^{2^{-k}}$ and $a^{2^{-k}}\to 1$ as $k\to \infty$. Now, Lemma 3 yields the uniqueness of f.

3. Discussion & Conclusions

The characterization given here is not only novel with respect to splitting the chain rule into its natural components, it is also novel in the sense that only a rather weak relationship is required for the chain rule, i.e. the proportionality factor itself is not given, but arises naturally. Furthermore, the recursion formula has the advantage that it allows an interpretation as an intrinsic property, since only a single distribution is involved and since recursion formulae in general may allow additional practical interpretations such as self-similarity or scale invariance.

In comparison to Faddeev’s approach, our proofs emphasize the role of the geometric distribution $g_a$, which arises as the infinite iteration of the Bernoulli distribution $b_a$, $a\in (0,1)$, i.e. ${\partial }_1^{\infty }{b}_a=g_a$. The uniqueness proof for the Shannon entropy uses almost only the resulting proportionality relation of the geometric distribution, simplifying Faddeev’s approach. Further, while Equation (2) is a rule, proportionality can be considered a law. This yields a deeper understanding of the Shannon entropy and may justify a broader application outside telecommunication.

The proofs rely strongly on distributions that generalize the uniform distribution and the geometric distribution, respectively. Possibly, these distributions play or will play a role also in other areas.

Although the characterization and the proofs of the Shannon and the Rényi entropy follow the same scheme, they show remarkable differences. For instance, the generalization $G_{n,a}$ of the geometric distribution, see Equation (17), does not play a role in case of the Shannon entropy. Also, the functional characterization in Equation (9) of the Shannon entropy is not the limit of the characterization in Equation (10) of the Rényi entropy. However, Equation (9) might be understood as the derivative of Equation (10) in the following sense, cf. [18]. Assume that f in Equation (10) is differentiable. Then,

$${\displaystyle \frac{d}{d\alpha }}S\left({\partial }_{1}p\right)={\displaystyle \frac{d}{d\alpha }}\left((1+f\left(p_1\right))S\left(p\right)-f\left(p_1\right)\right).$$

For $\alpha =1$ we define $S\left(p\right)={lim}_{\alpha \to 1}{\parallel p\parallel }_{\alpha }^{\alpha }$. Then, we get

$${\displaystyle \frac{d}{d\alpha }}S\left({\partial }_{1}p\right)=(1+f\left(p_1\right)){\displaystyle \frac{d}{d\alpha }}S\left(p\right).$$

Another interpretation of our results is that S is not a natural transformation of the Rényi entropy, in contrast to

$$T\left(p\right)=S\left(p\right)-S\left(\right\{1\left\}\right).$$

This definition bridges to the generalized information function [Section 6.2 [5] and to the definite functions, cf. [19], Chapter 3, Corollary 3.3 and proof of Theorem 2.2, and Chapter 6, Example 5.16. Indeed, Equation (10) can be rewritten in full analogy to Equation (9), namely as

$$\begin{array}{c}\hfill T\left({\partial }_{1}p\right)=(1+f\left(p_1\right))T\left(p\right),\end{array}$$

where $T\left({\partial }_{1}p\right)=S\left({\partial }_{1}p\right)-S\left(\left\{1\right\}\right)$. Note that also the interpretation $T\left({\partial }_{1}p\right)=S\left({\partial }_{1}p\right)-S\left({\partial }_1\left\{1\right\}\right)$ is possible.

Several technical assumptions had to be included in the characterization of the Rényi entropy, such as the condition that f shall be real analytic. Although we consider this fact as minor, we hope that these assumptions can be dropped in the future.