Discretization of Dynamical Systems Based on Observations

1. Introduction

Let $\mathfrak{S}=\left(M,\mathcal{M},\mu ,\nu ,T\right)$ be dynamical system on a compact subset $M$ of metric space $R$ with metric $\nu$ and additive Lebesgue measure $\mu$, where $\mathcal{M}$ is a $\sigma$-algebra of the subsets of $R$ and $T:M\to M$ is an endomorphism.

Assume that in the $\sigma$-algebra $\mathcal{M}$ are distinguished $m$ sets $X_1,X_2,\dots ,X_m\subset M$ which represent observations of the system and assume that these observations form a partition $\xi =\left\{X_i\right\}$ of the set $M$.

Then, using observations’ partition $\xi$ it is required to define a real number $\epsilon >0$ and partition ${\xi }_{\epsilon }=\left\{X_i|i\in \mathbb{N}\right\}$ of the set $M$ such that the diameters of the sets $X_i\in {\xi }_{\epsilon }$ are $d\left(X_i\right)=\underset{x,y\in X_i}{\sup }\nu \left(x,y\right)\le 2\epsilon$, which represents the system “as good as possible”.

Before formulating strict criterion, let us clarify the problem by the following examples.

Assume that $M=\left[x_0,x_m\right]\in \mathbb{R}$ is an interval of real numbers and assume that observations of the interval are represented by the partition $\xi =\left\{X_i\right\}$, which includes intervals $X_1=\left[x_0,x_1\right]$, $X_2=\left(\left.x_1,x_2\right]\right.$, $X_3=\left(\left.x_2,x_3\right]\right.$, …, $X_m=\left(\left.x_{m-1},x_m\right]\right.$ such that $x_j\in M$, $j=0,1,2,\dots ,m$, are drawn with respect to certain unknown distribution. Using the numbers $x_j$, $j=0,1,2,\dots ,m$, it is needed to plot a histogram, which represents this distribution.

This problem requires definition of the number of bins or the bin length such that the histogram “as good as possible” represents the distribution of the observed values. Despite existence of several formulas for calculating bin lengths, there is no strictly proven criteria for estimating goodness of the histogram.

From the other point of view, the problem can be considered in the terms of discretization of stochastic processes [5]. In this case, given observed realization of the process it is required to define a sequence $t_i$, $i=0,1,2,\dots$, of times, $2\epsilon =t_{i+1}-t_i$, $t_i=i\mathsf{\Delta }$, such that the intervals $\mathsf{\Delta }=2\epsilon$ between the values $x_i=x\left(t_i\right)$ “as good as possible” represent the process.

To formulate the criterion of consistency between the observations partition $\xi$ and the partition ${\xi }_{\epsilon }$, denote ${\xi }_{\epsilon }^n=\underset{i=0}{\overset{n}{\bigvee }}{T}^i{\xi }_{\epsilon }$ and ${\xi }^n=\underset{i=0}{\overset{n}{\bigvee }}{T}^i\xi$, where $\vee$ stands for multiplication of the partitions.

Given observations’ partition $\xi$, we say that partition ${\xi }_{\epsilon }$ represents dynamical system $\mathfrak{S}$ if a distance $dist\left({\xi }_{\epsilon }^n,{\xi }^n\right)$ between partitions ${\xi }_{\epsilon }^n$ and ${\xi }^n$ as a function of $\epsilon$ is bounded that is

$$a\le dist\left({\xi }_{\epsilon }^n,{\xi }^n\right)\le b,$$

for some constant numbers $a$ and $b$, $a\le b$, and that partition ${\xi }_{\epsilon }$ represents dynamical system $\mathfrak{S}$ “as good as possible” if

$$dist\left({\xi }_{\epsilon }^n,{\xi }^n\right)=const.$$

In the paper, we suggest a method for defining partition ${\xi }_{\epsilon }$. We assume that $M$ is a compact set with a metric $\nu :M\to \mathbb{R}$ and together with the Kolmogorov-Sinai entropy $h\left(T\right)$ of dynamical system we consider the Kolmogorov $\epsilon$-entropy and $\epsilon$-capacity [6]. Then, we define a diameter $2\epsilon$ of the sets $X_i$ that results in the required partition ${\xi }_{\epsilon }$.

The suggested method is also useful for definition of the bin lengths of histograms; in the paper, we apply the method to different datasets and compare it with the bin lengths obtained by the known methods.

2. Methods

We follow the line of the Schwarz information criterion [10] and apply the Kolmogorov $\epsilon$-entropy and $\epsilon$-capacity [6] (see also [2, 15]). Here we recall the required definitions and facts which are used in the solution of the problem.

Let $\mathfrak{S}=\left(M,\mathcal{M},\mu ,\nu ,T\right)$ be a dynamical system. We assume that $M$ is a compact subset of metric space $R$ with metric $\nu$ and additive Lebesgue measure $\mu$ such that $\mu \left(M\right)=1$; $\mathcal{M}$ is a $\sigma$-algebra of the subsets of $R$ and $T:M\to M$ is an endomorphism.

Let $\beta =\left\{B_i\right\}$, $B_i\in \mathcal{M},$ $\bigcup _i{B}_i=M$, $B_i\bigcap B_j=\mathrm{\varnothing }$ for $i\ne j$, $i,j\in \mathbb{N}$, be a partition of the set $M$.

Definition 1 [12,13]. A value

$${\mathcal{H}}_{\mu }\left(\beta \right)=-{\sum }_i\mu \left(B_i\right){\log }_2\mu \left(B_i\right)$$

is called entropy of the partition $\beta$.

Let $\beta =\left\{B_i|B_i\in \mathcal{M}\right\}$ and $\gamma =\left\{C_j|C_j\in \mathcal{M}\right\}$ be two partitions of the set $M$.

Definition 2 [12,13]. A value

$${\mathcal{H}}_{\mu }\left(\beta |\gamma \right)=-{\sum }_j\mu \left(C_j\right){\sum }_i{\log }_2\mu \left(B_i|C_j\right)$$

is called conditional entropy of the partition$\beta$relatively to the partition$\gamma$.

Definition 3 [9,12,13]. A value

$$dist\left(\beta ,\gamma \right)={\mathcal{H}}_{\mu }\left(\beta |\gamma \right)+{\mathcal{H}}_{\mu }\left(\gamma |\beta \right)$$

is called Rokhlin distance between the partitions$\beta$ and $\gamma$.

For any endomorphism $T$ and any partition $\beta$, denote by $T\beta$ a partition of $M$ to the sets $\left\{T{B}_i\right\}$, that is $T\beta =\left\{T{B}_i\right\}$.

Let $\beta \vee \gamma =\left\{B_i\cap C_j|B_i\in \beta ,C_i\in \gamma ,i,j\in \mathbb{N}\right\}$ be multiplication of the partitions $\beta =\left\{B_i|B_i\in \mathcal{M}\right\}$ and $\gamma =\left\{C_j|C_j\in \mathcal{M}\right\}$.

Definition 4 [12,13]. A value

$$h\left(\beta ,T\right)=\underset{n\to \mathrm{\infty }}{\lim }{\displaystyle \frac{1}{n}}{\mathcal{H}}_{\mu }\left(\beta \vee T\beta \vee T^2\beta \vee \dots \vee T^n\beta \right)$$

is called entropy per symbol of partition or entropy per unit of time.

Definition 5 [12,13]. A value

$$h\left(T\right)=\underset{\beta }{\sup }h\left(\beta ,T\right),$$

where supremum is taken over all finite or countable measurable partitions$\beta$of$M$such that${\mathcal{H}}_{\mu }\left(\beta \right)<\infty$, is called entropy of dynamical system.

Entropy $h\left(T\right)$ is an invariant of dynamical system, but direct calculation of the entropy $h\left(T\right)$ is possible only for certain endomorphisms; for examples see [12, chapter 15] and [13, chapter 7].

Theorem 1 [1,12,13]. If $T$ is invertible and $\mathcal{M}=\underset{n=-\infty }{\overset{\infty }{\bigvee }}{T}^n{\xi }^{*}$, then $h\left({\xi }^{*},T\right)=h\left(T\right)$.

Corollary 1 [1]. If $\mathcal{M}=\underset{n=0}{\overset{\mathrm{\infty }}{\bigvee }}{T}^n{\xi }^{\mathrm{*}}$, then $h\left({\xi }^{\mathrm{*}},T\right)=h\left(T\right)$.

Let $\epsilon >0$ be a real number.

Set $\alpha =\left\{A:A\subset R\right\}$ is called $\epsilon$-covering of the set $M$, if $M\subseteq \bigcup _{A\in \alpha }A$ and diameter of any $A\in \alpha$ is not greater than $2\epsilon$.

Set $M$ is said to be $\epsilon$-distinguishable, if any two of its distinct points are located at distance greater than $\epsilon$.

Lemma 1 [6]. Given a compact set $M$, for any $\epsilon >0$ there exists a finite $\epsilon$-covering of $M$. Also, for any $\epsilon >0$ any $\epsilon$-distinguishable set $M\subset R$ is finite.

Denote by ${\mathcal{N}}_{\epsilon }\left(M\right)$ a minimal number of sets in $\epsilon$-covering $\alpha$ of the set $M$, and by ${\mathcal{M}}_{\epsilon }\left(M\right)$ a maximal number of points in an $\epsilon$-distinguishable subset of the set $M$.

Definition 6 [6]. A value

$${\mathcal{H}}_{\epsilon }\left(M\right)={\log }_2{\mathcal{N}}_{\epsilon }\left(M\right)$$

is called $\epsilon$-entropy of the set $M$ and a value

$${\mathcal{E}}_{\epsilon }\left(M\right)={\log }_2{\mathcal{M}}_{\epsilon }\left(M\right)$$

is called $\epsilon$-capacity of the set $M$.

These values are interpreted as follows: $\epsilon$-entropy ${\mathcal{H}}_{\epsilon }\left(M\right)$ is a minimal number of bits required to transmit the set $M$ with the precision $\epsilon$, and $\epsilon$-capacity ${\mathcal{E}}_{\epsilon }\left(M\right)$ is a maximal number of bits, which can be memorized by $M$ with the precision $\epsilon$.

We will use the following property of $\epsilon$-entropy ${\mathcal{H}}_{\epsilon }\left(M\right)$ and $\epsilon$-capacity ${\mathcal{E}}_{\epsilon }\left(M\right)$.

Theorem 2 [6]. Given a compact set $M$, both $\epsilon$-entropy and $\epsilon$-capacity are non- increasing functions of $\epsilon$.

Finally, we will use the following property of the entropy of partitions.

Let $\beta =\left\{B:B\subset M\right\}$ and $\gamma =\left\{C:C\subset M\right\}$ be two partitions of the set $M$. If each set $C\in \gamma$ is a subset of some set $B\in \beta$, then it is said that partition $\gamma$ is a refinement of the partition $\beta$, which is written as $\gamma \succcurlyeq \beta$.

Lemma 2 [12]. If $\beta \preccurlyeq \gamma$, then ${\mathcal{H}}_{\mu }\left(\beta \right)\le {\mathcal{H}}_{\mu }\left(\gamma \right)$.

Let $\beta \vee \gamma$ be multiplication of the partitions $\beta$ and $\gamma$. Since each set $D\in \beta \vee \gamma$ is a subset of some set $B\in \beta$ and of some set $C\in \gamma$, partition $\beta \vee \gamma$ is a refinement of both $\beta$ and $\gamma$, that is $\beta \vee \gamma \succcurlyeq \beta$ and $\beta \vee \gamma \succcurlyeq \gamma$.

Thus, following lemma 2, ${\mathcal{H}}_{\mu }\left(\beta \right)\le {\mathcal{H}}_{\mu }\left(\beta \vee \gamma \right)$ and ${\mathcal{H}}_{\mu }\left(\gamma \right)\le {\mathcal{H}}_{\mu }\left(\beta \vee \gamma \right)$.

For other properties of the entropy ${\mathcal{H}}_{\mu }$ and its application for analysis of dynamical systems see the paper [9] and the books [7,13,15].

3. Suggested Solution

Let $M$ is a compact subset of metric space $R$ with metric $\nu$ and additive Lebesgue measure $\mu$ such that $\mu \left(M\right)=1$; $\mathcal{M}$ is a $\sigma$-algebra of the subsets of $R$.

Let $\beta =\left\{B_i|B_i\subset M\right\}$, $\bigcup _i{B}_i=M$, $B_i\bigcap B_j=\mathrm{\varnothing }$ for $i\ne j$, $i,j\in \mathbb{N}$, be a partition of $M$. If diameter $d\left(B_i\right)=\underset{x,y\in B_i}{\mathit{sup}}\nu \left(x,y\right)$ of each $B_i\in \beta$ is not greater than $2\epsilon$, then partition $\beta$ is an $\epsilon$-covering and is called $\epsilon$-partition.

Lemma 3. Let $\beta$ be $\epsilon$-partition of $M$. If diameter $d\left(M\right)=\underset{x,y\in M}{\mathit{sup}}\nu \left(x,y\right)=1$ and measure $\mu \left(B\right)=\underset{x,y\in B}{\mathit{sup}}\nu \left(x,y\right)$, $B\in \beta$, then ${\mathcal{H}}_{\mu }\left(\beta \right)\ge {\mathcal{H}}_{\epsilon }\left(M\right)$.

Proof. Consider entropy

$${\mathcal{H}}_{\epsilon }\left(M\right)={\log }_2{\mathcal{N}}_{\epsilon }\left(M\right)=-\sum {\displaystyle \frac{1}{{\mathcal{N}}_{\epsilon }\left(M\right)}}{\log }_2{\displaystyle \frac{1}{{\mathcal{N}}_{\epsilon }\left(M\right)}}.$$

Given $\epsilon$, a minimal number ${\mathcal{N}}_{\epsilon }\left(M\right)$ of sets in $\beta$ is reached if diameters $d\left(B\right)$ of its elements $B\in \beta$ are maximal that is $d\left(B\right)=2\epsilon$. That is possible either if ${\displaystyle \frac{d\left(M\right)}{d\left(B\right)}}={\displaystyle \frac{1}{2\epsilon }}$ subsets from $\beta$ completely cover $M$, or if $⌊{\displaystyle \frac{d\left(M\right)}{d\left(B\right)}}⌋$ subsets cover a part $M\text{'}$ of $M$ such that $d\left(M\backslash M\text{'}\right)<2\epsilon$ and to cover this part it is required one additional subset, where $⌊x⌋$ denotes maximal integer number smaller than or equal to $x$. In this case, the number of elements in $\beta$ is $⌊{\displaystyle \frac{d\left(M\right)}{d\left(B\right)}}⌋+1$.

Then, if ${\displaystyle \frac{1}{2\epsilon }}$ is integer,

$${\mathcal{N}}_{\epsilon }\left(M\right)={\displaystyle \frac{d\left(M\right)}{d\left(B\right)}}={\displaystyle \frac{1}{2\epsilon }};$$

otherwise

$${\mathcal{N}}_{\epsilon }\left(M\right)=⌊{\displaystyle \frac{d\left(M\right)}{d\left(B\right)}}⌋+1=⌊{\displaystyle \frac{1}{2\epsilon }}⌋+1.$$

Substituting these values into the definition of entropy, in the first case one obtains

$${\mathcal{H}}_{\epsilon }\left(M\right)=-{\sum }_{i=1}^{1/\left(2\epsilon \right)}{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{1/\left(2\epsilon \right)}},$$

and in the second case –

$${\mathcal{H}}_{\epsilon }\left(M\right)=-{\sum }_{i=1}^{⌊1/\left(2\epsilon \right)⌋+1}{\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}=$$

$$=-{\sum }_{i=1}^{⌊1/\left(2\epsilon \right)⌋}{\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}-{\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}.$$

Now consider entropy

$${\mathcal{H}}_{\mu }\left(\beta \right)=-{\sum }_{B\in \beta }\mu \left(B\right){\log }_2\mu \left(B\right).$$

If number ${\displaystyle \frac{1}{2\epsilon }}$ is integer, then

$${\mathcal{H}}_{\mu }\left(\beta \right)=-{\sum }_{i=1}^{1/\left(2\epsilon \right)}2\epsilon \times {\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(2\epsilon \right)=-{\sum }_{i=1}^{1/\left(2\epsilon \right)}{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{1/\left(2\epsilon \right)}},$$

otherwise

$${\mathcal{H}}_{\mu }\left(\beta \right)=-{\sum }_{i=1}^{⌊1/\left(2\epsilon \right)⌋+1}2\epsilon \times {\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(2\epsilon \right)==-{\sum }_{i=1}^{⌊1/\left(2\epsilon \right)⌋}{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}-{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}.$$

Thus, if number ${\displaystyle \frac{1}{2\epsilon }}$ is integer, then ${\mathcal{H}}_{\epsilon }\left(M\right)={\mathcal{H}}_{\mu }\left(\beta \right)$.

Consider the case of non-integer ${\displaystyle \frac{1}{2\epsilon }}$. For any $\epsilon$ such that number ${\displaystyle \frac{1}{2\epsilon }}$ is not integer, holds

$${\displaystyle \frac{1}{⌊1/\left(2\epsilon \right)⌋+1}}<{\displaystyle \frac{1}{1/\left(2\epsilon \right)}}.$$

Thus, because of monotonicity of the $log$ function, ${\mathcal{H}}_{\mu }\left(\beta \right)\ge {\mathcal{H}}_{\epsilon }\left(M\right)$. ■

Note that lemma 3 is also true for a measure $\mu$ and a metric $\nu$ such that $\mu \left(M\right)=d\left(M\right)$.

From the proof of lemma 3 follows the next fact.

Corollary 2. ${\mathcal{H}}_{\mu }\left(\beta \right)={\mathcal{H}}_{\epsilon }\left(M\right)+c\left(\epsilon \right)$, where $c\left(\epsilon \right)=-2\epsilon {\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(2\epsilon \right)$ monotonously increases with $\epsilon$ up to a single maximum and then monotonously decreases.

In addition, let us consider $\epsilon$-entropy of multiplication of partitions.

Let $\beta =\left\{B_i|B_i\subset M\right\}$ and $\gamma =\left\{C_j|C_j\subset M\right\}$ be two $\epsilon$-partitions of the set $M$ and let $\beta \vee \gamma =\left\{D_k|D_k\subset M\right\}$ be multiplication of the partitions $\beta$ and $\gamma$.

Lemma 4. Let $d\left(B_i\right)\le 2{\epsilon }_B$, $B_i\in \beta$, and $d\left(C_j\right)\le 2{\epsilon }_C$, $C_j\in \gamma$, and let $\epsilon =\mathit{max}\left\{{\epsilon }_B,{\epsilon }_C\right\}$. Then $\beta \vee \gamma$ is $\epsilon$-partition with $d\left(D_k\right)\le 2{\epsilon }_D\le \mathit{max}\left\{{\epsilon }_B,{\epsilon }_C\right\}$, $D_k\in \beta \vee \gamma$, and

$${\mathcal{H}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{H}}_{\min \left\{{\epsilon }_B,{\epsilon }_C\right\}}\left(M\right).$$

Proof. Since $\beta \vee \gamma \succcurlyeq \beta$, holds ${\mathcal{N}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{N}}_{{\epsilon }_B}\left(M\right)$ and ${\mathcal{H}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{H}}_{{\epsilon }_B}\left(M\right)$; and since $\beta \vee \gamma \succcurlyeq \gamma$, holds ${\mathcal{N}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{N}}_{{\epsilon }_C}\left(M\right)$ and ${\mathcal{H}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{H}}_{{\epsilon }_C}\left(M\right)$.

Assume that ${\epsilon }_B\ge {\epsilon }_C$. Then, since ${\mathcal{N}}_{{\epsilon }_B}\left(M\right)\le {\mathcal{N}}_{{\epsilon }_C}\left(M\right)$, holds ${\mathcal{H}}_{{\epsilon }_B}\left(M\right)\le {\mathcal{H}}_{{\epsilon }_C}\left(M\right)$, and consequently ${\mathcal{H}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{H}}_{{\epsilon }_C}\left(M\right)\ge {\mathcal{H}}_{{\epsilon }_B}\left(M\right)$.

On the other hand, if ${\epsilon }_C\ge {\epsilon }_B$, then ${\mathcal{N}}_{{\epsilon }_C}\left(M\right)\le {\mathcal{N}}_{{\epsilon }_B}\left(M\right)$ and ${\mathcal{H}}_{{\epsilon }_C}\left(M\right)\le {\mathcal{H}}_{{\epsilon }_B}\left(M\right)$. Thus ${\mathcal{H}}_{{\epsilon }_D}\left(M\right)\ge {\mathcal{H}}_{{\epsilon }_B}\left(M\right)\ge {\mathcal{H}}_{{\epsilon }_C}\left(M\right)$. ■

The suggested method is based on the concept of $\epsilon$-information which is defined as follows.

Let $\xi =\left\{X_i|i=1,2,\dots ,m\right\}$ be a partition of the set $M$ obtained by observations and let ${\epsilon }_m>0$ be a radius that provides maximal $\epsilon$-entropy ${\mathcal{H}}_{{\epsilon }_m}\left(M\right)$ with respect to $\xi$.

Radius ${\epsilon }_m$ depends on the diameter of the set $M$ and the structure of the partition $\xi$ and cannot be specified directly. Nevertheless, we can prove the following fact.

Lemma 5.

$${\mathcal{H}}_{{\epsilon }_m}\left(M\right)={\log }_{2}m.$$

Proof. Denote by ${\xi }_{{\epsilon }_m}=\left\{Y_j|j=1,2,\dots ,m_{{\epsilon }_m}\right\}$ partition of $M$ such that $d\left(Y_j\right)=\underset{x,y\in Y}{\mathit{sup}}\nu \left(x,y\right)=2{\epsilon }_m$ for each $Y_j\in {\xi }_{{\epsilon }_m}$. In other words, partition ${\xi }_{{\epsilon }_m}$ splits the set $M$ to the subsets with equal diameters $2{\epsilon }_m$.

Then ${\epsilon }_m$ is a radius such that an average number of bits in the description of the elements of $\xi$ using the elements of ${\xi }_{{\epsilon }_m}$ is maximal. Formally it means that for the radius ${\epsilon }_m$ holds

$${\displaystyle \frac{1}{n}}{\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\bigvee \xi \right)\to max,$$

where $n=\left|{\xi }_{{\epsilon }_m}\bigvee \xi \right|$ is a number of subsets in the partition ${\xi }_{{\epsilon }_m}\bigvee \xi$.

Note that the number of elements in the partition ${\xi }_{{\epsilon }_m}\bigvee \xi$ is $n\ge \max \left(m,m_{{\epsilon }_m}\right)$, and that the function $f\left(x\right)={\displaystyle \frac{1}{x}}{\log }_{2}x$ has a single maximum in the point $x=e$.

Hence, given partition $\xi$ of the size $m$, maximum of the entropy ${\displaystyle \frac{1}{n}}{\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\bigvee \xi \right)$ is reached for partition ${\xi }_{{\epsilon }_m}$ of the size $m_{{\epsilon }_m}=m$. Diameters of the elements $Y_j\in {\xi }_{{\epsilon }_m}$ of this partition are

$$d\left(Y_j\right)=2{\epsilon }_m={\displaystyle \frac{d\left(M\right)}{m}},j=1,2,\dots ,m.$$

Recall that the measure of the set $M$ is $\mu \left(M\right)=1$. Since partition ${\xi }_{{\epsilon }_m}$ includes $m$ elements with equal diameters, the measures of these elements are $\mu \left(Y_j\right)={\displaystyle \frac{1}{m}}$. Thus, entropy of the partition ${\xi }_{{\epsilon }_m}$ is

$${\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\right)=-{\sum }_{j=1}^m\mu \left(Y_j\right){\log }_2\mu \left(Y_j\right)=-{\sum }_{j=1}^m{\displaystyle \frac{1}{m}}{\log }_2{\displaystyle \frac{1}{m}}={\log }_{2}m.$$

Finally, since partition ${\xi }_{{\epsilon }_m}$ includes the elements with equal diameters $2{\epsilon }_m$, this partition is $\epsilon$-partition of $M$ and for given ${\epsilon }_m$ its $\epsilon$-entropy is

$${\mathcal{H}}_{{\epsilon }_m}\left(M\right)={\log }_{2}m,$$

which is required. ■

Let $\beta =\left\{B:B\subset M\right\}$ be a partition of the set $M$. Similar to definition 6, denote by ${\mathcal{N}}_{\epsilon }\left(\beta \right)$ a number of sets $B\in \beta$ such that $d\left(B\right)\le 2\epsilon$.

Definition 7. A value

$${\mathcal{H}}_{\epsilon }\left(\beta \right)={\log }_2{\mathcal{N}}_{\epsilon }\left(\beta \right)$$

is called$\epsilon$-entropy of the partition $\beta$.

Let ${\mathcal{H}}_{{\epsilon }_m}\left(M\right)$ be maximal $\epsilon$-entropy of the set $M$ with respect to the observations $\xi =\left\{X_i|i=1,2,\dots ,m\right\}$ and let $\epsilon >0$ be a radius; recall that radius $\epsilon$ represents certain precision of coding the set $M$ (see definition 6 and remarks after it). Then

$${\mathcal{H}}_{{\epsilon }_m}\left(M\right)-{\mathcal{H}}_{\epsilon }\left(M\right)$$

is the difference between minimal number of bits required to transmit the set $M$ with the precision ${\epsilon }_m$ provided by observations and minimal number of bits required to transmit the set $M$ with the desired precision $\epsilon$.

This difference can be negative which indicates that the observations do not allow transmitting the set $M$ with the desired precision.

Similar to the proof of lemma 5, denote by ${\xi }_{\epsilon }=\left\{Y|Y\subset M\right\}$ partition of $M$ such that $d\left(Y\right)=\underset{x,y\in Y}{\mathit{sup}}\nu \left(x,y\right)=2\epsilon$ for each $Y\in {\xi }_{\epsilon }$. Then

$${\mathcal{H}}_{{\epsilon }_m}\left(M\right)-{\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$$

is the difference between minimal number of bits required to transmit the set $M$ with the precision ${\epsilon }_m$ provided by observations and the number of bits required to transmit the observations $\xi$ of the set $M$ using the partition ${\xi }_{\epsilon }$.

The sum of these differences results in the following definition.

Definition 8. A value

$${\mathcal{I}}_{\epsilon }\left(M\right)=2{\mathcal{H}}_{{\epsilon }_m}\left(M\right)-{\mathcal{H}}_{\epsilon }\left(M\right)-{\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$$

is called$\epsilon$-information of the set$M$.

In the formula of $\epsilon$-information ${\mathcal{I}}_{\epsilon }\left(M\right)$, the value ${\mathcal{H}}_{{\epsilon }_m}\left(M\right)$ represents minimal number of bits required to transmit the set $M$ with maximal precision allowed by the observations $\xi$, the second term ${\mathcal{H}}_{\epsilon }\left(M\right)$ represents minimal number of bits required to transmit the set $M$ with precision $\epsilon$, and the last term ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$ represents the number of bits required to transmit the results of observations of the $M$ measured using ${\xi }_{\epsilon }$.

Thus, the value ${\mathcal{I}}_{\epsilon }\left(M\right)$ is the number of bits remaining after the transmitting the observed part of the set $M$ with the precision $\epsilon$.

Theorem 3. Let $\xi$ be the observations’ partition of the size $m$. Then, $\epsilon$-information ${\mathcal{I}}_{\epsilon }\left(M\right)$ as a function of $\epsilon$ increases and its upper bound is ${\mathit{log}}_{2}m$.

Proof. In the definition of $\epsilon$-information the first term $2{\mathcal{H}}_{{\epsilon }_m}\left(M\right)$ is constant and the second term ${\mathcal{H}}_{\epsilon }\left(M\right)$, by theorem 2, does not increase with increasing $\epsilon$. Thus, it remains to prove that the last term ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$ decreases with increasing $\epsilon$.

Since diameter of each $Z\in \xi \vee {\xi }_{\epsilon }$ is $d\left(Z\right)\le 2\epsilon$, the number ${\mathcal{N}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$ is equivalent to the number $n=\left|\xi \vee {\xi }_{\epsilon }\right|$ of elements in the partition $\xi \vee {\xi }_{\epsilon }$, and so $\epsilon$-entropy is

$${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)={\log }_{2}n.$$

The number $n$ of elements in the partition $\xi \vee {\xi }_{\epsilon }$ decreases (not necessarily monotonously) with increasing $\epsilon$ down to the number $m=\left|\xi \right|$ of elements in the partition $\xi$. Hence, entropy ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$ also decreases down to the value ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)={\log }_{2}m$, which results in increasing of $\epsilon$-information ${\mathcal{I}}_{\epsilon }\left(M\right)$ at maximum up to the value $2{\log }_{2}m-{\log }_{2}m={\log }_{2}m$. ■

Let us return to the initial problem.

Let $\mathfrak{S}=\left(M,\mathcal{M},\mu ,\nu ,T\right)$ be a dynamical system, let $\xi =\left\{X_i|X_i\subset \mathcal{M},i=1,2,\dots ,m\right\}$ be an observations’ partition, and ${\xi }_{\epsilon }^n=\underset{i=0}{\overset{n}{\bigvee }}{T}^i{\xi }_{\epsilon }$ and ${\xi }^n=\underset{i=0}{\overset{n}{\bigvee }}{T}^i\xi$.

Theorem 4. If $T$ is inversible and

$$\left|{\mathcal{I}}_{\epsilon }\left(M\right)-{\mathcal{E}}_{\epsilon }\left(M\right)\right|=0,$$

then Rokhlin distance

$$dist\left({\xi }_{\epsilon }^n,{\xi }^n\right)={\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }^n|{\xi }^n\right)+{\mathcal{H}}_{\mu }\left({\xi }^n|{\xi }_{\epsilon }^n\right)$$

as a function of$\epsilon$is bounded.

Before proving the theorem, let us recall the following facts. Let $\beta =\left\{B_i\right\}$ and $\gamma =\left\{C_j\right\}$ be partitions of the set $M$.

Lemma 6 [12,13].

$${\mathcal{H}}_{\mu }\left(\beta \bigvee \gamma \right)={\mathcal{H}}_{\mu }\left(\beta |\gamma \right)+{\mathcal{H}}_{\mu }\left(\gamma \right)$$

Lemma 7 [12]. If endomorphism $T$ is inversible, then

$${\mathcal{H}}_{\mu }\left(T\beta |T\gamma \right)={\mathcal{H}}_{\mu }\left(\beta |\gamma \right)$$

Proof of theorem 4. Following lemma 7,

$$dist\left({\xi }_{\epsilon }^n,{\xi }^n\right)=dist\left({\xi }_{\epsilon },\xi \right)={\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }|\xi \right)+{\mathcal{H}}_{\mu }\left(\xi |{\xi }_{\epsilon }\right)$$

Appling lemma 6 for each term, we obtain

$$dist\left({\xi }_{\epsilon },\xi \right)=2{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\vee \xi \right)-{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)-{\mathcal{H}}_{\mu }\left(\xi \right)$$

The last term ${\mathcal{H}}_{\mu }\left(\xi \right)$ does not depend on $\epsilon$; hence we are interested in the difference

$$2{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\vee \xi \right)-{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)$$

Consider $\epsilon$-information ${\mathcal{I}}_{\epsilon }\left(M\right)$. From corollary 2 it follows that

$$\begin{gathered} {\mathcal{H}}_{{\epsilon }_m}\left(M\right)={\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\right)+c_1 \\ {\mathcal{H}}_{\epsilon }\left(M\right)={\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)+c_2\left(\epsilon \right) \\ {\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)={\mathcal{H}}_{\mu }\left(\xi \vee {\xi }_{\epsilon }\right)+c_3\left(\epsilon \right) \end{gathered}$$

where $c_1$ is constant and $c_2$ and $c_3$ monotonously increase with $\epsilon$ up to a single maximum and then monotonously decrease. Then

$${\mathcal{I}}_{\epsilon }\left(M\right)=2{\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\right)-{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)-{\mathcal{H}}_{\mu }\left(\xi \vee {\xi }_{\epsilon }\right)+c$$

where $c\left(\epsilon \right)=2c_1-c_2\left(\epsilon \right)-c_3\left(\epsilon \right)$ monotonously decreases down to a single minimum and then monotonously increases.

Similarly, for $\epsilon$-capacity holds

$${\mathcal{E}}_{\epsilon }\left(M\right)={\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)+c_4\left(\epsilon \right)$$

where $c_4\left(\epsilon \right)=c_2\left(\epsilon \right)+const$ monotonously increases with $\epsilon$ up to a single maximum and then monotonously decreases.

Zero difference between $\epsilon$-information and $\epsilon$-capacity leads to

$$2{\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\right)+c_5\left(\epsilon \right)={\mathcal{H}}_{\mu }\left(\xi \vee {\xi }_{\epsilon }\right)+2{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)$$

where $c_5\left(\epsilon \right)=c\left(\epsilon \right)-c_2\left(\epsilon \right)-const$ is bounded.

Since $2{\mathcal{H}}_{\mu }\left({\xi }_{{\epsilon }_m}\right)$ does not depend on $\epsilon$,

$${\mathcal{H}}_{\mu }\left(\xi \vee {\xi }_{\epsilon }\right)+2{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)=c_5\left(\epsilon \right)+const$$

Consequently, the difference

$$2{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\vee \xi \right)-{\mathcal{H}}_{\mu }\left({\xi }_{\epsilon }\right)$$

is bounded and theorem 4 is proven. ■

This theorem motivates formulation of the problem of finding a value $\mathsf{\Delta }=2\epsilon$ in the terms of $\epsilon$-information and $\epsilon$-capacity of the set $M$.

Problem. Find radius $\epsilon$ such that

$$\left|{\mathcal{I}}_{\epsilon }\left(M\right)-{\mathcal{E}}_{\epsilon }\left(M\right)\right|\to min$$

Calculation of $\epsilon$-information and $\epsilon$-capacity is possible only for certain sets $M$. Example of such calculations for evenly distributed observations over an interval of real numbers are presented in section 4. However, an algorithmic solution of the problem is rather simple.

Below we present an algorithm of finding $\epsilon$ for the interval $M=\left[x_0,x_m\right]\subset R$ of real numbers, which can be directly extended to the subset of any lattice with appropriate $\sigma$-algebra.

Algorithm 1. Computing radius $\epsilon$

Input:  Set $M=\left[x_0,x_m\right]\subset R$;

observations’ partition $\xi =\left\{X_1,X_2,\dots ,X_m\right\}$, $X_{i+1}=\left[x_i,x_{i+1}\right]$, $x_i<x_{i+1}$,

$i=0,1,2,\dots ,m-1$;

step $s>0$.

Output: Radius $\epsilon$.

1. Calculate ${\epsilon }_m=\left(x_m-x_0\right)/\left(2m\right)$.

2. Calculate ${\mathcal{H}}_{{\epsilon }_m}\left(M\right)={\log }_{2}m$.

3. For $\epsilon ={\epsilon }_m$ to $\left(x_m-x_0\right)/2$ with step $s$ do:

4.   Calculate ${\mathcal{H}}_{\epsilon }\left(M\right)={\log }_2⌈\left(x_m-x_0\right)/2\epsilon ⌉$.

5.   Create partition ${\xi }_{\epsilon }=\left\{Y_1,Y_2,\dots ,Y_{m_{\epsilon }}\right\}$ of $M$ such that

$Y_{j+1}=\left[y_j,y_{j+1}\right]$, $y_j<y_{j+1}$, $y_0=x_0$, $y_{m_{\epsilon }}=x_m$, and $y_{j+1}-y_j=2\epsilon$, $j=0,1,2,\dots ,m_{\epsilon }-1$.

6.   Compute ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)=eps\_entropy\left(\xi ,{\xi }_{\epsilon }\right)$.

7.   Calculate ${\mathcal{I}}_{\epsilon }\left(M\right)=2{\mathcal{H}}_{{\epsilon }_m}\left(M\right)-{\mathcal{H}}_{\epsilon }\left(M\right)-{\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$.

8.   Compute ${\mathcal{E}}_{\epsilon }\left(M\right)=eps\_capacity\left(\xi ,\epsilon \right)$.

9.   If ${\mathcal{I}}_{\epsilon }\left(M\right)>{\mathcal{E}}_{\epsilon }\left(M\right)$ then

10.     Break

11.   End if.

12. End for.

13. Return $\epsilon$.

The algorithm includes two functions, $eps\_entropy\left(\xi ,{\xi }_{\epsilon }\right)$ and $eps\_capacity\left(\xi ,\epsilon \right)$ which are defined as follows.

Function 1. $eps\_entropy\left(\xi ,{\xi }_{\epsilon }\right)$

Input:  Partition $\xi =\left\{X_1,X_2,\dots ,X_m\right\}$, $X_{i+1}=\left[x_i,x_{i+1}\right]$, $x_i<x_{i+1}$,

$i=0,1,2,\dots ,m-1$.

Partition ${\xi }_{\epsilon }=\left\{Y_1,Y_2,\dots ,Y_{m_{\epsilon }}\right\}$, $Y_{j+1}=\left[y_j,y_{i+1}\right]$ $y_j<y_{j+1}$,

$j=0,1,2,\dots ,m_{\epsilon }-1$,

$y_0=x_0$, $y_{m_{\epsilon }}=x_m$, and $y_{j+1}-y_j=2\epsilon$, $j=0,1,2,\dots ,m_{\epsilon }-2$.

Output: $\epsilon$-entropy ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$ of the partition $\xi \vee {\xi }_{\epsilon }$.

1. From the partitions $\xi$ and ${\xi }_{\epsilon }$ create, respectively, the sets of points

$\mathcal{X}=\left\{x_0,x_1,x_2,\dots ,x_m\right\}$ and $\mathcal{Y}=\left\{y_0,y_1,y_2,\dots ,y_m\right\}$.

2. Join the sets $\mathcal{X}$ and $\mathcal{Y}$: ${\mathcal{X}}_{joint}=\mathcal{X}\cup \mathcal{Y}$.

3. Find a number $N\left({\mathcal{X}}_{joint}\right)$ of elements in the set ${\mathcal{X}}_{joint}$.

4. Set ${\mathcal{N}}_{\epsilon }\left({\mathcal{X}}_{joint}\right)=N\left({\mathcal{X}}_{joint}\right)-1$.

5. Set ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)={\log }_2{\mathcal{N}}_{\epsilon }\left({\mathcal{X}}_{joint}\right).$

6. Return ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$.

Function $eps\_entropy$ can be implemented using concatenation of the sets with further removing of the doubling elements.

Function 2. $eps\_capacity\left(\xi ,\epsilon \right)$

Input:  Partition $\xi =\left\{X_1,X_2,\dots ,X_m\right\}$, $X_{i+1}=\left[x_i,x_{i+1}\right]$, $x_i<x_{i+1}$,

$i=0,1,2,\dots ,m-1$.

Radius $\epsilon >0$.

Output:  $\epsilon$-capacity ${\mathcal{E}}_{\epsilon }\left(M\right)$.

1. From the partition $\xi$ create the sets of points $\mathcal{X}=\left\{x_0,x_1,x_2,\dots ,x_m\right\}$.

2. If $\left(x_m-x_0\right)\le \epsilon$ then

3.   Set ${\mathcal{M}}_{\epsilon }\left(M\right)=1$.

4. Else

5.   Set ${\mathcal{M}}_{\epsilon }\left(M\right)=2$.

6.   Set $j=0$.

7.   For $i=1$ to $m-1$ do:

8.     If $\left(x_i-x_j\right)\le \epsilon$ or $\left(x_m-x_i\right)\le \epsilon$ then

9.       Continue.

10.     Else

11.       Set ${\mathcal{M}}_{\epsilon }\left(M\right)={\mathcal{M}}_{\epsilon }\left(M\right)+1$.

12.       Set $j=i$.

13.     End if.

14.   End for.

15. End if.

16. Set ${\mathcal{E}}_{\epsilon }\left(M\right)={\log }_2{\mathcal{M}}_{\epsilon }\left(M\right)$.

17. Return ${\mathcal{E}}_{\epsilon }\left(M\right)$.

Time complexity $\mathcal{C}$ of Algorithm 1 includes the following terms: $\mathcal{O}\left(1\right)$ – complexity of the lines 1-4; $\mathcal{O}\left(m\right)$ – complexity of the line 5; $\mathcal{O}\left(m\log m\right)$ – complexity of the line 6; $\mathcal{O}\left(1\right)$ – complexity of the line 7; $\mathcal{O}\left(m\right)$ – complexity of the line 8 and $\mathcal{O}\left(1\right)$ – complexity of the lines 9-13. Then, time complexity of each iteration of the algorithm is $\mathcal{O}\left(m\log m\right)$. The maximal number of iterations is $n=\left(t_m-t_0\right)/2s$; hence complexity of Algorithm 1 is

$\mathcal{C}=n\times \mathcal{O}\left(m\log m\right)$.

Convergence of Algorithm 1 is guaranteed by the indicated above fact that $\epsilon$-information ${\mathcal{I}}_{\epsilon }\left(M\right)$ increases with increasing $\epsilon$ while $\epsilon$-capacity ${\mathcal{E}}_{\epsilon }\left(M\right)$ decreases with increasing $\epsilon$. Since the interval $M=\left[x_0,x_m\right]$ is bounded, the difference between increasing $\epsilon$-information ${\mathcal{I}}_{\epsilon }\left(M\right)$ and decreasing $\epsilon$-capacity ${\mathcal{E}}_{\epsilon }\left(M\right)$ has its minimum in $M$, which is a terminating point of the algorithm.

Algorithm 1 returns radius $\epsilon$ for which the difference $\left|{\mathcal{I}}_{\epsilon }\left(M\right)-{\mathcal{E}}_{\epsilon }\left(M\right)\right|$ between $\epsilon$-information and $\epsilon$-capacity is minimal. Consequently, partition ${\xi }_{\epsilon }=\left\{Y|Y\subset M\right\}$, in which diameters of the sets $Y\in {\xi }_{\epsilon }$ are $d\left(Y\right)=2\epsilon$, is a required partition.

Dependence of the functions ${\mathcal{I}}_{\epsilon }\left(M\right)$ and ${\mathcal{E}}_{\epsilon }\left(M\right)$ on the diameter $2\epsilon$ is illustrated in Figure 1. In this example, $M=\left[x_0,x_m\right]$, $x_0=0$, $x_m=100$, and in observations partition $\xi =\left\{X_1,X_2,\dots ,X_m\right\}$, $X_{i+1}=\left[x_i,x_{i+1}\right]$, $x_i<x_{i+1}$, $i=0,1,2,\dots ,m-1$, points $x_i$ are evenly distributed on the interval $M$.

The computed radius is $\epsilon \approx 7.1$ and diameter is $d=2\epsilon \approx 14.2$. The values of $\epsilon$-information and $\epsilon$-capacity are ${\mathcal{I}}_{\epsilon }\left(M\right)={\mathcal{E}}_{\epsilon }\left(M\right)\approx 3.7$ bit.

Note that the accuracy of computing radius $\epsilon$ increases with decreasing the step $s$.

4. Numerical Examples

Let $M=\left[x_0,x_m\right]\subset R$ be an interval of real numbers, $x_m>x_0$. Given $\epsilon >0$, a minimal number of sets in the $\epsilon$-covering $\alpha$ of the interval $M$ is

$${\mathcal{N}}_{\epsilon }\left(M\right)=⌈{\displaystyle \frac{x_m-x_0}{2\epsilon }}⌉.$$

Then, the $\epsilon$-entropy of the set $M$ is

$${\mathcal{H}}_{\epsilon }\left(M\right)={\log }_2{\mathcal{N}}_{\epsilon }\left(M\right)={\log }_2⌈{\displaystyle \frac{x_m-x_0}{2\epsilon }}⌉.$$

Let ${\epsilon }_m$ be a minimal value of $\epsilon$ which provides maximal $\epsilon$-entropy

$${\mathcal{H}}_{{\epsilon }_m}\left(M\right)={\log }_2⌈{\displaystyle \frac{x_m-x_0}{2{\epsilon }_m}}⌉$$

of the interval $M$. Following lemma 5

$${\mathcal{H}}_{{\epsilon }_m}\left(M\right)={\log }_{2}m;$$

thus

$${\epsilon }_m={\displaystyle \frac{1}{2}}{\displaystyle \frac{x_m-x_0}{m}}.$$

Let

$$\xi =\left\{X_1,X_2,\dots ,X_m\right\},X_{i+1}=\left[x_i,x_{i+1}\right],x_i<x_{i+1},i=0,1,2,\dots ,m-1,$$

be an observations partition of the interval $M$ and

$${\xi }_{\epsilon }=\left\{Y_1,Y_2,\dots ,Y_{m_{\epsilon }}\right\},Y_{j+1}=\left[y_j,y_{i+1}\right],y_j<y_{j+1},y_0=x_0,y_{m_{\epsilon }}=x_m,$$

$$y_{j+1}-y_j=2\epsilon ,j=0,1,2,\dots ,m_{\epsilon }-1.$$

be a partition of the interval $M$ such that all its elements are of the length $2\epsilon$.

Then, $\epsilon$-information of the interval $M$ is

$${\mathcal{I}}_{\epsilon }\left(M\right)=2{\log }_{2}m-{\log }_2⌈{\displaystyle \frac{x_m-x_0}{2\epsilon }}⌉-{\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right).$$

Entropy ${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)$ depends on distribution of the values $x_i$, $i=0,1,2,\dots ,m$, over the interval $M$.

Assume that these values are distributed evenly and $\xi \preccurlyeq {\xi }_{\epsilon }$. Then $\xi \vee {\xi }_{\epsilon }={\xi }_{\epsilon }$ and

$${\mathcal{H}}_{\epsilon }\left(\xi \vee {\xi }_{\epsilon }\right)={\mathcal{H}}_{\epsilon }\left(M\right)={\log }_2⌈{\displaystyle \frac{x_m-x_0}{2\epsilon }}⌉.$$

Note that in general this equivalence does not hold and calculation of the entropy of multiplication of partitions is processed according to function 1 (see section 3).

Similarly, $\epsilon$-capacity ${\mathcal{E}}_{\epsilon }\left(M\right)$ depends on the distribution of the values $x_i$, $i=0,1,2,\dots ,m$, over the interval $M$. If these values are distributed evenly such that $x_{j+1}-x_j=x_{j+2}-x_{j+1}$ and $x_{j+1}-x_j>\epsilon$ for any $j=0,1,2,\dots ,m-2$, then

$${\mathcal{M}}_{\epsilon }\left(M\right)=⌈{\displaystyle \frac{x_m-x_0}{\epsilon }}⌉$$

and

$${\mathcal{E}}_{\epsilon }\left(M\right)={\log }_2{\mathcal{M}}_{\epsilon }\left(M\right)={\log }_2⌈{\displaystyle \frac{x_m-x_0}{\epsilon }}⌉.$$

If distribution of the values $x_i$ is such that $x_{m-1}-x_0\le \epsilon$, which means that all the values except $x_m$ are located between $x_0$ and $x_{m-1}$, and $x_m-x_0>\epsilon$, then

$${\mathcal{M}}_{\epsilon }\left(M\right)=2$$

and

$${\mathcal{E}}_{\epsilon }\left(M\right)={\log }_{2}2=1.$$

If $x_m-x_0\le \epsilon$, then the interval $M$ does not contain $\epsilon$-distinguishable subsets, and we assume that

$${\mathcal{M}}_{\epsilon }\left(M\right)=1$$

and

$${\mathcal{E}}_{\epsilon }\left(M\right)={\log }_{2}1=0.$$

In general case, $\epsilon$-capacity is calculated using function 2 (see section 3).

Finally, using the formulated criterion (see the problem in section 3), for evenly distributed values $x_i$ we obtain (for simplicity we omit the notion of ceiling)

$${\mathcal{I}}_{\epsilon }\left(M\right)-{\mathcal{E}}_{\epsilon }\left(M\right)=2{\log }_{2}m-2{\log }_2{\displaystyle \frac{x_m-x_0}{2\epsilon }}-{\log }_2{\displaystyle \frac{x_m-x_0}{\epsilon }}={\log }_2{m}^2-{\log }_2{\displaystyle \frac{{\left(x_m-x_0\right)}^3}{4{\epsilon }^3}}.$$

Thus, it is required to find a value $\epsilon$ such that

$${\displaystyle \frac{{\left(x_m-x_0\right)}^3}{4{\epsilon }^3}}=m^2,$$

which is

$$\epsilon ={\displaystyle \frac{x_m-x_0}{\sqrt[3]{4m^2}}}.$$

Consequently, diameter $d\left(Y_j\right)$ of the intervals $Y_j$, $j=1,2,\dots ,m_{\epsilon }$, in the required partition ${\xi }_{\epsilon }=\left\{Y_1,Y_2,\dots ,Y_{m_{\epsilon }}\right\}$ is

$$d\left(Y_j\right)=2{\displaystyle \frac{x_m-x_0}{\sqrt[3]{4m^2}}}.$$

To illustrate the suggested method, let us calculate diameters $d\left(Y_j\right)$, $j=1,2,\dots ,m_{\epsilon }$, of intervals in the partition ${\xi }_{\epsilon }$ and compare the obtained results with the bin lengths provided by the known methods appearing in the literature.

Let $M=\left[0,10\right]$ and $\xi =\left\{X_1,X_2,\dots ,X_{10}\right\}$, $X_1=\left[0,1\right]$, $X_2=\left(\left.1,2\right]\right.$,…, $X_{10}=\left(\left.9,10\right]\right.$. Then,

$$d\left(Y_j\right)=2{\displaystyle \frac{10-0}{\sqrt[3]{4{\times 11}^2}}}=2.55,$$

for all $j=1,\dots ,m_{\epsilon }=4$. For this interval $M$, bin lengths calculated by the known methods used for plotting histograms are the following:

-

the simplest rule: $d_1={\displaystyle \frac{\max A-\min A}{\sqrt{m}}}={\displaystyle \frac{10-0}{\sqrt{11}}}=3.01$,

-

the Sturges rule [14]: $d_2={\displaystyle \frac{\max A-\min A}{{\log }_{2}m+1}}={\displaystyle \frac{10-0}{{\log }_{2}11+1}}=2.24$,

-

the Scott rule [11]: $d_3={\displaystyle \frac{3.49s}{\sqrt[3]{m}}}={\displaystyle \frac{3.49\times 3.32}{\sqrt[3]{11}}}=5.20$,

-

the Freedman-Diaconis rule [3]: $d_4={\displaystyle \frac{2\left(Q_3-Q_1\right)}{\sqrt[3]{m}}}={\displaystyle \frac{2\times \left(8-3\right)}{\sqrt[3]{11}}}=4.94$.

It is seen that in this example with evenly distributed observations, diameter calculated using the suggested method is compatible with the bin lengths calculated using the known rules.

Now let us consider examples with different distributions of observed values $x_i$, $i=0,1,2,\dots ,m$, and so – different observations partitions $\xi =\left\{X_i|X_i\subset M\right\}$. In all examples we assume that $M=\left[x_0,x_m\right]$, $x_0=0$, $x_m=100$ and $m=100$.

In these examples, we used uniform distribution with $a=0$ and $b=x_m$, normal distribution with $\mu =x_m/2$ and $\sigma =x_m/6$, and exponential distribution with $\mu =2$. Diameters $d$ calculated by the suggested method and bin lengths $d_1,\dots ,d_4$ calculated by the known rules are the following:

-

evenly distributed data:

$$d=14.21,d_1=9.90,d_2=12.95,d_3=21.81\mathrm{and}{d}_4=21.54,$$

-

uniform distribution with $a=0$ and $b=x_m$:

$$d=14.11,d_1=9.83,d_2=12.87,d_3=22.69\mathrm{and}{d}_4=22.65,$$

-

normal distribution with $\mu =x_m/2$ and $\sigma =x_m/6$:

$$d=11.91,d_1=8.27,d_2=10.82,d_3=13.11\mathrm{and}{d}_4=10.38,$$

-

exponential distribution with $\mu =2$:

$$d=1.20,d_1=1.07,d_2=1.41,d_3=1.35\mathrm{and}{d}_4=0.91.$$

It is seen that the suggested method results in the diameters $d$ that are close to the bin lengths $d_1,\dots ,d_4$ provided by the conventional methods with respect to the distribution of the data. For evenly and uniformly distributed data diameter $d$ is close to the bin length $d_2$ resulted by the Sturges method, for normal distribution $d_2<d<d_3$ and for exponential distribution $d_1<d<d_2$.

Histograms of these data with the bin lengths $d_3$ calculated using the Scott rule and with the bin lengths $d$ calculated by the suggested method are shown in Figure 2.

Finally, let us consider specific distribution, which appears in the analysis of queues [4].

Consider a clerk serving the clients of some office during a day. Then, arrival $\lambda$ and departure $\mu$ rates measured per day describe the state of the system at the end of the day, but do not provide any information about the system during the day.

On other hand, the rates $\lambda$ and $\mu$ measured per minute are also useless since the clients do not arrive and the clerk does not serve with such velocities.

Thus, it is required to define correct time units per which arrival and service rates should be specified. In such a form the problem was formulated by Yaakov Reis [8].

Assume that the office, where the mentioned above clerk works, serves $480$ clients $8$ hours during a day. Also, assume that the clients arrive by three “waves” – in the morning, in the midday and in the evening.

For this data, diameter $d$ calculated by the suggested method and bin lengths $d_1,\dots ,d_4$ calculated by the known rules are the following:

$$d=22.0,d_1=21.91,d_2=48.45,d_3=67.99\mathrm{and}{d}_4=66.41$$

Histogram for this data with the bin lengths $d_3$ calculated using the Scott rule is shown in Figure 3.a, and histogram with the bin lengths $d$ calculated by the suggested method is shown in Figure 3.b.

Dependence of the functions ${\mathcal{I}}_{\epsilon }\left(M\right)$ and ${\mathcal{E}}_{\epsilon }\left(M\right)$ on $\epsilon$ for this distribution is shown in Figure 3.c.

It is seen that for this data the suggested method results in the diameter $d$ which is close to the bin length $d_1$ provided by the simplest rule. Following this result, arrival rates during a day should be calculated each $22$ minutes, while the Sturges rule they should be calculated each $48$ minutes, by the Scott rule – each $68$ minutes, and by the Freedman-Diaconis rule – each 66 minutes.

5. Conclusion

In the paper, we suggested a method of discretization of dynamical system basing on its observations. The method results in the partition ${\xi }_{\epsilon }$, which represents the system, and is as close as possible to the observations partition $\xi$.

The method utilizes the Kolmogorov and Tikhomirov $\epsilon$-entropy and $\epsilon$-capacity and the Rokhlin distance between partitions.

Along with theoretical results, the method is presented in the form of ready-to-use algorithm.

The suggested method is also useful for definition of the bin lengths of histograms, especially for the datasets with multimodal distributions.