A New Logical Basis for Variance and Covariance

1. Introduction: A Fundamental Duality in the Exact Sciences

A recent development in mathematical logic was the development of the logic of partitions (or equivalence relations) ([1,2,3]) that is category-theoretically dual to the usual Boolean logic of subsets. This initiated a series of developments, some new and some reformulations of older ideas, that showed how the duality extended throughout the mathematical sciences [4] (including quantum mechanics and beyond to the life sciences). The dual logics have quantitative versions. The quantitative version of subset logic is probability theory which is why George Boole’s book was entitled “An Investigation of the Laws of Thought on which are founded the Mathematical Theories of Logic and Probabilities”[5]. Gian-Carlo Rota conjectured on numerous occasions [6] that subsets are to probability as partitions are to information:

${\displaystyle \frac{\mathrm{Subsets}}{\mathrm{Probability}}}\approx {\displaystyle \frac{\mathrm{Partitions}}{\mathrm{Information}}}$

Accordingly, the quantitative version of partitions defined the notion of logical entropy which provides a logical foundation for the information theory ordinarily uses only the better-known notion of Shannon entropy.

There is a certain methodology used in developing this duality. That is, the notions associated with subset logic are defined in terms of single elements and the notions associated with partition logic are defined in terms of pairs of elements (or even ordered pairs). The simplest notion in mathematical statistics is the mean of a random variable (r.v.) which is associated with the single samplings of a random variable. The partition-motivated notion of a pair of samples of a r.v. turns out to be the logical basis for variance (and similarly covariance). This is not the usual definition of variance (or covariance) in the textbooks. It seems to be so little known that it has been announced as a new discovery [7], although that derivation of variance is much older [8] (p. 42) and was known as the variance formula that does not use the mean.

Our purpose here is to place that derivation of variance and covariance in the context of the fundamental category-theoretic duality starting with the dual logics of subsets and partitions. Moreover, this treatment of variance and covariance place those notions firmly on the information-theoretic side of the duality, not the probability-theoretic side. Hence we begin with the duality at the logical level.

2. Methods

2.1. The Duality of Subsets and Partitions

The mathematical logic that is currently presented in the form of “propositional” logic is really a special case of the Boolean logic of subsets.

The algebra of logic has its beginning in 1847, in the publications of Boole and De Morgan. This concerned itself at first with an algebra or calculus of classes, to which a similar algebra of relations was later added. Though it was foreshadowed in Boole’s treatment of `Secondary Propositions,’ a true propositional calculus perhaps first appeared from this point of view in the work of Hugh MacColl, beginning in 1877. [9] (pp. 155-56)

Category theory dates back to 1945 [10]. It provides a formal (turn around the arrows) notion of duality, and the dual of the notion of a subset is the notion of a quotient set or, equivalently, a partition or an equivalence relation. In general a subobject may be called a “part” and “The dual notion (obtained by reversing the arrows) of `part’ is the notion of partition.” [11] (p. 85)

This duality is missed when logic is only presented as “propositional” logic since propositions don’t have duals. Hence one needs the general Boolean notion of the logic of subsets to suggest a dual logic of partitions or equivalence relations. It was known in the 19th century (Richard Dedekind and Ernst Schröder) that partitions form a lattice with the operations of join ∨ and meet ∧. Gian-Carlo Rota and colleagues published a paper on the logic of certain equivalence relations but with only the lattice operations of join and meet and without any implication operation [12]. Indeed, no new operations of partitions or equivalence relations were developed in the 20th century.

Equivalence relations are so ubiquitous in everyday life that we often forget about their proactive existence. Much is still unknown about equivalence relations. Were this situation remedied, the theory of equivalence relations could initiate a chain reaction generating new insights and discoveries in many fields dependent upon it.

This paper springs from a simple acknowledgement: the only operations on the family of equivalence relations fully studied, understood and deployed are the binary join ∨ and meet ∧ operations. [13] (p. 445)

2.2. The Two Lattices of Subsets and of Partitions

A partition $\pi =\left\{B_1,\dots ,B_m\right\}$ on a universe set $U=\left\{u_1,\dots ,u_n\right\}$ ($\left|U\right|\ge 2$) is set of non-empty subsets $B_1,\dots ,B_m$ of U (called the blocks of the partition) that are disjoint and whose union is U. A distinction or dit of$\pi$ is an ordered pair $\left(u_i,u_j\right)$ of elements $u_i,u_j\in U$ in distinct blocks of the partition $\pi$. The set of distinctions or ditset is $dit\left(\pi \right)\subseteq U\times U$ and its complement in $U\times U$ is the set of indistinctions or indits$indit\left(\pi \right)$ which is just the equivalence relation version of the partition $\pi$ where the blocks of the partition are the equivalence classes of the equivalence relation.

In the Boolean lattice of subsets $\wp \left(U\right)$, the join is the union of subsets, the meet is the intersection, and the partial order is the inclusion relation between subsets. The implication or conditional operation on subsets of U is denoted $S\supset T$ and is defined as $S^c\cup T$ (where $S^c$ is the complement of S). The addition of the conditional operation turns the Boolean lattice into the Boolean algebra of subsets.

Partitions on U also form a lattice, the partition lattice $\mathsf{\Pi }\left(U\right)$. Given $\pi =\left\{B_1,\dots ,B_m\right\}$ and $\sigma =\left\{C_1,\dots ,C_{m^{\prime }}\right\}$ partitions on U, the join $\pi \vee \sigma$ is the partition on U whose blocks are the non-empty intersections $B_j\cap C_{j^{\prime }}$ for $j=1,\dots ,m$ and $j^{\prime }=1,\dots ,m^{\prime }$. The ditset of the join is just the union of the ditsets, i.e., $dit\left(\pi \vee \sigma \right)=dit\left(\pi \right)\cup dit\left(\sigma \right)$. Since the join $S\cup T$ in the lattice of subsets is just the union of the elements of the two subsets $S,T\subseteq U$, we see that the distinctions or “Dits” of a partition play the corresponding role as the elements or “Its” of a subset in the duality between subsets and partition.

To define the meet $\pi \wedge \sigma$ of two partitions, suppose that if two blocks $B_j$ and $C_{j^{\prime }}$ overlap (i.e., the intersection is non-empty) then they unite like two blobs of mercury, i.e., take their union $B_j\cup C_{j^{\prime }}$. Doing this for all overlaps, we eventually arrive as subsets that are the exact union of blocks of $\pi$ and also the exact union of blocks of $\sigma$ and are minimal in that regard. Those are the blocks of the meet $\pi \wedge \sigma$.

The new implication operation partitions, denoted $\sigma \Rightarrow \pi$, is defined as the partition that is like $\pi$ except when a block $B_j\in \pi$ is contained in some block $C_{j^{\prime }}\in \sigma$, i.e., $B_j\subseteq C_{j^{\prime }}$, then $B_j$ is replaced by its discretization, i.e., by the singleton blocks of its elements.

The partial order for partitions is called refinement where $\pi$ refines $\sigma$, written $\sigma \precsim \pi$, if for every block $B_j\in \pi$, there is a block $C_{j^{\prime }}$ of $\sigma$ such that $B_j\subseteq C_{j^{\prime }}$. Intuitively, $\pi$ can be obtained from $\sigma$ by chopping up some blocks of $\sigma$. Some older texts ([14,15]) define the “lattice of partitions” with the reverse partial order (which reverses the join and meet) which Gian-Carlo Rota called “unrefinement” or “reverse refinement” [6] (p. 30). With the refinement partial order, refinement is just the inclusion of distinctions, i.e., $\sigma \precsim \pi$ if and only if $dit\left(\sigma \right)\subseteq dit\left(\pi \right)$, just as the partial order in the Boolean lattice is inclusion of elements. That is one way the dual connection between “elements” and “distinctions” shows itself. The top of the lattice of partitions is the discrete partition ${\mathbf{1}}_U={\left\{\left\{u\right\}\right\}}_{u\in U}$ where all blocks are singletons and the bottom is the indiscrete partition ${\mathbf{0}}_U=\left\{U\right\}$ where the only block is U. The addition of the implication operation turns the (19th century) notion of a partition lattice into the (21st century) notion of a partition algebra.

When the respective implications are equal to the top element, then and only then the partial order relation holds, i.e.,

$S\supset T=U$ iff $S\subseteq T$

$\sigma \Rightarrow \pi ={\mathbf{1}}_U$ iff $\sigma \precsim \pi$.

For $U=\left\{a,b,c\right\}$, the two lattices are illustrated in Figure 1.

Table 1 summarizes the duality between the lattice of subsets $\wp \left(U\right)$ and the lattice of partitions $\mathsf{\Pi }\left(U\right)$.

Hence we can extend Rota’s duality statement to that between elements of subsets and distinctions of partitions:

${\displaystyle \frac{\mathrm{Elements}}{\mathrm{Subsets}}}\approx {\displaystyle \frac{\mathrm{Distinctions}}{\mathrm{Partitions}}}$.

One aspect of the dual concepts of elements and distinctions (that we will employ later), is that elements are a one-variable concept while distinctions are pairs of elements. The basic questions about elements are, for instance, does a predicate apply to it or not which is a question about existence, i.e., does it exist in the subset of elements with that property or not. The basic questions about distinctions are about, given two elements, are they distinct or not in the partition or are they equivalent or not in the corresponding equivalence relation. As we will see, this duality between single-element concepts and pair-concepts comes out in statistics.

2.3. Fundamental Status of the Two Lattices

The notions of a subset of a set and a partition on a set are defined without any additional structure on the sets. Hence the lattices, algebras, and logics of those two notions have a certain fundamental status. Given a topology or a partial order on a set, then other lattices can be defined in terms of that additional structure. But the notions of subsets and partitions require no such additional structure and thus their fundamentality. Moreover, subsets and partitions are category-theoretically dual concepts.

The two dual logics provide a modern model for the old Aristotelian duality of substance versus form(as in in-form-ation) [16]. We can tell two abstract “creation stories” by moving up in the two lattices from the bottom to the top. We see that the existence of substance increases in the subset lattice while form stays constant (always classical in the sense of being fully distinct), and the reverse happens in the partition lattice.

For each lattice where $U=\left\{a,b,c\right\}$, start at the bottom and move towards the top in Figure 2.

There is no substance or “Its” (elements) at the bottom ∅ of the subset lattice, and as one moves up, new “Its” or elements appear but always fully formed, i.e., no indefiniteness, until one reaches the full universe U. At the bottom ${\mathbf{0}}_U$ of the partition lattice, there are no dits (distinctions), i.e., $dit\left({\mathbf{0}}_U\right)=\varnothing$, and all the substance already appears but with no form (i.e., no “Dits” in ${\mathbf{0}}_U$ just as no “Its” in ∅). As one moves up the lattice of partitions, form (as in in-form-ation) is created by making new dits or distinctions until reaching the partition ${\mathbf{1}}_U$ that makes all possible distinctions, i.e., $dit\left({\mathbf{1}}_U\right)=U\times U-\Delta$ where $\Delta =\left\{\left(u_i,u_i\right)|u_i\in U\right\}$ is the diagonal of all the self-pairs of elements of U, since an element cannot be distinguished from itself.

The progress from bottom to top of the two lattices can be described as two creation stories.

• Subset creation story: “In the Beginning was the Void”, and then elements are created, fully propertied and distinguished from one another, until finally reaching all the elements of the universe set U.
• Partition creation story: “In the Beginning was undifferentiated Substance (e.g., “Formless Chaos”), and then there is a “Big Bang” where the substance is being objectively in-formed by the making of distinctions (i.e., symmetry-breaking) until the result is finally the singletons which designate the elements of the universe U.

Heisenberg identifies “substance” with energy.

Energy is in fact the substance from which all elementary particles, all atoms and therefore all things are made, and energy is that which moves. Energy is a substance, since its total amount does not change, and the elementary particles can actually be made from this substance as is seen in many experiments on the creation of elementary particles [17] (p. 63).

Thus the partition creation story, can be viewed as a bare-bones description of the Big Bang theory of creation–constant substance (energy) in-formed by symmetry-breaking distinctions [18].

2.4. Logical Entropy

2.4.1. A Little History of Information-As-Distinctions

When the logic of partitions was developed [3], then it could imitate Boole’s development of logical probability as the quantitative version of subsets [5]. Logical probability was the number of elements in a subset ${\displaystyle \frac{\left|S\right|}{\left|U\right|}}$ normalized by the cardinality of U. The dual to the notion of an element of a subset is a distinction of a partition.

The wide-ranging anthropologist, Gregory Bateson, described information as “differences that make a difference.” [19] (p. 99). One of the founders of quantum information theory, Charles Bennett, has described information as “the notion of distinguishability abstracted away from what we are distinguishing, or from the carrier of information... .” [20] (p. 155). But the notion of information as the quantification of distinctions or differences goes back almost four centuries. In The Information: A History, A Theory, A Flood by James Gleick, he noted the focus on distinctions or differences in the 17th century polymath and founder of the Royal Society, John Wilkins. In 1641, the year before Newton was born, Wilkins published one of the earliest books on cryptography, Mercury or the Secret and Swift Messenger, which not only pointed out the fundamental role of differences but noted that any (finite) set of different things could be encoded by words in a binary code.

For in the general we must note, that whatever is capable of a competent difference, perceptible to any sense, may be a sufficient means whereby to express the cogitations. It is more convenient, indeed, that these differences should be of as great variety as the letters of the alphabet ; but it is sufficient if they be but twofold, because two alone may, with somewhat more labour and time, be well enough contrived to express all the rest [21] (p. 67).

Wilkins explains that a five letter binary code would be sufficient to code the letters of the alphabet since $2^5=32$.

Thus any two letters or numbers, suppose A. B. being transposed through five places, will yield thirty-two differences, and so consequently will superabundantly serve for the four and twenty letters,... [21] (pp. 67-8).

As Gleick noted:

Any difference meant a binary choice. Any binary choice began the expressing of cogitations. Here, in this arcane and anonymous treatise of 1641, the essential idea of information theory poked to the surface of human thought, saw its shadow, and disappeared again for [three] hundred years [22] (p. 161).1

Gleick is here dating the development of modern information theory from Claude Shannon’s work published in 1948 [25]. This is how the development of the logic of partitions, dual to the Boolean logic of subsets, and using the duality between the elements-of-a-subset (or Its) and the distinctions-of-a-partition (or Dits), reveals a dual relationship between logical probability theory and logical information theory.

2.4.2. The Mathematics of Logical Entropy

We have seen how probability theory starts with the notion of logical probability of a subset (or event) as the normalized number of elements in the subset S:

$Pr\left(S\right)={\displaystyle \frac{\left|S\right|}{\left|U\right|}}$.

We have also seen the duality between subsets and partitions expressed as:

${\displaystyle \frac{\mathrm{Subsets}}{\mathrm{Probability}}}\approx {\displaystyle \frac{\mathrm{Partitions}}{\mathrm{Information}}}$ and ${\displaystyle \frac{\mathrm{Elements}}{\mathrm{Subsets}}}\approx {\displaystyle \frac{\mathrm{Distinctions}}{\mathrm{Partitions}}}$

which also means that:

${\displaystyle \frac{\mathrm{Elements}}{\mathrm{Probability}}}\approx {\displaystyle \frac{\mathrm{Distinctions}}{\mathrm{Information}}}$.

Hence the logical notion of information in a partition should be the normalized number of distinctions in the partition. Following Shannon’s labeling of his quantification of information as “entropy,” we will call the notion based on quantifying the logic of partitions as “logical entropy.” ([23,24]) For a partition $\pi =\left\{B_1,\dots ,B_m\right\}$, the logical entropy$h\left(\pi \right)$ of $\pi$ is the normalized number of distinctions:

$h\left(\pi \right)={\displaystyle \frac{\left|dit\left(\pi \right)\right|}{\left|U\times U\right|}}={\displaystyle \frac{\left|U\times U-indit\left(\pi \right)\right|}{\left|U\times U\right|}}=1-{\displaystyle \frac{\left|{\cup }_{j=1}^m{B}_j\times B_j\right|}{\left|U\times U\right|}}=1-{\sum }_{j=1}^m{\left({\displaystyle \frac{\left|B_j\right|}{\left|U\right|}}\right)}^2$

$=1-{\sum }_{j=1}^{m}Pr{\left(B_j\right)}^2={\sum }_{j\ne k}Pr\left(B_j\right)Pr\left(B_k\right)=2{\sum }_{j<k}Pr\left(B_j\right)Pr\left(B_k\right)$

where the product version of the formula follows from:

$1={\left({\sum }_{j=1}^{m}Pr\left(B_j\right)\right)}^2={\sum }_{j=1}^{m}Pr{\left(B_j\right)}^2+2{\sum }_{j<k}Pr\left(B_j\right)Pr\left(B_k\right)$.

As in the case of logical probability, we are, for the moment, assuming equiprobable points in U. Moreover, it should be noted that for probability, we are dealing with single samples and for information, we are dealing with pairs, indeed ordered pairs of elements drawn from U so that each pair of distinct blocks $B_j$ and $B_k$ is counted twice in the sum ${\sum }_{j\ne k}Pr\left(B_j\right)Pr\left(B_k\right)$. This yields an immediate simple interpretation of logical entropy, namely that $h\left(\pi \right)$ is the probability that in two independent samples from U, one obtains a distinction of $\pi$.

The formulas generalize immediately to the general case of finite probability theory where the points $\left\{u_1,\dots ,u_n\right\}$ of U have the respective point probabilities of $p=\left(p_1,\dots ,p_n\right)$. Then $Pr\left(S\right)={\sum }_{u_i\in S}{p}_i$ and

$h\left(\pi \right)=1-{\sum }_{j=1}^{m}Pr{\left(B_j\right)}^2={\sum }_{j\ne k}Pr\left(B_j\right)Pr\left(B_k\right)=2{\sum }_{j<k}Pr\left(B_j\right)Pr\left(B_k\right)$

When the given data is just the probability distribution p on U, then

$h\left(p\right)=1-{\sum }_{i=1}^n{p}_i^2={\sum }_{i\ne k}{p}_i{p}_k=2{\sum }_{i<k}{p}_i{p}_k$ which could also be viewed as the logical entropy of the discrete partition ${\mathbf{1}}_U$, i.e., $h\left(p\right)=h\left({\mathbf{1}}_U\right)$. In all the cases, the logical entropy is the two-sample probability of getting a distinction. The least value of logical entropy is 0 for the indiscrete partition ${\mathbf{0}}_U$ or for p where some $p_i=1$. The maximum value for logical entropy is for the discrete partition with equiprobable points:

$h\left({\mathbf{1}}_U\right)=1-{\sum }_{i=1}^n{\left({\displaystyle \frac{1}{n}}\right)}^2=1-{\displaystyle \frac{n}{n^2}}=1-{\displaystyle \frac{1}{n}}$

which is interpreted simply as the probability that the second draw from U will be different from the first draw.

Diagrammatically, the logical entropy is illustrated using a box diagram with a unit square. For instance for $p=\left(1/2.1/4.1/4\right)$, the box diagram is given in Figure 3 where each of the halves is:

$h\left(p\right)/2={\sum }_{i<k}{p}_i{p}_k=p_1{p}_2+p_1{p}_3+p_2{p}_3={\displaystyle \frac{1}{2}}{\displaystyle \frac{1}{4}}+{\displaystyle \frac{1}{2}}{\displaystyle \frac{1}{4}}+{\displaystyle \frac{1}{4}}{\displaystyle \frac{1}{4}}={\displaystyle \frac{5}{16}}$.

Since the logical entropy of a partition has a probability interpretation, it is can be obtained as the value of a subset on a probability measure. The probability measure is the product measure $p\times p$ on the set $U\times U$ and the subset giving the logical entropy is $dit\left(\pi \right)$, i.e.,

$h\left(\pi \right)=Pr\left(dit\left(\pi \right)\right)=p\times p\left(dit\left(\pi \right)\right)$.

Since $dit\left(\pi \vee \sigma \right)=dit\left(\pi \right)\cup dit\left(\sigma \right)$, $h\left(\pi \vee \sigma \right)=p\times p\left(dit\left(\pi \vee \sigma \right)\right)$ which is sometimes written as $h\left(\pi ,\sigma \right)$ as the joint entropy. There is a natural closure operation on subsets $S\subseteq U\times U$, namely the reflexive-symmetric-transitive (RST) closure of S which is the smallest indit set or equivalence relation containing S so its complementary ditset will define a partition. Although they are the complements of a set under a closure operation, ditsets are not like the open sets in a topological space and the RST closure is not a topological closure operation. The arbitrary union of ditsets is a ditset but the intersection of ditsets is not necessarily a ditset (unlike open sets). But that intersection is nevertheless a subset of $U\times U$ so its probability measure defines the mutual information: $m\left(\pi ,\sigma \right)=p\times p\left(dit\left(\pi \right)\cap dit\left(\sigma \right)\right)$. Similarly, the difference information is: $h\left(\pi |\sigma \right)=p\times p\left(dit\left(\pi \right)-dit\left(\sigma \right)\right)$ which is interpreted as the information in $\pi$ that is not in $\sigma$. Then we have the usual Venn diagram relationships illustrated in Figure 4:

$h\left(\pi ,\sigma \right)=h\left(\pi |\sigma \right)+m\left(\pi ,\sigma \right)+h\left(\sigma |\pi \right)$.

2.4.3. The Relationship with Shannon Entropy

The well-known Shannon entropy [25], $H\left(p\right)={\sum }_{i=1}^n{p}_{i}log\left({\displaystyle \frac{1}{p_i}}\right)$, is not defined as the values of certain subsets of a measure on a set. Shannon defined his notions of joint, mutual, and difference or conditional entropy directly in terms of probabilities and defined them so that the Venn diagrams relationships nevertheless held.

Andrei Kolmogorov objected to Shannon defining information directly in terms of probabilities and instead thought it should be based on a prior combinatorial structure.

Information theory must precede probability theory, and not be based on it. By the very essence of this discipline, the foundations of information theory have a finite combinatorial character [26] (p. 39).

Kolmogorov had his own ideas but it might be noticed that the logical entropy definition of information-as-distinctions is based on probability-free sets of a combinatorial nature, namely, ditsets.

The fact that the compound notions of Shannon entropy satisfy the Venn diagram relations, in spite of not being defined as a measure in the sense of measure theory, is explained by the fact that logical entropy is defined as a measure and there is a uniform transformation between the compound logical entropy and Shannon entropy formulas that preserves Venn diagrams. This raises the question of that relationship between the two entropies.

If an outcome has probability close to or equal to 1, then the occurrence of that outcome intuitively gives little or no information. Therefore, it might be said that information is related to the complement of 1–but there are two complements to 1, the additive complement $1-p_i$ and the multiplicative complement ${\displaystyle \frac{1}{p_i}}$. The additive and multiplicative averages of the respective 1-complements give the two entropies. The additive probabilistic average of the additive 1-complements is the logical entropy: ${\sum }_{i=1}^n{p}_i\left(1-p_i\right)=h\left(p\right)$. The multiplicative average of the multiplicative 1-complements is the Shannon entropy in its log-free or anti-log form: ${\prod }_{i=1}^n{\left({\displaystyle \frac{1}{p_i}}\right)}^{p_i}={log}^{-1}\left(H\left(p\right)\right)$. One chooses the particular log for the context, e.g., $H\left(p\right)={log}_2\left({\prod }_{i=1}^n{\left({\displaystyle \frac{1}{p_i}}\right)}^{p_i}\right)={\sum }_{i=1}^n{p}_i{log}_2\left({\displaystyle \frac{1}{p_i}}\right)$ for coding theory or natural logs for statistical mechanics. Since the log of the multiplicative average transforms it to an additive average, the two additive averages can be transformed, one into the other, by the dit-bit transform $1-p_i⇝log\left({\displaystyle \frac{1}{p_i}}\right)$. Once the compound formulas are expressed in terms of the additive 1-complements, then this non-linear but monotonic dit-bit transform yields the corresponding compound formulas for Shannon entropy [23]. And the dit-bit transform preserves Venn diagrams.

2.4.4. Some History of the Logical Entropy Formula

The formula for logical entropy is not new; it is only the derivation as the quantitative version of the new partition logic that is new. The formula goes back at least to 1912 in the index of mutability of Corrado Gini [27]. The formula resurfaced in the coding-breaking activity in World War II ([28,29]) where it was the additive 1-complement of Alan Turing’s repeat rate ${\sum }_{i=1}^n{p}_i^2$ where it became part of the mathematics of cryptography [30]. After the war, Edmund Simpson published the formula as a quantification of biodiversity [31]. Hence the formula is often know as the Gini-Simpson formula for biodiversity [32]. But Simpson along with I. J. Good worked with Turing at Bletchley Park during WWII, and, according to Good, “E. H. Simpson and I both obtained the notion [the repeat rate] from Turing.” [28] (p. 395) When Simpson published the index in 1948, he (again, according to Good) did not acknowledge Turing “fearing that to acknowledge him would be regarded as a breach of security” [33] (p. 562).

Two elements from $U=\left\{u_1,\dots ,u_n\right\}$ are either identical or distinct. Gini [27] introduced $d_{ij}$ as the “distance” between the $i^{th}$ and $j^{th}$ elements where $d_{ij}=1$ for $i\ne j$ and $d_{ii}=0$, i.e., $d_{ij}=1-{\delta }_{ij}$, the complement to the Kronecker delta. Since $1=\left(p_1+\dots +p_n\right)\left(p_1+\dots +p_n\right)={\sum }_i{p}_i^2+{\sum }_{i\ne j}{p}_i{p}_j$, the logical entropy, i.e., Gini’s index of mutability, $h\left(p\right)=1-{\sum }_i{p}_i^2={\sum }_{i\ne j}{p}_i{p}_j$, is the average logical distance between a pair of independently drawn elements. In 1982, C. R. (Calyampudi Radhakrishna) Rao generalized this by allowing other non-negative distances $d_{ij}=d_{ji}$ for $i\ne j$ (but always $d_{ii}=0$) between the elements of U so that $Q={\sum }_{i,j}{d}_{ij}{p}_i{p}_j$ would be the average distance between a pair of independently drawn elements from U which was know as quadratic entropy [34].

3. Results: The Logical Basis for Variance and Covariance

At the logical level, the fundamental duality starts with the duality between the Boolean logic of subsets and the logic of partitions. At a more granular level, it is the duality between elements (Its) of a subset and distinctions (Dits) of a partition. The elements of a subsets are certain singular elements from the universe set and the distinctions of a partition are certain pairs (or ordered pairs) of elements of the universe set.

Given $U=\left\{u_1,\dots ,u_n\right\}$ with the probability distribution $p=\left(p_1,\dots ,p_n\right)$, consider a real-valued random variable (r.v.) $X:U\to \mathbb{R}$. The inverse-image defines a partition $X^{-1}={\left\{X^{-1}\left(x_j\right)\right\}}_{x_j\in X\left(U\right)}$ on U where the values in the image $X\left(U\right)$ of the r.v. are $x_1,\dots ,x_m$. Then each value has the probability: $Pr\left(X=x_j\right)=Pr\left(x_j\right)={\sum }_{u_i\in X^{-1}\left(x_j\right)}{p}_i$.

On the one (single-values) side of the duality, the probability average of the single values is the usual mean: ${\mu }_X=E\left(X\right)={\sum }_{j=1}^{m}Pr\left(x_j\right)x_j$. But what is the appropriate notion on the other (pair-of-values) side of the duality?

In the same 1912 book [27] where Gini suggested the index of mutability ${\sum }_{i\ne k}{p}_i{p}_k$ of a probability distribution, he suggested the mean difference ${\sum }_{j\ne k}Pr\left(x_j\right)Pr\left(x_k\right)\left|x_j-x_k\right|$. Maurice Kendall noted that the mean difference “has a certain theoretical attraction, being dependent on the spread of the variate-values among themselves and not on the deviations from some central value.” [8] (p. 42) But Kendall went on to note: “It is, however, more difficult to compute than the standard deviation, and the appearance of the absolute values in the defining equations indicates, as for the mean deviation, the appearance of difficulties in the theory of sampling.” [8] (p. 42) At a later date, Kendall summarized his criticism; the mean difference (in comparison to the variance) lacked “ease of calculation, mathematical tractability and sampling simplicity.” [35] (p. 223)

Kendall noted that if one tried to improve the mean difference formula by using the square of the difference in values, then that “is nothing but twice the variance” [8] (p. 42) since:

${\sum }_{j\ne k}Pr\left(x_j\right)Pr\left(x_k\right){\left(x_j-x_k\right)}^2={\sum }_{j,k}Pr\left(x_j\right)Pr\left(x_k\right)\left[x_j^2-2x_j{x}_k+x_k^2\right]$

$={\sum }_{j=1}^{m}Pr\left(x_j\right)x_j^2-2\left({\sum }_{j-1}^{m}Pr\left(x_j\right)x_j\right)\left({\sum }_{k=1}^{m}Pr\left(x_k\right)x_k\right)+{\sum }_{k=1}^{m}Pr\left(x_k\right)x_k^2$

$=2E\left(X^2\right)-2E{\left(X\right)}^2=2Var\left(X\right)$.

This means the C. R. Rao’s quadratic entropy for the distance function $d_{jk}={\left(x_j-x_k\right)}^2$ is twice the variance. We noted previously that the consideration of pairs could take the form of just pairs $\left\{x_j,x_k\right\}$ or ordered pairs $\left(x_j,x_k\right)$ (where $j\ne k$) . Hence the variance may be defined using just simple pairs:

$Var\left(X\right)={\sum }_{j<k}Pr\left(x_j\right)Pr\left(x_k\right){\left(x_j-x_k\right)}^2$

This interesting relation shows that the variance may in fact be defined as half the mean square of all possible variate differences, that is to say, without reference to deviations from a central value, the mean. [8] (p. 42)

The “interesting relation” also extends to the covariance. Suppose there are two random variables X with values $x_i$ for $i=1,\dots ,n$ and Y with values $y_j$ for $j=1,\dots ,m$ with probabilities given by a joint distribution $p\left(x_i,y_j\right):X\times Y\to \mathbb{R}$. The two samples or two draws methodology gives two ordered pairs $\left(x_i,y_j\right)$ and $\left(x_{i^{\prime }},y_{j^{\prime }}\right)$ so the double-variance formula ${\sum }_{j\ne k}Pr\left(x_j\right)Pr\left(x_k\right){\left(x_j-x_k\right)}^2$ generalizes to:

${\sum }_{\left(i,j\right)\ne \left(i^{\prime },j^{\prime }\right)}p\left(x_i,y_j\right)p\left(x_{i^{\prime }},y_{j^{\prime }}\right)\left(x_i-x_{i^{\prime }}\right)\left(y_j-y_{j^{\prime }}\right)$

which is similarly equal to twice the covariance $Cov(X,Y)=E\left(XY\right)-E\left(X\right)E\left(Y\right)$.

Since $\left(x_i-x_{i^{\prime }}\right)\left(y_j-y_{j^{\prime }}\right)=0$ if $i=i^{\prime }$ or $j=j^{\prime }$, we can sum over all $i,j$. Abbreviating $p\left(x_i,y_j\right)=p_{ij}$, we have:

${\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}\left(x_i-x_{i^{\prime }}\right)\left(y_j-y_{j^{\prime }}\right)={\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}\left[x_i{y}_j-x_i{y}_{j^{\prime }}-x_{i^{\prime }}{y}_j+x_{i^{\prime }}{y}_{j^{\prime }}\right]$

$={\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}{x}_i{y}_j-{\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}{x}_i{y}_{j^{\prime }}-{\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}{x}_{i^{\prime }}{y}_j+{\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}{x}_{i^{\prime }}{y}_{j^{\prime }}$.

Then using:

${\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}{x}_i{y}_j={\sum }_{i,j}{p}_{ij}{x}_i{y}_j{\sum }_{i^{\prime },j^{\prime }}{p}_{i^{\prime }{j}^{\prime }}={\sum }_{ij}{p}_{ij}{x}_i{y}_j=E\left(XY\right)$, and

${\sum }_{i,j,i^{\prime },j^{\prime }}{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}{x}_i{y}_{j^{\prime }}={\sum }_{i,j^{\prime }}{x}_i{y}_{j^{\prime }}{\sum }_{i^{\prime }}{\sum }_j{p}_{ij}{p}_{i^{\prime }{j}^{\prime }}={\sum }_{i,j^{\prime }}{x}_i{y}_{j^{\prime }}{\sum }_{i^{\prime }}{p}_i{p}_{i^{\prime }{j}^{\prime }}$

$={\sum }_{i,j/}{p}_i{x}_i{y}_{j^{\prime }}{p}_{j^{\prime }}=\left({\sum }_i{p}_i{x}_i\right)\left({\sum }_{j^{\prime }}{p}_{j^{\prime }}{y}_{j^{\prime }}\right)=E\left(X\right)E\left(Y\right)$

and similarly for the other cases, we have:

${\sum }_{\left(i,j\right)\ne \left(i^{\prime },j^{\prime }\right)}p\left(x_i,y_j\right)p\left(x_{i^{\prime }},y_{j^{\prime }}\right)\left(x_i-x_{i^{\prime }}\right)\left(y_j-y_{j^{\prime }}\right)$

$=E\left(XY\right)-E\left(X\right)E\left(Y\right)-E\left(Y\right)E\left(X\right)+E\left(XY\right)=2Cov(X,Y)$.

Using the lexicographical ordering of the ordered pairs of indices (i.e., ordering according to first index, or if $i=i^{\prime }$, then according to the second index), we have:

${\sum }_{\left(i,j\right)<\left(i^{\prime },j^{\prime }\right)}p\left(x_i,y_j\right)p\left(x_{i^{\prime }},y_{j^{\prime }}\right)\left(x_i-x_{i^{\prime }}\right)\left(y_j-y_{j^{\prime }}\right)=Cov\left(X,Y\right)$.

4. Discussion and Conclusions

These results shed new light on the variance in terms of the fundamental duality that starts in the two dual logics of subsets and partitions, and extends throughout the exact sciences [4]. The aspects discussed in this paper are given in Table 2.

We see the one-draw methodology in the notion of the probability of a subset (the one-draw-from-U probability of getting an element of a subset S) and the two-draw methodology in the notion of the logical entropy of a partition (the two-draw-from-U probability of getting a distinction of a partition $\pi$). Applied to a random variable $X:U\to \mathbb{R}$, the one-draw method gives the mean and the two-draw method gives the variance.

The quantitative versions of the two logics associates the one-draw method with probability theory and the two-draw method with information theory–which implies that the variance should be seen as an information-theoretic concept. The usual definition of the variance $Var\left(X\right)=E\left[{\left(X-E\left(X\right)\right)}^2\right]$ gives no hint of the two-draw approach. The Shannon entropy is often taken as a quantification of uncertainty with some integration into statistical theory ([36,37] (Chap. 11)). But the much older notion of uncertainty in statistical theory is the variance. The logical entropy is the two-draw probability of the distinctions of a partition and the variance is the two-draw probability of the square of the value-distinctions of a random variable (and similarly for the covariance). That is the new logical basis for variance and covariance that positions those concepts on the partitions/information-theory side of the fundamental subsets-partitions duality.