Non-Negative Decomposition of Multivariate Information: From Minimum to Blackwell Specific Information

1. Introduction

From computer science to neuroscience, we can find the following problem: We would like to know information about a random variable T, called the target, that we cannot observe directly. However, we can obtain information about the target indirectly from another set of variables $\mathbf{V}=\{V_1,...,V_n\}$. We can use information measures to quantify how much information any set of variables provides about the target. When doing so, we can identify the concept of redundancy: For example, if we have two identical variables $V_1=V_2$, then we can use one variable to predict the other and thus anything that this other variable can predict. Similarly, we can identify the concept of synergy: For example, if we have two independent variables and a target that corresponds to their XOR operation $T=\left(V_1\mathrm{XOR}{V}_2\right)$, then both variables provide no advantage on their own for predicting the state of T, yet their combination fully determines it. Williams and Beer [1] suggested that it is possible to characterize information as visualized by the Venn diagram for two variables $\mathbf{V}=\{V_1,V_2\}$ in Figure 1a. This decomposition attributes the total information about the target to being redundant, synergetic, or unique to a particular variable. As indicated in Figure 1a by $I(·,T)$, we can quantify three of the areas using information measures. However, this is insufficient to determine the four partial areas that represent the individual contributions. This causes the necessity to extend an information measure to either quantify the amount of redundancy or synergy between a set of variables.

Williams and Beer [1] first proposed a framework for Partial Information Decompositions (PIDs) and found favor by the community [2]. However, the proposed measure of redundancy was criticized for not distinguishing ”the same information and the same amount of information“ [3,4,5,6]. The proposal of Williams and Beer [1] focused specifically on mutual information. This work additionally studies the decomposition of any f-information or Rényi-information at discrete random variables. They have significance, among others, in parameter estimations, high-dimensional statistics, hypothesis testing, channel coding, data compression and privacy analyses [7,8].

1.1. Related Work

Most of the literature focuses on the decomposition of mutual information. Here, many alternative measures have been proposed but cannot suitably replace the original measure of Williams and Beer [1]: They are either limited to two observable variables [5,9,10], provide negative partial contributions [11,12,13], or provide results in which the partial contributions do not sum to the total amount [14]. Griffith et al. [3] studied the decomposition of zero-error information and obtained negative partial contributions. Kolchinsky [14] proposed a decomposition framework that is applicable beyond Shannon information theory, however, where the partial contributions do not sum to the total amount.

In this work, we propose a decomposition measure for replacing the one presented by William and Beer’s [14] while maintaining its desired properties. To achieve this, we combine several concepts from the literature: We use the Blackwell order, a preorder of information channels, for the decomposition and for deriving its operational interpretation, similar to Bertschinger et al. [9] and Kolchinsky [14]. We use its special case for binary input channels, the zonogon order studied by Bertschinger and Rauh [15], to achieve non-negativity at an arbitrary number of variables and provide it with a practical meaning. To utilize this special case for a general decomposition, we use the concept of a target pointwise decomposition as demonstrated by Williams and Beer [1] and related to Lizier et al. [16], Finn and Lizier [11], and Ince [12]. Finally, we apply the concepts from measuring on lattices, discussed by Knuth [17], to transform a non-negative decomposition with inclusion-exclusion relation from one information measure to another while maintaining the decomposition properties.

Remark 1.

We use the term ’target pointwise’ or simply ’pointwise’ within this work to refer to the analysis of each target state individually. This differs from [11,12,16], who use the latter term for the analysis of all joint sources-target realizations.

1.2. Contributions

In a recent work [18], we presented a decomposition of mutual information on the redundancy lattice (Figure 1b). This work aims to simplify, generalize and extend these ideas to make the following contributions to the area of Partial Information Decompositions:

• We propose a representation of distinct uncertainty and distinct information, which is used to demonstrate the unexpected behavior of the measure by Williams and Beer [1] (Section 2.2 and Section 3).
• We propose a decomposition for any f-information on both the redundancy lattice (Figure 1b) and synergy lattice (Figure 1c) that satisfies an inclusion-exclusion relation and provides a meaningful operational interpretation (Section 3.2).
• We prove that the proposed decomposition satisfies the original axioms of Williams and Beer [1] and guarantees non-negative partial information (Theorem 3).
• We propose to transform the non-negative decomposition of one information measure into another. This transformation maintains the non-negativity and its inclusion-exclusion relation under a re-definition of information addition (Section 3.3).
• We demonstrate the transformation of an f-information decomposition into a decomposition for Rényi- and Bhattacharyya-information (Section 3.3).
• We demonstrate that the proposed decomposition obtains different properties from different information measures and analyze the behavior of total variation in more detail (Section 4).
• We demonstrate the analysis of partial information flows through Markov chains (Figure 1) for each information measure on both the redundancy and synergy lattice (Section 4.2).

2. Background

This section aims to provide the required background information and introduce the used notation. Section 2.1 discusses the Blackwell order and its special case at binary targets, the zonogon order, which will be used for operational interpretations and the representation of f-information for its decomposition. Section 2.2 discusses the PID framework of Williams and Beer [1] and the relation between a decomposition based on the redundancy lattice and one based on the synergy lattice. We also demonstrate the unintuitive behavior of the original decomposition measure which will be resolved by our proposal in Section 3. Section 2.3 provides the considered definitions of f-information, Rényi-information, and Bhattacharyya information for the later demonstration of transforming decomposition results between measures.

Notation 1

(Random variables and their distribution). We use the notation T (upper case) to represent a random variable, ranging over the event space $\mathcal{T}$ (calligraphic) containing events $t\in \mathcal{T}$ (lower case), and use the notation $P_T$ (P with subscript) to indicate its probability distribution. The same convention applies to other variables, such as a random variable S with events $s\in \mathcal{S}$ and distribution $P_S$. We indicate the outer product of two probability distributions as $P_S\otimes P_T$, which assigns the product of their marginals $P_S\left(s\right)·P_T\left(t\right)$ to each event $(s,t)$ of the Cartesian product $\mathcal{S}\times \mathcal{T}$. Unless stated otherwise, we use the notation T, S and V to represent random variables throughout this work.

2.1. Blackwell and Zonogon Order

Definition 1

(Channel). A channel $\mu =T\to S$ from $\mathcal{T}$ to $\mathcal{S}$ represents a garbling of the input variable T that results in variable S. Within this work, we represent an information channel μ as (row) stochastic matrix, where each element is non-negative, and all rows sum to one.

For the context of this work, we consider a variable S to be the observation of the output from an information channel $T\to S$ from the target variable T, such that the corresponding channel can be obtained from their conditional probability distribution, as shown in Equation 1 where $\mathcal{T}=\{t_1,...,t_n\}$ and $\mathcal{S}=\{s_1,...,s_m\}$.

$$\mu =(T\to S)=P_{(S\mid T)}=\left[\begin{array}{ccc}p(s_1\mid t_1)& \cdots & p(s_m\mid t_1)\\ ⋮& \ddots & ⋮\\ p(s_1\mid t_n)& \cdots & p(s_m\mid t_n)\end{array}\right]$$

(1)

Notation 2

(Binary input channels). Throughout this work, we reserve the symbol κ for binary input channels, meaning κ signals a stochastic matrix of dimension $2\times m$. We use the notation $\overrightarrow{v}\in \kappa$ to indicate a column of this matrix.

Definition 2

(More informative [15,19]). An information channel ${\mu }_1=T\to S_1$ is more informative than another channel ${\mu }_2=T\to S_2$ if - for any decision problem involving a set of actions $a\in \Omega$ and a reward function $u:(\Omega ,\mathcal{T})\to \mathbb{R}$ that depends on the chosen action and state of the variable T - an agent with access to $S_1$ can always achieve an expected reward at least as high as another agent with access to $S_2$.

Definition 3

(Blackwell order [15,19]). The Blackwell order is a preorder of channels. A channel ${\mu }_1$ is Blackwell superior to channel ${\mu }_2$, if we can pass its output through a second channel λ to obtain an equivalent channel to ${\mu }_2$, as shown in Equation 2.

$${\mu }_2⊑{\mu }_1⟺{\mu }_2={\mu }_1·\lambda \mathit{for}\mathit{some}\mathit{stochastic}\mathit{matrix}\lambda$$

(2)

Blackwell 19 showed that a channel is more informative if and only if it is Blackwell superior. Bertschinger and Rauh [15] showed that the Blackwell order does not form a lattice for channels $\mu =T\to S$ if $\left|\mathcal{T}\right|>2$ since the ordering does not provide unique meet and join elements. However, binary target variables $\left|\mathcal{T}\right|=2$ are a special case where the Blackwell order is equivalent to the zonogon order (discussed next) and does form a lattice [15].

Definition 4

(Zonogon [15]). The zonogon $Z\left(\kappa \right)$ of a binary input channel $\kappa =T\to S$ is defined using the Minkowski sum from the collection of vector segments as shown in Equation 3. The zonogon $Z\left(\kappa \right)$ can similarly be defined as image of the unit cube ${[0,1]}^{\left|\mathcal{S}\right|}$ under the linear map of κ.

$$Z\left(\kappa \right)\left\{\sum _i{x}_i{\overrightarrow{v}}_i:0\le x_i\le 1,{\overrightarrow{v}}_i\in \kappa \right\}=\left\{\kappa a:a\in {[0,1]}^{\left|\mathcal{S}\right|}\right\}$$

(3)

The zonogon $Z\left(\kappa \right)$ is a centrally symmetric convex polygon, and the set of vectors ${\overrightarrow{v}}_i\in \kappa$ span its perimeter. Figure 2 shows the example of a binary input channel and its corresponding zonogon.

Definition 5

(Zonogon sum). The addition of two zonogons corresponds to their Minkowski sum as shown in Equation 4.

$$Z\left({\kappa }_1\right)+Z\left({\kappa }_2\right)\left\{a+b:a\in Z\left({\kappa }_1\right),b\in Z\left({\kappa }_2\right)\right\}=Z\left(\left[\begin{array}{cc}{\kappa }_1& {\kappa }_2\end{array}\right]\right)$$

(4)

Definition 6

(Zonogon order [15]). A zonogon $Z\left({\kappa }_1\right)$ is zonogon superior to another $Z\left({\kappa }_2\right)$ if and only if $Z\left({\kappa }_2\right)\subseteq Z\left({\kappa }_1\right)$.

Bertschinger and Rauh [15] showed that for binary input channels, the zonogon order is equivalent to the Blackwell order and forms a lattice (Equation 5). In the remaining work, we will only discuss binary input channels, such that the orderings of Definition 2, 3 and 6 are equivalent and can be thought of as zonogons with subset relation.

$${\kappa }_1⊑{\kappa }_2⟺Z\left({\kappa }_1\right)\subseteq Z\left({\kappa }_2\right)$$

(5)

For obtaining an interpretation of what a channel zonogon $Z\left(\kappa \right)$ represents, we can consider a binary decision problem by aiming to predict the state $t\in \mathcal{T}$ of a binary target variable T using the output of channel $\kappa =T\to S$. Any decision strategy $\lambda \in {[0,1]}^{\left|\mathcal{S}\right|\times 2}$ for obtaining a binary prediction $\widehat{T}$ can be fully characterized by its resulting pair of True-Positive Rate (TPR) and False-Positive Rate (FPR), as shown in Equation 6

$$\kappa ·\lambda =(T\to S\to \widehat{T})=P_{(\widehat{T}\mid T)}=\left[\begin{array}{cc}p(\widehat{T}=t\mid T=t)& p(\widehat{T}\ne t\mid T=t)\\ p(\widehat{T}=t\mid T\ne t)& p(\widehat{T}\ne t\mid T\ne t)\end{array}\right]=\left[\begin{array}{cc}\mathrm{TPR}& 1-\mathrm{TPR}\\ \mathrm{FPR}& 1-\mathrm{FPR}\end{array}\right]$$

(6)

Therefore, a channel zonogon $Z\left(\kappa \right)$ provides the set of all achievable (TPR,FPR)-pairs for a given channel $\kappa$ [18,20]. This can also be seen from Equation 3, where the unit cube $a\in {[0,1]}^{\left|\mathcal{S}\right|}$ represents all possible first columns of the decision strategy $\lambda$. The first column of $\lambda$ fully determines the second since each row has to sum to one. As a result, $\kappa a$ provides the (TPR,FPR)-pair for the decision strategy $\lambda =\left[\begin{array}{cc}a& (1-a)\end{array}\right]$ and the definition of Equation 3 all achievable (TPR,FPR)-pairs for predicting the state of a binary target variable. Since this will be helpful for operational interpretations, we label the axis of zonogon plots accordingly, as shown in Figure 2, and refer to regions within this plot as reachable decision regions:

Definition 7

(Reachable decision region). A reachable decision region for a binary decision problem is a set of achievable (TPR,FPR) performance pairs and can be visualized in a TPR/FPR-plot such as Figure 2.

Notation 3

(Channel lattice). We use the notation ${\kappa }_1\sqcap {\kappa }_2$ for the meet element of binary input channels under the Blackwell order and ${\kappa }_1\bigsqcup {\kappa }_2$ for their join element. We use the notation ${\top }_{\mathrm{BW}}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$ for the top element of binary input channels under the Blackwell order and ${\perp }_{\mathrm{BW}}=\left[\begin{array}{c}1\\ 1\end{array}\right]$ for the bottom element.

For binary input channels, the meet element of the Blackwell order corresponds to the zonogon intersection $Z({\kappa }_1\sqcap {\kappa }_2)=Z\left({\kappa }_1\right)\cap Z\left({\kappa }_2\right)$ and the join element of the Blackwell order corresponds to the convex hull of their union $Z({\kappa }_1\bigsqcup {\kappa }_2)=Conv(Z\left({\kappa }_1\right)\cup Z\left({\kappa }_2\right))$. Equation 7 describes this for an arbitrary number of channels.

$$Z\left(\underset{\kappa \in \mathbf{A}}{\sqcap }\kappa \right)=\bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)\mathrm{and}Z\left(\underset{\kappa \in \mathbf{A}}{⨆}\kappa \right)=Conv\left(\bigcup _{\kappa \in \mathbf{A}}Z\left(\kappa \right)\right)$$

(7)

2.2. Partial Information Decomposition

The commonly used framework for PIDs was introduced by Williams and Beer [1]. A PID is computed with respect to a particular random variable that we would like to know information about, called the target, and tries to identify from which variables that we have access to, called visible variables, we obtain this information. Therefore, this section considers sets of variables that represent their joint distribution.

Notation 4.

Throughout this work, we use the notation T for the target variable and $\mathbf{V}=\{V_1,...,V_n\}$ for the set of visible variables. We use the notation $\mathcal{P}\left(\mathbf{V}\right)$ for the power set of $\mathbf{V}$, and ${\mathcal{P}}_1\left(\mathbf{V}\right)=\mathcal{P}\left(\mathbf{V}\right)\setminus \varnothing$ for its power set without the empty set.

Definition 8

(Sources, Atoms [1]).

• A source ${\mathbf{S}}_i\in {\mathcal{P}}_1\left(\mathbf{V}\right)$ is a non-empty set of visible variables.
• An atoms $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ is a set of sources constructed by Equation 8.

  $$\mathcal{A}\left(\mathbf{V}\right)=\{\alpha \in {\mathcal{P}}_1\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right):\forall {\mathbf{S}}_a,{\mathbf{S}}_b\in \alpha ,{\mathbf{S}}_a\neg \subset {\mathbf{S}}_b\},$$

  (8)

The used filter for obtaining the set of atoms (Equation 8) removes sets that would be equivalent to other elements. This is required for obtaining a lattice from the following two ordering relations:

Definition 9

(Redundancy-/Gain-lattice [1]). The redundancy lattice $\left(\mathcal{A}\right(\mathbf{V}),\preccurlyeq )$ is obtained by applying the ordering relation of Equation 9 to all atoms $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)$.

$$\alpha \preccurlyeq \beta ⟺\forall {\mathbf{S}}_b\in \beta ,\exists {\mathbf{S}}_a\in \alpha ,{\mathbf{S}}_a\subseteq {\mathbf{S}}_b$$

(9)

The redundancy lattice for three visible variables is visualized in Figure 3a. On this lattice, we can think of an atom as representing the information that can be obtained from all of its sources about the target T (their redundancy or informational intersection). For example, the atom $\alpha =\{\{V_1,V_2\},\{V_1,V_3\}\}$ represents on the redundancy lattice the information that is contained in both $(V_1,V_2)$ and $(V_1,V_3)$ about T. Since both sources in $\alpha$ provide the information of $V_1$, their redundancy contains at least this information, and the atom $\beta =\left\{\left\{V_1\right\}\right\}$ is considered its predecessor. Therefore, the ordering indicates an informational subset relation for the redundancy of atoms, and the information that is represented by an atom increases as we move up. The up-set of an atom $\alpha$ on the redundancy lattice indicates the information that is lost when losing all of its sources. Considering the example from above, if we lose access to $\left\{V_1\left(\mathrm{or}\right)V_2\right\}$ and $\left\{V_1\left(\mathrm{or}\right)V_3\right\}$, then we lose access to all atoms in the up-set of $\alpha =\{\{V_1,V_2\},\{V_1,V_3\}\}$.

Definition 10

(Synergy-/Loss-lattice [21]). The synergy lattice $\left(\mathcal{A}\right(\mathbf{V}),⪯)$ is obtained by applying the ordering relation of Equation 10 to all atoms $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)$.

$$\alpha ⪯\beta ⟺\forall {\mathbf{S}}_b\in \beta ,\exists {\mathbf{S}}_a\in \alpha ,{\mathbf{S}}_b\subseteq {\mathbf{S}}_a$$

(10)

The synergy lattice for three visible variables is visualized in Figure 3b. On this lattice, we can think of an atom as representing the information that is contained in neither of its sources (information outside their union). For example, the atom $\alpha =\{\{V_1,V_2\},\{V_1,V_3\}\}$ represents on the synergy lattice the information that is obtained from neither $(V_1,V_2)$ nor $(V_1,V_3)$ about T. The ordering again indicates their expected subset relation: the information that is obtained from neither $\left\{V_1\left(\mathrm{and}\right)V_2\right\}$ nor $\left\{V_1\left(\mathrm{and}\right)V_3\right\}$ is fully contained in the information that cannot be obtained from $\beta =\left\{\left\{V_1\right\}\right\}$ and thus $\alpha$ is a predecessor of $\beta$.

With an intuition for both ordering relations in mind, we can see how the filter in the construction of atoms (Equation 8) removes sets that would be equivalent to another atom: the set $\{\{V_1,V_2\},\left\{V_1\right\}\}$ is removed from the power set of sources since it would be equivalent to the atom $\left\{\left\{V_1\right\}\right\}$ under the ordering of the redundancy lattice and equivalent to the atom $\left\{\{V_1,V_2\}\right\}$ under the ordering of the synergy lattice.

Notation 5

(Redundancy/Synergy lattices). We use the notation $\left(\mathcal{A}\right(V),⋎,⋏)$ for the join and meet operators on the redundancy lattice, and $\left(\mathcal{A}\right(V),\vee ,\wedge )$ for the join and meet operators on the synergy lattice. We use the notation ${\top }_{\mathrm{RL}}=\left\{\mathbf{V}\right\}$ for the top and ${\perp }_{\mathrm{RL}}=\varnothing$ for the bottom atom on the redundancy lattice, and ${\top }_{\mathrm{SL}}=\varnothing$ and ${\perp }_{\mathrm{SL}}=\left\{\mathbf{V}\right\}$ for the top and bottom atom on the synergy lattice. For an atom α, we use the notation $\downarrow \alpha$ for its down-set, $\dot{\downarrow }\alpha$ for its strict down-set, and ${\alpha }^{-}$ for its cover set. These definitions will only appear in the Möbius inverse of a function that is directly associated with either the synergy or redundancy lattice such that there is no ambiguity about which ordering relation has to be considered.

The redundant, unique, or synergetic information (partial contributions) can be calculated based on either lattice. They are obtained by quantifying each atom of the redundancy or synergy lattice with a cumulative measure that increases as we move up in the lattice. The partial contributions are then obtained in a second step from a Möbius inverse.

Definition 11

([Cumulative] redundancy measure [1]). A redundancy measure $I_{\cap }(\alpha ;T)$ is a function that assigns a real value to each atom of the redundancy lattice. It is interpreted as a cumulative information measure that quantifies the redundancy between all sources $\mathbf{S}\in \alpha$ of an atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ about the target T.

Definition 12

([Cumulative] loss measure [21]). A loss measure $I_{\cup }(\alpha ;T)$ is a function that assigns a real value to each atom of the synergy lattice. It is interpreted as a cumulative measure that quantifies the information about T that is provided by neither of the sources $\mathbf{S}\in \alpha$ of an atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$.

To ensure that a redundancy measure actually captures the desired concept of redundancy, Williams and Beer [1] defined three axioms that a measure $I_{\cap }$ should satisfy. For the synergy lattice, we consider the equivalent axioms discussed by Chicharro and Panzeri [21]:

Axiom 1

(Commutativity [1,21]). Invariance in the order of sources (σ permuting the order of indices):

$$\begin{array}{cc}\hfill I_{\cap }(\{{\mathbf{S}}_1,...,{\mathbf{S}}_i\};T)& =I_{\cap }(\{{\mathbf{S}}_{\sigma \left(1\right)},...,{\mathbf{S}}_{\sigma \left(i\right)}\};T)\hfill \\ \hfill I_{\cup }(\{{\mathbf{S}}_1,...,{\mathbf{S}}_i\};T)& =I_{\cup }(\{{\mathbf{S}}_{\sigma \left(1\right)},...,{\mathbf{S}}_{\sigma \left(i\right)}\};T)\hfill \end{array}$$

Axiom 2

(Monotonicity [1,21]). Additional sources can only decrease redundant information. Additional sources can only decrease the information that is in neither source.

$$\begin{array}{cc}\hfill I_{\cap }(\{{\mathbf{S}}_1,...,{\mathbf{S}}_{i-1}\};T)& \ge I_{\cap }(\{{\mathbf{S}}_1,...,{\mathbf{S}}_i\};T)\hfill \\ \hfill I_{\cup }(\{{\mathbf{S}}_1,...,{\mathbf{S}}_{i-1}\};T)& \ge I_{\cup }(\{{\mathbf{S}}_1,...,{\mathbf{S}}_i\};T)\hfill \end{array}$$

Axiom 3

(Self-redundancy [1,21]). For a single source, redundancy equals mutual information. For a single source, the information loss equals the difference between the total available mutual information and the mutual information of the considered source with the target.

$$\begin{array}{c}\hfill I_{\cap }(\left\{{\mathbf{S}}_i\right\};T)=I({\mathbf{S}}_i;T)and{I}_{\cup }(\left\{{\mathbf{S}}_i\right\};T)=I(\mathbf{V};T)-I({\mathbf{S}}_i;T)\end{array}$$

The first axiom states that an atom’s redundancy and information loss should not depend on the order of its sources. The second axiom states that adding sources to an atom can only decrease the redundancy of all sources (redundancy lattice) and decrease the information from neither source (synergy lattice). The third axiom binds the measures to be consistent with mutual information and ensures that the bottom element of both lattices is quantified to zero.

Once a lattice with corresponding cumulative measure ($I_{\cap }$/$I_{\cup }$) is defined, we can use the Möbius inverse to compute the partial contribution of each atom. This partial information can be visualized as partial area in a Venn diagram (see Figure 1a) and corresponds to the desired redundant, unique, and synergetic contributions. However, the same atom represents different partial contributions on each lattice: As visualized for the case of two visible variables in Figure 1, the unique information of variable $V_1$ is represented by $\alpha =\left\{\left\{V_1\right\}\right\}$ on the redundancy lattice and by $\beta =\left\{\left\{V_2\right\}\right\}$ on the synergy lattice.

Definition 13

(Partial information [1,21]). Partial information $\Delta I_{\cap }(\alpha ;T)$ and $\Delta I_{\cup }(\alpha ;T)$ corresponds to the Möbius inverse of its corresponding cumulative measure on the respective lattice.

$$\begin{array}{ccccc}\hfill \mathit{Redundancy}\text{-}\mathit{Lattice}\text{:}& \hfill & \hfill \Delta I_{\cap }(\alpha ;T)& =I_{\cap }(\alpha ;T)-\sum _{\beta \in \dot{\downarrow }\alpha }\Delta I_{\cap }(\beta ;T),\hfill & \end{array}$$

(11a)

$$\begin{array}{cccc}\hfill \mathit{Synergy}\text{-}\mathit{Lattice}\text{:}& \hfill & \hfill \Delta I_{\cup }(\alpha ;T)& =I_{\cup }(\alpha ;T)-\sum _{\beta \in \dot{\downarrow }\alpha }\Delta I_{\cup }(\beta ;T).\hfill \end{array}$$

(11b)

Remark 2.

Using the Möbius inverse for defining partial information enforces an inclusion-exclusion relation in that all partial information contributions have to sum to the corresponding cumulative measure. Kolchinsky [14] argues that an inclusion-exclusion relation should not be expected to hold for PIDs and proposes an alternative decomposition framework. In this case, the sum of partial contributions (unique/redundant/synergetic information) is no longer expected to sum to the total amount $I(\mathbf{V};T)$.

Property 1

(Local positivity, non-negativity [1]). A partial information decomposition satisfies non-negativity or local positivity if its partial information contributions are always non-negative, as shown in Equation 12.

$$\forall \alpha \in \mathcal{A}\left(V\right).\Delta I_{\cap }(\alpha ;T)\ge 0or\Delta I_{\cup }(\alpha ;T)\ge 0$$

(12)

The non-negativity property is important if we assume an inclusion-exclusion relation since it states that the unique, redundant, or synergetic information cannot be negative. If an atom $\alpha$ provides a negative partial contribution in the framework of Williams and Beer [1], then this may indicate that we over-counted some information in its down-set.

Remark 3.

Several additional axioms and properties have been suggested since the original proposal of Williams and Beer [1], such as target monotonicity and target chain rule [4]. However, this work will only consider the axioms and properties of Williams and Beer [1]. To the best of our knowledge, no other measure since the original proposal (discussed below) has been able to satisfy these properties for an arbitrary number of visible variables while ensuring an inclusion-exclusion relation for their partial contributions.

It is possible to convert between both representations due to a lattice duality:

Definition 14

(Lattice duality and dual decompositions [21]). Let $C=\left(\mathcal{A}\right(\mathbf{V}),\preccurlyeq )$ be a redundancy lattice with associated measure $I_{\cap }$ and let $D=\left(\mathcal{A}\right(\mathbf{V}),⪯)$ be a synergy lattice with measure $I_{\cup }$, then the two decompositions are said to be dual if and only if the down-set on one lattice corresponds to the up-set in the other as shown in Equation 13.

$$\begin{array}{cc}\hfill \forall \alpha \in C,\exists \beta \in D& :\Delta I_{\cap }(\alpha ;T)=\Delta I_{\cup }(\beta ;T)\hfill \end{array}$$

(13a)

$$\begin{array}{cc}\hfill \forall \alpha \in D,\exists \beta \in C& :\Delta I_{\cup }(\alpha ;T)=\Delta I_{\cap }(\beta ;T)\hfill \end{array}$$

(13b)

$$\begin{array}{cc}\hfill \forall \alpha \in C,\exists \beta \in D& :\phantom{\Delta }{I}_{\cap }(\alpha ;T)=\sum _{\gamma \in \downarrow \alpha }\Delta I_{\cap }(\gamma ;T)=\sum _{\gamma \in \uparrow \beta }\Delta I_{\cup }(\gamma ;T)\hfill \end{array}$$

(13c)

$$\begin{array}{cc}\hfill \forall \alpha \in D,\exists \beta \in C& :\phantom{\Delta }{I}_{\cup }(\alpha ;T)=\sum _{\gamma \in \downarrow \alpha }\Delta I_{\cup }(\gamma ;T)=\sum _{\gamma \in \uparrow \beta }\Delta I_{\cap }(\gamma ;T)\hfill \end{array}$$

(13d)

Williams and Beer [1] proposed $I_{\cap }^{min}$, as shown in Equation 14, to be used as measure of redundancy and demonstrated that it satisfies the three required axioms and local positivity. They define redundancy (Equation 14b) as the expected value of the minimum specific information (Equation 14a).

Remark 4.

Throughout this work, we use the term ’target pointwise information’ or simply ’pointwise information’ to refer to ’specific information’. This shall avoid confusion when naming their corresponding binary input channels in Section 3.

$$\begin{array}{cc}\hfill I({\mathbf{S}}_i;T=t)& =\sum _{s\in {\mathcal{S}}_i}p(s\mid t)\left[log\left(\frac{1}{p\left(t\right)}\right)-log\left(\frac{1}{p(t\mid s)}\right)\right]\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill I_{\cap }^{min}({\mathbf{S}}_1,...,{\mathbf{S}}_k;T)& =\sum _{t\in \mathcal{T}}p\left(t\right)\underset{i\in 1..k}{min}I({\mathbf{S}}_i;T=t).\hfill \end{array}$$

(14b)

To the best of our knowledge, this measure is the only existing non-negative decomposition that satisfies all three axioms listed above for an arbitrary number of visible variables while providing an inclusion-exclusion relation of partial information.

However, the measure $I_{\cap }^{min}$ could be criticized for not providing a notion of distinct information due to its use of a pointwise minimum (for each $t\in \mathcal{T}$) over the sources. This leads to the question of distinguishing ”the same information and the same amount of information“ [3,4,5,6]. We can use the definition through a pointwise minimum (Equation 14) to construct examples of unexpected behavior: consider for example a uniform binary target variable T and two visible variables as output of the channels visualized in Figure 4. The channels are constructed to be equivalent for both target states and provide access to distinct decision regions while ensuring a constant pointwise information $\forall t\in \mathcal{T}:I(V_x,T=t)=0.2$.

Even though our ability to predict the target variable significantly depends on which of the two indicated channel outputs we observe (blue or green in Figure 4), the measure $I_{\cap }^{min}$ concludes full redundancy between them $I(V_1;T)=I_{\cap }^{min}(\{V_1,V_2\};T)=I(V_2,T)=0.2$. We think this behavior is undesired and, as discussed in the literature, caused by an underlying lack of distinguishing the same information. To resolve this issue, we will present a representation of f-information in Section 3.1, which allows the use of all (TPR,FPR)-pairs for each state of the target variable to represent a distinct notion of uncertainty.

2.3. Information Measures

This section discusses two generalizations of mutual information at discrete random variables based on f-divergences and Rényi divergences [22,23]. While mutual information has interpretational significance in channel coding and data compression, other f-divergences have their significance in parameter estimations, high-dimensional statistics, and hypothesis testing [7, p. 88], while Rényi-divergences can be found among others in privacy analysis [8]. Finally, we introduce Bhattacharyya information for demonstrating that it is possible to chain decomposition transformations in Section 3.3. All definitions in this section only consider the case of discrete random variables (which is what we need for the context of this work).

Definition 15

(f-divergence [22]). Let $f:(0,\infty )\to \mathbb{R}$ be a function that satisfies the following three properties.

• f is convex,
• $f\left(1\right)=0$,
• $f\left(z\right)$ is finite for all $z>0$.

By convention we understand that $f\left(0\right)={lim}_{z\to 0^{+}}f\left(z\right)$ and $0f\left(\frac{0}{0}\right)=0$. For any such function f and two discrete probability distributions P and Q over the event space $\mathcal{X}$, the f-divergence for discrete random variables is defined as shown in Equation 15.

$$D_f(P\Vert Q)\sum _{x\in \mathcal{X}}Q\left(x\right)f\left(\frac{P\left(x\right)}{Q\left(x\right)}\right)={\mathbb{E}}_Q\left[f\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)\right]$$

(15)

Notation 6.

Throughout this work, we reserve the name f for functions that satisfy the required properties for an f-divergence of Definition 15.

An f-divergence quantifies a notion of dissimilarity between two probability distributions P and Q. Key properties of f-divergences are their non-negativity, their invariance under bijective transformations, and them satisfying a data-processing inequality [7, p. 89]. A list of commonly used f-divergences is shown in Table 1. Notably, the continuation for $a=1$ of both the Hellinger- and $\alpha$-divergence result in the KL-divergence [24].

The generator function of an f-divergence is not unique since $D_{f\left(z\right)}=D_{f\left(z\right)+c(z-1)}$ for a real constant $c\in \mathbb{R}$ [7]. As a result, the considered $\alpha$-divergence is a linear scaling of the Hellinger divergence ($D_{{\mathrm{H}}_a}=a·D_{\alpha =a}$) as shown in Equation 16.

$$\frac{z^a-1}{a-1}+c(z-1)=a·\frac{z^a-1-a(z-1)}{a(a-1)}\mathrm{for}c=-\frac{a}{a-1}$$

(16)

Definition 16

(f-information [7]). An f-information is defined based on an f-divergence from the joint distribution of two discrete random variables and the product of their marginals as shown in Equation 17.

$$\begin{array}{cc}\hfill I_f(S;T)& :=D_f\left(P_{(S,T)}\Vert P_S\otimes P_T\right)\hfill \\ \hfill & =\sum _{(s,t)\in \mathcal{S}\times \mathcal{T}}{P}_S\left(s\right)·P_T\left(t\right)·f\left(\frac{P_{(S,T)}(s,t)}{P_S\left(s\right)·P_T\left(t\right)}\right)\hfill \\ \hfill & =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)\left[\sum _{s\in \mathcal{S}}{P}_S\left(s\right)·f\left(\frac{P_{S\mid T}(s\mid t)}{P_S\left(s\right)}\right)\right]\hfill \end{array}$$

(17)

Definition 17

(f-entropy). A notion of f-entropy for a discrete random variable is obtained from the self-information of a variable $H_f\left(T\right)I_f(T;T)$.

Notation 7.

Using the KL-divergence results in the definition of mutual information and Shannon entropy. Therefore, we use the notation $I_{\mathrm{KL}}$ for mutual information (KL-information) and $H_{\mathrm{KL}}$ (KL-entropy ) for the Shannon entropy.

The remaining part of this section will define Rényi- and Bhattacharyya-information to highlight that they can be represented as an invertible transformation of Hellinger-information. This will be used in Section 3.3 to transform the decomposition of Hellinger-information to a decomposition of Rényi- and Bhattacharyya-information.

Remark 5.

We could similarly choose to represent Renyi divergence as a transformation of the α-divergence. A liner scaling of the considered f-divergence will however not affect our later results (see Section 3.3).

Definition 18

(Rényi divergence [23]). Let P and Q be two discrete probability distributions over the event space $\mathcal{X}$, then Rényi divergence $R_a$ is defined as shown in Equation 18 for $a\in (0,1)\cup (1,\infty )$, and extended to $a\in \{0,1,\infty \}$ by continuation.

$$\begin{array}{cc}\hfill R_a(P\Vert Q)& :=\frac{1}{a-1}log\left({\mathbb{E}}_Q\left[{\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)}^a\right]\right)\hfill \\ \hfill & =\frac{1}{a-1}log\left(1+(a-1){\mathbb{E}}_Q\left[\frac{{\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)}^a-1}{a-1}\right]\right)\hfill \\ \hfill & =\frac{1}{a-1}log\left(1+(a-1)D_{{\mathrm{H}}_a}(P\Vert Q)\right)\hfill \end{array}$$

(18)

Notably, the continuation of Rényi divergence for $a=1$ also equals the KL-divergence [7, p. 116]. Renyi divergence can be expressed as an invertible transformation of the Hellinger divergence ($D_{{\mathrm{H}}_a}$, see Equation 18) [24].

Definition 19

(Rényi-information [7]). Rényi-information is defined equivalent to f-information as shown in Equation 19 and corresponds to an invertible transformation of Hellinger-information ($I_{{\mathrm{H}}_a}$).

$$\begin{array}{cc}\hfill I_{R_a}(S;T)& :=R_a\left(P_{(S;T)}\Vert P_S\otimes P_T\right)\hfill \\ \hfill & =\frac{1}{a-1}log\left(1+(a-1)I_{{\mathrm{H}}_a}(S;T)\right)\hfill \end{array}$$

(19)

Finally, we consider the Bhattacharyya distance (Definition 20), which is equivalent to a linear scaling from a special case of Rényi divergence (Equation 20) [24]. It is applied, among others, in signal processing [25] and coding theory [26]. The corresponding information measure (Equation 21) is like its distance the scaling of a special case of Rényi-information.

Definition 20

(Bhattacharyya distance [27]). Let P and Q be two discrete probability distributions over the event space $\mathcal{X}$, then the Bhattacharyya distance is defined as shown in Equation 20.

$$\begin{array}{cc}\hfill B(P\Vert Q)& :=-log\left(\sum _{x\in \mathcal{X}}\sqrt{P\left(x\right)Q\left(x\right)}\right)\hfill \\ \hfill & =-log\left(\sum _{x\in \mathcal{X}}Q\left(x\right)\sqrt{\frac{P\left(x\right)}{Q\left(x\right)}}\right)\hfill \\ \hfill & =-log\left(1-0.5·{\mathbb{E}}_Q\left[\frac{{\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)}^{0.5}-1}{0.5-1}\right]\right)\hfill \\ \hfill & =-log\left(1-0.5·D_{{\mathrm{H}}_{0.5}}(P\Vert Q)\right)\hfill \\ \hfill & =0.5·R_{0.5}(P\Vert Q)\hfill \end{array}$$

(20)

Definition 21

(Bhattacharyya-information). Bhattacharyya-information is defined equivalent to f-information as shown in Equation 21.

$$I_{\mathrm{B}}(S;T)B\left(P_{(S,T)}\Vert P_S\otimes P_T\right)=0.5·I_{R_{0.5}}(S;T)$$

(21)

3. Decomposition Methodology

To construct a partial information decomposition in the framework of Williams and Beer [1], we only have to define a cumulative redundancy measure ($I_{\cap }$) or cumulative loss measure ($I_{\cup }$). However, doing this requires a meaningful definition of when information is the same. Therefore, Section 3.1 presents an interpretation of f-information that enables a representation of distinct information. Specifically, we demonstrate that f-information quantifies the expected perimeter for the channel zonogons of each state $t\in \mathcal{T}$. This allows for the interpretation that each distinct (TPR,FPR)-pair for predicting a state of the target variable provides a distinct notion of uncertainty. Information corresponds accordingly to reachable decision regions, which are quantified by their perimeter. This interpretation of f-information is used in Section 3.2 to construct a partial information decomposition under the zonogon order for each state $t\in \mathcal{T}$ individually. These individual decompositions are then combined into the final result. Therefore, we decompose specific information based on the Blackwell order rather than using its minimum, like Williams and Beer [1]. We use the resulting decomposition of any f-information in Section 3.3 to transform a Hellinger-information decomposition into a Rényi-information decomposition while maintaining its non-negativity and an inclusion-exclusion relation. In Section 3.2 and Section 3.3, we first demonstrate the decomposition on the synergy lattice and then its corresponding dual decomposition on the redundancy lattice. To achieve the desired axioms and properties, we combine different aspects of the existing literature:

• Like Bertschinger et al. [9] and Kolchinsky [14] we base the decomposition on the Blackwell order and use this to obtain the operational interpretation of the decomposition.
• Like Williams and Beer [1] and related to Lizier et al. [16], Finn and Lizier [11], and Ince [12], we perform a decomposition from a pointwise perspective but only for the target variable.
• In a similar manner to how Finn and Lizier [11] used probability mass exclusion to differentiate distinct information, we use the achievable decision regions for each state of a target variable to differentiate distinct information.
• We propose applying the concepts about lattice re-graduations discussed by Knuth [17] to PIDs to transform the decomposition of one information measure to another while maintaining its consistency.

We extend Axiom 3 of Williams and Beer [1] as shown below, to allow binding any information measure to the decomposition.

Axiom 3* (Self-redundancy).

For a single source, redundancy $I_{\cap ,*}$ and information loss $I_{\cup ,*}$ correspond to information measure $I_{*}$ as shown below:

$$I_{\cap ,*}(\left\{{\mathbf{S}}_i\right\};T)=I_{*}({\mathbf{S}}_i;T)\mathit{and}{I}_{\cup ,*}(\left\{{\mathbf{S}}_i\right\};T)=I_{*}(\mathbf{V};T)-I_{*}({\mathbf{S}}_i;T)$$

(22)

3.1. Representing f-Information

We begin with an interpretation of f-information, for which we define a pointwise variable $\pi (T,t)$ that represents one state of the target variable (Equation 23a) and construct its pointwise information channel (Definition 22). Then, we define a function $r_f$ based on the generator function of an f-divergence. This function acts like a pseudo-distance for measuring half the length of the zonogon perimeters for each pointwise information channel (see Figure 2). These zonogon perimeter lengths are pointwise f-information.

Definition 22

([Target] pointwise binary input channel). We define a target pointwise binary input channel $\kappa (\mathbf{S},T,t)$ from one state of the target variable $t\in \mathcal{T}$ to an information source $\mathbf{S}$ with event space $\mathcal{S}=\{s_1,\cdots ,s_m\}$ as shown in Equation 23b.

$$\begin{array}{cc}\hfill \pi (T,t)& \left\{\begin{array}{cc}1\hfill & \mathit{if}T=t\hfill \\ 0\hfill & \mathit{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(23a)

$$\begin{array}{cc}\hfill \kappa (\mathbf{S},T,t)\pi (T,t)\to \mathbf{S}& =\left[\begin{array}{ccc}p\left(S=s_1\mid T=t\right)& \cdots & p\left(S=s_m\mid T=t\right)\\ p\left(S=s_1\mid T\ne t\right)& \cdots & p\left(S=s_m\mid T\ne t\right)\end{array}\right]\hfill \end{array}$$

(23b)

Definition 23

([Target] pointwise f-information).

• We define a function $r_f$ as shown in Equation 24a to quantify a vector, where $0\le p,x,y\le 1$.
• We define a target pointwise f-information function $i_f$, as shown in Equation 24b, to quantify half the zonogon perimeter for the corresponding pointwise channel $Z\left(\kappa \right(\mathbf{S},T,t\left)\right)$.

$$\begin{array}{cc}\hfill r_f\left(p,\left[\begin{array}{c}x\\ y\end{array}\right]\right)& :=\left(px+(1-p)y\right)·f\left(\frac{x}{px+(1-p)y}\right)\hfill \end{array}$$

(24a)

$$\begin{array}{cc}\hfill i_f\left(p,\kappa \right)& :=\sum _{\overrightarrow{v}\in \kappa }{r}_f(p,\overrightarrow{v})\hfill \end{array}$$

(24b)

Theorem 1

(Properties of $r_f$). For a constant $0\le p\le 1$: (1) the function $r_f(p,\overrightarrow{v})$ is convex in $\overrightarrow{v}$, (2) scales linearly in $\overrightarrow{v}$, (3) satisfies a triangle inequality in $\overrightarrow{v}$, (4) quantifies any vector of slope one to zero, and (5) quantifies the zero vector to zero.

Proof.

• The convexity of $r_f(p,\overrightarrow{v})$ in $\overrightarrow{v}$ is shown separately in Lemma A1 of Appendix A.
• That $r_f(p,\ell \overrightarrow{v})=\ell r_f(p,\overrightarrow{v})$ scales linearly in $\overrightarrow{v}$ can directly be seen from Equation 24a.
• The triangle inequality of $r_f(p,\overrightarrow{v})$ in $\overrightarrow{v}$ is shown separately in Corollary A1 of Appendix A.
• A vector of slope one is quantified to zero $r_f(p,\left[\begin{array}{c}\ell \\ \ell \end{array}\right])=\ell ·f\left(1\right)=0$, since $f\left(1\right)=0$ is a requirement on the generator function of an f-divergence (Definition 15).
• The zero vector is quantified to zero $r_f(p,\left[\begin{array}{c}0\\ 0\end{array}\right])=0·f\left(\frac{0}{0}\right)=0$ by the convention of generator functions for an f-divergence (Definition 15).

□

The properties of the function $r_f$ are useful since we can interpret it as a pseudo-distance when quantifying the perimeter length of a channel zonogon $Z\left(\kappa \right)$. The function $r_f$ provides the following properties to the pointwise information measure $i_f$.

Theorem 2

(Properties of $i_f$). The pointwise information measure $i_f$ (1) maintains the ordering relation of the Blackwell order for binary input channels and (2) is non-negative.

Proof.

• That the function $r_f$ maintains the ordering relation of the Blackwell order on binary input channels is shown separately in Lemma A2 of Appendix A (Equation 25a).
• The bottom element ${\perp }_{\mathrm{BW}}=\left[\begin{array}{c}1\\ 1\end{array}\right]$ consists of a single vector of slope one, which is quantified to zero by Theorem 1 (Equation 25b). The combination with Equation 25a ensures the non-negativity.

$$\begin{array}{cc}\hfill {\kappa }_1⊑{\kappa }_2⟹& i_f(p,{\kappa }_1)\le i_f(p,{\kappa }_2),\hfill \end{array}$$

(25a)

$$\begin{array}{cc}\hfill & i_f(p,{\perp }_{\mathrm{BW}})=0.\hfill \end{array}$$

(25b)

□

An f-information corresponds to the expected value of the target pointwise f-information function defined above (Equation 26). As a result, we can interpret f-information as quantifying (half) the expected zonogon perimeter length for the pointwise channels $Z\left(\kappa \right(\mathbf{S},T,t\left)\right)$, where the function $r_f$ acts as a pseudo-distance.

$$\begin{array}{cc}\hfill I_f(\mathbf{S};T)& =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·i_f\left(P_T\left(t\right),\kappa (\mathbf{S},T,t)\right)\hfill \\ \hfill & =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·\left[\sum _{\overrightarrow{v}\in \kappa (\mathbf{S},T,t)}{r}_f\left(P_T\left(t\right),\overrightarrow{v}\right)\right]\hfill \\ \hfill & =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·\left[\sum _{s\in \mathcal{S}}{P}_S\left(s\right)·f\left(\frac{P_{S\mid T}(s\mid t)}{P_S\left(s\right)}\right)\right]\hfill \end{array}$$

(26)

3.2. Decomposing f-Information

With the representation of Section 3.1 in mind, we can define a non-negative partial information decomposition for a set of visible variables $\mathbf{V}=\{V_1,...,V_n\}$ about a target variable T for any f-information. The decomposition is performed from a pointwise perspective, which means that we decompose the pointwise measure $i_f$ on the synergy lattice $\left(\mathcal{A}\right(\mathbf{V}),⪯)$ for each $t\in \mathcal{T}$. The pointwise synergy lattices are then combined using a weighted sum to obtain the decomposition of $I_f$.

We map each atom of the synergy lattice to the join of pointwise channels for its contained sources.

Definition 24

(From atoms to channels). We define the channel corresponding to an atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ as shown in Eqation 27.

$${\kappa }_{\bigsqcup }(\alpha ,T,t)\left\{\begin{array}{cc}{\perp }_{\mathrm{BW}}\hfill & \mathit{if}\alpha ={\top }_{\mathrm{SL}}=\varnothing \hfill \\ \underset{\mathbf{S}\in \alpha }{⨆}\kappa (\mathbf{S},T,t))\hfill & \mathit{otherwise}\hfill \end{array}\right.$$

(27)

Lemma 1.

For any set of sources $\alpha ,\beta \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ and target variable T with state $t\in \mathcal{T}$, the function ${\kappa }_{\bigsqcup }$ maintains the ordering of the synergy lattice under the Blackwell order as shown in Equation 28.

$$\alpha ⪯\beta ⟹{\kappa }_{\bigsqcup }(\beta ,T,t)⊑{\kappa }_{\bigsqcup }(\alpha ,T,t)$$

(28)

Lemma 1 is shown separately in Section B.1 of Appendix B. The mapping from Definition 24 provides a lattice that can be quantified using pointwise f-information to construct a cumulative loss measure for its decomposition using the Möbius inverse.

Definition 25

([Target] pointwise cumulative and partial loss measures). We define the target pointwise cumulative and partial loss functions as shown in Equation 29a and 29b.

$$\begin{array}{cc}\hfill i_{\cup ,f}(\alpha ,T,t)& :=i_f\left(P_T\left(t\right),\kappa (\mathbf{V},T,t)\right)-i_f(P_T\left(t\right),{\kappa }_{\bigsqcup }(\alpha ,T,t))\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill \Delta i_{\cup ,f}(\alpha ,T,t)& :=i_{\cup ,f}(\alpha ,T,t)-\sum _{\beta \in \dot{\downarrow }\alpha }\Delta i_{\cup ,f}(\beta ,T,t)\hfill \end{array}$$

(29b)

The combined cumulative and partial measures are the expected value of their corresponding pointwise measures. This corresponds to combining the pointwise decomposition lattices by a weighted sum.

Definition 26

(Combined cumulative and partial loss measures). The cumulative loss measure $I_{\cup ,f}$ is defined by Equation 30 and the decomposition result $\Delta I_{\cup ,f}$ by Equation 31.

$$I_{\cup ,f}(\alpha ;T):=\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·i_{\cup ,f}(\alpha ,T,t)$$

(30)

$$\begin{array}{cc}\hfill \Delta I_{\cup ,f}(\alpha ;T)& :=\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·\Delta i_{\cup ,f}(\alpha ,T,t)\hfill \\ \hfill & =I_{\cup ,f}(\alpha ;T)-\sum _{\beta \in \dot{\downarrow }\alpha }\Delta I_{\cup ,f}(\beta ;T)\hfill \end{array}$$

(31)

Theorem 3.

The presented definitions for the pointwise and expected loss measures ($i_{\cup ,f}$ and $I_{\cup ,f}$) provide a non-negative PID on the synergy lattice with inclusion-exclusion relation that satisfies the Axioms 1, 2 and 3* for any f-information measure.

Proof. 

• Axiom 1: The measure $i_{\cup ,f}$ (Equation 29a) is invariant to permuting the order of sources in $\alpha$, since the join operator of the zonogon order (${⨆}_{\mathbf{S}\in \alpha }$) is. Therefore, also $I_{\cup ,f}$ satisfies Axiom 1.
• Axiom 2: The monotonicity of both $i_{\cup ,f}$ and $I_{\cup ,f}$ on the synergy lattice is shown separately as Corollary A2 in Appendix B.
• Axiom 3*: For a single source, $i_{\cup ,f}$ equals the pointwise information loss by definition (see Equation 22, 24b and 29a). Therefore, $I_{\cup ,f}$ satisfies Axiom 3*.
• Non-negativity: The non-negativity of $\Delta i_{\cup ,f}$ and $\Delta I_{\cup ,f}$ is shown separately as Lemma A7 in Appendix B.

□

Consider the pointwise and combined redundancy measures ($i_{\cap ,f}$ and $I_{\cap ,f}$) as shown in Equation 32, which are defined through an additional inclusion-exclusion relation on the sources of an atom (pointwise using Equation 32a and 32b or combined using Equation 32c). The partial contributions ($\Delta i_{\cap ,f}$ and $\Delta I_{\cap ,f}$) are obtained from the Möbius inverse. This defines the corresponding dual decomposition on the redundancy lattice.

Definition 27

(Dual decomposition on the redundancy lattice). We define the pointwise and cumulative redundancy measure as shown in Equation 32.

$$\begin{array}{cc}\hfill i_{\cap ,f}(\alpha ,T,t)& :=\sum _{\beta \in {\mathcal{P}}_1\left(\alpha \right)}{(-1)}^{\left|\beta \right|-1}{i}_f(P_T\left(t\right),{\kappa }_{\bigsqcup }(\beta ,T,t))\hfill \end{array}$$

(32a)

$$\begin{array}{cc}\hfill I_{\cap ,f}(\alpha ;T)& :=\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·i_{\cap ,f}(\alpha ,T,t)\hfill \end{array}$$

(32b)

$$\begin{array}{cc}\hfill & =I_f(\mathbf{V};T)-\sum _{\beta \in {\mathcal{P}}_1\left(\alpha \right)}{(-1)}^{\left|\beta \right|-1}{I}_{\cup ,f}(\beta ;T)\hfill \end{array}$$

(32c)

Corollary 1.

The dual decomposition as defined by Equation 32 provides a non-negative PID which satisfies an inclusion-exclusion relation and the axioms of Williams and Beer [1] on the redundancy lattice.

Proof. 

The Axioms 1 and 3* are transformed from Theorem 3 by Equation 32c. The non-negativity is obtained from Theorem 3 since the partial contributions are identical between dual decompositions. The non-negativity ensures monotonicity (Axiom 2) since the cumulative measure $I_{\cap ,f}$ is the sum of (non-negative) partial contributions in its down-set due to the Möbius inverse. □

Remark 6.

As discussed before [18], it is possible to further split the redundant component in the decomposition for extracting the pointwise meet under the Blackwell order (zonogon intersection). The second component of redundancy as defined above contains decision regions that are part of the convex hull but not the individual channel zonogons (discussed as shared information in [18]).

From a pointwise perspective, there always exists a dependency between the sources for which the synergy of this state becomes zero. This dependence corresponds, by definition, to the join of their channels. This is helpful for the operational interpretation in the following paragraph since, individually, each pointwise synergy becomes fully volatile to the dependence between the sources.

Operational interpretation: The decomposition obtains the operational interpretation that if a variable provides pointwise unique information, then there exists a unique decision region for some $t\in \mathcal{T}$ that this variable provides access to. Moreover, if a set of variables provides synergetic information, then a decision region for some $t\in \mathcal{T}$ may become inaccessible if the dependence between the variables changes. Due to the equivalence of the zonogon and Blackwell order for binary input variables, these interpretations can also be transferred to a set of actions $a\in \Omega$ and a pointwise reward function $u(a,\pi (T,t\left)\right)$, which only depends on one state of the target variable $\pi (T,t)$ (see Section 2.1): If a variable provides unique information, then it provides an advantage for some set of actions and pointwise reward function, while synergy indicates that the advantage for some pointwise reward function is based on the dependence between variables.

The implication of the interpretation does not hold in the other direction, which we will also highlight in the example of $I_{\cup ,\mathrm{TV}}$ in Section 4.1. Finally, the definition of the Blackwell order through the chaining of channels (Equation 2) highlights its suitability for tracing the flows of information in Markov chains (see Section 4.2).

3.3. Decomposing Rényi-Information

Since Rényi-information is an invertible transformation of Hellinger-information and $\alpha$-information, we argue that their decompositions should be consistent. We propose to view the decomposition of Rényi-information as a transformation from an f-information and demonstrate the approach by transferring the Hellinger-information decomposition to a Rényi-information decomposition. Then, we demonstrate that the result is invariant to a linear scaling of the considered f-information, such that the transformation from $\alpha$-information provides identical results. The obtained Rényi-information decomposition is non-negative and satisfies the three axioms proposed by Williams and Beer [1] (see below). However, its inclusion-exclusion relation is based on a transformed addition operator. For transforming the decomposition, we consider Rényi-information to be a re-graduation of Hellinger-information, as shown in Equation 33.

$$\begin{array}{cc}\hfill v_a\left(z\right)& :=\frac{1}{a-1}log\left(1+(a-1)z\right)\hfill \end{array}$$

(33a)

$$\begin{array}{cc}\hfill I_{{\mathrm{R}}_a}(\mathbf{S};T)& =v_a\left(I_{{\mathrm{H}}_a}(\mathbf{S};T)\right)\hfill \end{array}$$

(33b)

To maintain consistency when transforming the measure, we also have to transform its operators [17, p. 6 ff.]:

Definition 28

(Addition of Rényi-information). We define the addition of Rényi-information ${\oplus }_a$ with its corresponding inverse function ${\ominus }_a$ by Equation 34.

$$\begin{array}{cc}\hfill x{\oplus }_{a}y& :=v_a(v_a^{-1}\left(x\right)+v_a^{-1}\left(y\right))=\frac{log\left(e^{(a-1)x}+e^{(a-1)y}-1\right)}{a-1}\hfill \end{array}$$

(34a)

$$\begin{array}{cc}\hfill x{\ominus }_{a}y& :=v_a(v_a^{-1}\left(x\right)-v_a^{-1}\left(y\right))=\frac{log\left(e^{(a-1)x}-e^{(a-1)y}+1\right)}{a-1}\hfill \end{array}$$

(34b)

To transform a decomposition of the synergy lattice, we define the cumulative loss measures as shown in Equation 35 and use the transformed operators when computing the Möbius inverse (Equation 36a) to maintain consistency in the results (Equation 36b).

Definition 29.

The cumulative and partial Rényi-information loss measures are defined as transformations of the cumulative and partial Hellinger-information loss measures, as shown in Equation 35 and 36.

$$I_{\cup ,{\mathrm{R}}_a}(\alpha ;T):=v_a\left(I_{\cup ,{\mathrm{H}}_a}(\alpha ;T)\right)$$

(35)

$$\begin{array}{cc}\hfill \Delta I_{\cup ,{\mathrm{R}}_a}(\alpha ;T)& :=I_{\cup ,{\mathrm{R}}_a}(\alpha ;T){\ominus }_a\sum _{\beta \in \dot{\downarrow }\alpha }\Delta I_{\cup ,{\mathrm{R}}_a}(\beta ;T)\mathit{where}\text{:}+:={\oplus }_a\hfill \end{array}$$

(36a)

$$\begin{array}{cc}\hfill & =v_a(\Delta I_{\cup ,{\mathrm{H}}_a}(\alpha ;T))\hfill \end{array}$$

(36b)

Remark 7.

We show in Lemma A8 of Appendix C that re-scaling the original f-information does not affect the resulting decomposition or transformed operators. Therefore, transforming a Hellinger-information decomposition or a α-information decomposition to a Rényi-information decomposition provides identical results.

The operational interpretation presented in Section 3.2 is similarly applicable to partial Rényi-information ($\Delta I_{\cup ,{\mathrm{R}}_a}$ , Equation 36b), since the function $v_a$ satisfies $v_a\left(0\right)=0$ and $x\le 0⟹0\le v_a\left(x\right)$.

Theorem 4.

The presented definitions for the cumulative loss measure $I_{\cup ,{\mathrm{R}}_a}$ provide a non-negative PID on the synergy lattice with inclusion-exclusion relation under the transformed addition (Definition 28) that satisfies the Axioms 1, 2 and 3* for any Rényi-information measure.

Proof. 

• Axiom 1: $I_{\cup ,{\mathrm{R}}_a}(\alpha ;T)$ is invariant to permuting the order of sources, since $I_{\cup ,{\mathrm{H}}_a}(\mathbf{S};T)$ satisfies Axiom 1 (see Section 3.2).
• Axiom 2: $I_{\cup ,{\mathrm{R}}_a}(\alpha ;T)$ satisfies monotonicity, since $I_{\cup ,{\mathrm{H}}_a}(\mathbf{S};T)$ satisfies Axiom 2 (see Section 3.2) and the transformation function $v_a$ is monotonically increasing for $a\in (0,1)\cup (1,\infty )$.
• Axiom 3*: Since $I_{\cup ,{\mathrm{H}}_a}$ satisfies Axiom 3* (see Section 3.2, Equation 33 and 35), $I_{\cup ,{\mathrm{R}}_a}$ satisfies the self-redundancy axiom by definition, however, at a transformed operator: $I_{\cup ,{\mathrm{R}}_a}(\left\{{\mathbf{S}}_i\right\};T)=I_{{\mathrm{R}}_a}(\left\{\mathbf{V}\right\};T)\ominus I_{{\mathrm{R}}_a}(\left\{{\mathbf{S}}_i\right\};T)$.
• Non-negativity: The decomposition of $I_{\cup ,{\mathrm{R}}_a}$ is non negative, since $\Delta I_{\cup ,{\mathrm{H}}_a}$ is non-negative (see Section 3.2), the Möbius inverse is computed with transformed operators (Equation 36b) and the function $v_a$ satisfies $x\le 0⟹0\le v_a\left(x\right)$.

□

Remark 8.

To obtain an equivalent decomposition of Rényi-information on the redundancy lattice, we can correspondingly transform the dual decomposition from the redundancy lattice of Hellinger-Information as shown in Equation 37. The resulting decomposition will satisfy the non-negativity, axioms of Williams and Beer [1] and an inclusion-exclusion relation under the transformed operators (Definition 28) for the same reasons described above from Corollary 1.

$$\begin{array}{cc}\hfill I_{\cap ,{\mathrm{R}}_a}(\alpha ;T)& :=v_a\left(I_{\cap ,{\mathrm{H}}_a}(\alpha ;T)\right)\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\hfill \Delta I_{\cap ,{\mathrm{R}}_a}(\alpha ;T)& :=v_a(\Delta I_{\cap ,{\mathrm{H}}_a}(\alpha ;T))\hfill \end{array}$$

(37b)

Remark 9.

The relation between the redundancy and synergy lattice can be used for the definition of a bi-valution [17] in calculations as discussed in [18]. This is also possible for Rényi-information at transformed operators.

When taking the limit of Rényi-information for $a\to 1$, we obtain mutual information ($I_{\mathrm{KL}}$). Since mutual information is also an f-information, we expect its operators in the Möbius inverse to be addition. This is indeed the case (Equation 38), and the measures will be consistent.

$$\begin{array}{cc}\hfill \underset{a\to 1}{lim}x{\oplus }_{a}y& =x+y\hfill \\ \hfill \underset{a\to 1}{lim}x{\ominus }_{a}y& =x-y\hfill \end{array}$$

(38)

Finally, the decomposition of Bhattacharyya-information can be obtained by re-scaling the decomposition of Rényi-information at $a=0.5$, which causes another transform of the addition operator for the inclusion-exclusion relation.

4. Evaluation

A comparison of the proposed decomposition with other methods of the literature can be found in [18] for mutual information. Therefore, this section first compares different f-information measures at typical decomposition examples and discusses the special case of total variation (TV)-information to explain its distinct behavior. Since we can see larger differences between measures in more complex scenarios, we compare the measures by analyzing the information flows in a Markov chain. We provide the used implementation for both dual decompositions of f-information and the examples used in this work at [28].

4.1. Partial Information Decomposition

4.1.1. Comparison of Different f-Information Measures

We use the examples discussed by Finn and Lizier [11] to compare different f-information decompositions and add a generic example from [18]. All used probability distributions and their abbreviations can be found in Appendix D. We normalize the decomposition results to the f-Entropy of the target variable for the visualization in Figure 5.

Since all results are based on the same framework, they behave similarly at examples that analyze a specific aspect of the decomposition function (XOR, Unq, PwUnq, RdnErr, Tbc, AND). However, it can be observed that the decomposition of total variation (TV) appears to differ from others: (1) In all examples, total variation attributes more information to being redundant than other measures. (2) In the generic example, total variation is the only measure that does not attribute any information to being unique to variable one or synergetic. We discuss the case of total variation in Section 4.1.2 to explain its distinct behavior.

We visualize the zonogons for the generic example in Figure A2, which shall highlight that the implication of the operational interpretation does not hold in the other direction: the existence of partial information implies an advantage for the expected reward towards some state of the target variable, but an advantage for the expected reward towards some state of the target variable does not imply partial information in the example of total variation.

4.1.2. The Special Case of Total Variation

The behavior of total variation appears different compared to other f-information measures (Figure 5). This is due to total variation measuring the perimeter of a zonogon such that the result corresponds to a linear scaling of the maximal (Euclidean) height $h^{*}$ that the zonogon reaches above the diagonal as visualized in Figure 6:

Lemma 2.

a)

The pointwise total variation ($i_{\mathrm{TV}}$) is a linear scaling of the maximal (Euclidean) height $h^{*}$ that the corresponding zonogon reaches above the diagonal, as visualized in Figure 6 (Equation 39a).

b)

For a non-empty set of pointwise channels $\mathbf{A}$, pointwise total variation $i_{\mathrm{TV}}$ quantifies the join element to the maximum of its individual channels (Equation 39b).

c)

The loss measure $i_{\cup ,\mathrm{TV}}$ quantifies the meet for a set of sources on the synergy lattice to their minimum (Equation 39c).

$$\begin{array}{cc}\hfill i_{\mathrm{TV}}(p,\kappa )& =\frac{1-p}{2}\sum _{v\in \kappa }|v_x-v_y|=(1-p)\frac{h^{*}}{\sqrt{2}}\hfill \end{array}$$

(39a)

$$\begin{array}{cc}\hfill i_{\mathrm{TV}}(p,\underset{\kappa \in \mathbf{A}}{⨆}\kappa )& =\underset{\kappa \in \mathbf{A}}{max}{i}_{\mathrm{TV}}(p,\kappa )\hfill \end{array}$$

(39b)

$$\begin{array}{cc}\hfill i_{\cup ,\mathrm{TV}}(\underset{\alpha \in \mathbf{A}}{\bigwedge }\alpha ,T,t)& =\underset{\alpha \in \mathbf{A}}{min}{i}_{\cup ,\mathrm{TV}}(\alpha ,T,t)\hfill \end{array}$$

(39c)

Proof. 

The proof of the first two statements (Equation 39b and 39b) is provided separately in Appendix E, which imply the third (Equation 39c) by Definition 25. □

Quantifying the meet element on the synergy lattice to the minimum has the following consequences for total variation: (1) It attributes a minimum amount of synergy, and therefore more information to redundancy than other measures. (2) For each state of the target, at most one variable can provide unique information. In the case of $\left|\mathcal{T}\right|=2$, the pointwise channels are symmetric (see Equation 6), such that the same variable provides the maximal zonogon height both times. This is the case in the generic example of Figure 5, and the reason why at most one variable can provide unique information in this setting. However, beyond binary targets ($\left|\mathcal{T}\right|>2$), both variables may provide unique information at the same time since different sources can provide the maximal zonogon height for different target states (see later example in Figure 7).

Remark 10.

Using the pointwise minimum on the synergy lattice results in a similar structure to the proposed measure of Williams and Beer [1]. However, TV-information is based on a different pointwise measure $i_{TV}$, which displays the same behavior (Equation 39b), unlike pointwise KL-information.

4.2. Information Flow Analysis

The differences between f-information measures in Section 4.1 appear more visible in complex scenarios. Therefore, this section compares different measures in the information flow analysis of a Markov chain.

Consider a Markov chain ${\mathbf{M}}_1\to {\mathbf{M}}_2\to \cdots \to {\mathbf{M}}_5$, where ${\mathbf{M}}_i=(X_i,Y_i)$ is the joint distribution of two variables. Assume that we are interested in state three and thus define $T={\mathbf{M}}_3$ as the target variable. Using the approach described in Section 3, we can compute an information decomposition for each state ${\mathbf{M}}_i$ of the Markov chain with respect to the target. Now, we are additionally interested in how the partial information decomposition from stage ${\mathbf{M}}_i$ propagates into the next ${\mathbf{M}}_{i+1}$, as visualized in Figure 7.

Definition 30

(Partial information flow). The partial information flow of an atom $\alpha \in \mathcal{A}\left({\mathbf{M}}_i\right)$ into the atom $\beta \in \mathcal{A}\left({\mathbf{M}}_{i+1}\right)$ quantifies the redundancy between the partial contributions of their respective decomposition lattices.

Notation 8.

We use the notation $I_{\circ ,f}$ with $\circ \in \{\cup ,\cap \}$ to refer to either the loss measure $I_{\cup ,f}$ or redundancy measure $I_{\cap ,f}$. The same applies to the functions $J_{\circ \to \circ ,f}$ and $J_{\Delta \to \circ ,f}$ of Equation 40.

Let $\alpha \in \mathcal{A}\left({\mathbf{M}}_i\right)$ and $\beta \in \mathcal{A}\left({\mathbf{M}}_{i+1}\right)$, then we compute information flows equivalently on the redundancy or synergy lattice as shown in Equation 40. When using a redundancy measure $\circ =\cap$, then the strict down-set of $\alpha$ refers to the strict down-set on its redundancy lattice $(\mathcal{A}\left({\mathbf{M}}_i\right),\preccurlyeq )$ and when using a loss measure $\circ =\cup$, then the strict down-set refers to the strict down-set on its synergy lattice $(\mathcal{A}\left({\mathbf{M}}_i\right),⪯)$. We obtain the intersection of cumulative measures by quantifying their meet, which is on both lattice equivalent to their union of sources ($J_{\circ \to \circ ,f}$, Equation 40a). To obtain how much of the partial contribution of $\alpha$ can be found in the cumulative measure of $\beta$ ($J_{\Delta \to \circ ,f}$), we remove the contributions of its down-set ($\dot{\downarrow }\alpha$ on lattice for $\mathcal{A}\left({\mathbf{M}}_i\right)$, see Equation 40b). To finally obtain the flow from the partial contribution of $\alpha$ to the partial contribution of $\beta$ ($J_{\Delta \to \Delta ,f}$), we similarly remove the contributions of the down-set of $\beta$ ($\dot{\downarrow }\beta$ on lattice for $\mathcal{A}\left({\mathbf{M}}_{i+1}\right)$, see Equation 40c). The approach can be extended for tracing information flows over multiple steps, however, we will only trace one step in this example.

$$\begin{array}{cc}\hfill J_{\circ \to \circ ,f}(\alpha ,\beta ,T)& :=I_{\circ ,f}(\alpha \cup \beta ;T)\hfill \end{array}$$

(40a)

$$\begin{array}{cc}\hfill J_{\Delta \to \circ ,f}(\alpha ,\beta ,T)& :=J_{\circ \to \circ ,f}(\alpha ,\beta ,T)-\sum _{\gamma \in \dot{\downarrow }\alpha }{J}_{\Delta \to \circ ,f}(\gamma ,\beta ,T)\hfill \end{array}$$

(40b)

$$\begin{array}{cc}\hfill J_{\Delta \to \Delta ,f}(\alpha ,\beta ,T)& :=J_{\Delta \to \circ ,f}(\alpha ,\beta ,T)-\sum _{\gamma \in \dot{\downarrow }\beta }{J}_{\Delta \to \Delta ,f}(\alpha ,\gamma ,T)\hfill \end{array}$$

(40c)

Remark 11.

The resulting partial information flows are equivalent (dual) between the redundancy and loss measure except for the bottom element since their functionality differs: The flow from or to the bottom element on the redundancy lattice is always zero. In contrast, the flow from or to the bottom element on the synergy lattice quantifies the information gained or lost in the step.

Remark 12.

The information flow analysis of Rényi- and Bhattacharyya-information can be obtained as a transformation of the information flow from Hellinger-information. Alternatively, the information flow can be computed directly using Equation 40 under the corresponding definition of addition and subtraction for the used information measure.

Figure 7. Analysis of the Markov chain information flow (Equation A27). Visualized results for the information measures: KL, TV, and ${\chi }^2$. The remaining results (${\mathrm{H}}^2$-, LC-, and JS-information) can be found in Figure A3.

Figure 7. Analysis of the Markov chain information flow (Equation A27). Visualized results for the information measures: KL, TV, and ${\chi }^2$. The remaining results (${\mathrm{H}}^2$-, LC-, and JS-information) can be found in Figure A3.

We randomly generate an initial distribution and each row of a transition matrix under the constraint that at least one value shall be above 0.8 to avoid an information decay that is too rapid through the chain. The specific parameters of the example are shown in Appendix F. The used event spaces are $\mathcal{X}=\{0,1,2\}$ and $\mathcal{Y}=\{0,1\}$ such that $|{\mathcal{M}}_i|=6$. We construct a Markov chain of five steps with the target $T={\mathbf{M}}_3$ and trace each partial information for one step using Equation 40. We visualized the results for KL-, TV-, and ${\chi }^2$-information in Figure 7, and the results for ${\mathrm{H}}^2$-, LC- and JS-information in Figure A3 of Appendix F.

All results display the expected behavior that the information that ${\mathbf{M}}_i$ provides about ${\mathbf{M}}_3$ increases for $1\le i\le 3$ and decreases for $3\le i\le 5$. The information flow results of KL-, ${\mathrm{H}}^2$-, LC-, and JS-information are conceptually similar. Their main differences appear in the rate at which the information decays and, therefore, how much of the total information we can trace. In contrast, the results of TV- and ${\chi }^2$-information display different behavior, as shown in Figure 7: TV-information indicates significantly more redundancy, and ${\chi }^2$-information displays significantly more synergy than the other measures. Additionally, the decomposition of TV-information contains fever information flows. For example, it is the only analysis that does not show any information flow from ${\mathbf{M}}_2$ into the unique contribution of $Y_3$ or from ${\mathbf{M}}_2$ into the synergy of $(X_3,Y_3)$. This demonstrates that the same decomposition method can obtain different behaviors from different f-divergences.

5. Discussion

We perform the decomposition on a pointwise lattice using the zonogon join since it is possible to represent f-information as quantification of the zonogon perimeter on pointwise channels. Correspondingly, if we identified an information measure that was based on quantifying the zonogons of the pointwise channels by their area, then we would need to decompose it on a pointwise lattice using the zonogon meet to achieve non-negativity.

In the literature, PIDs have been defined based on different ordering relations [14]. This diversity is desirable since each approach provides a different operational interpretation of redundancy and synergy. For this reason, we wonder if obtaining a non-negative decomposition with inclusion-exclusion relation for other ordering relations was possible when transferring them to a pointwise perspective or from mutual information to other information measures.

We think studying the relations between different information measures for the same decomposition method may provide further insights into their properties, as demonstrated by the example of total variation in Section 4.2. Similarly, allowing the re-definition of addition for different information measures, as demonstrated in the example of Rényi-information in Section 3.3, opens new possibilities for satisfying the inclusion-exclusion relation and providing consistent decomposition results between related information measures.

Finally, there is currently no universally accepted definition of conditional Rényi information. Assuming that $I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_i\mid {\mathbf{S}}_j)$ should capture the information that ${\mathbf{S}}_i$ provides about T when already knowing the information from ${\mathbf{S}}_j$, then one could propose that this quantity should correspond to the according partial information contributions (unique/synergetic) and thus the definition of Equation 41.

$$I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_i\mid {\mathbf{S}}_j):=I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_i,{\mathbf{S}}_j)\ominus I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_j)$$

(41)

6. Conclusions

In this work, we demonstrated a non-negative PID in the framework of Williams and Beer [1] for any f-information with practical operational interpretation. We demonstrated that the decomposition of f-information can be used to obtain a non-negative decomposition of Rényi-information, for which we re-defined the addition to demonstrate that its results satisfy an inclusion-exclusion relation. Finally, we demonstrated how the proposed decomposition method can be used for tracing the flow of information through Markov chains and how the decomposition obtains different properties depending on the chosen information measure.

Appendix B.1. Properties of the Loss Measure on the Synergy Lattice

We require some of the following properties to hold for any set of sources $\alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$. Therefore, we define an equivalence relation from the ordering of the synergy lattice (≅) as shown in Equation A2.

Notation A1

(Equivalence under synergy-ordering). We use the notation $\alpha \cong \beta$ for the equivalence of two sets of sources $\alpha ,\beta \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ on the synergy lattice.

$$(\alpha \cong \beta )⟺(\alpha ⪯\beta \mathit{and}\beta ⪯\alpha )$$

(A2)

Lemma A4.

Any set of sources $\alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ is equivalent (≅) to some atom of the synergy lattice $\gamma \in \mathcal{A}\left(\mathbf{V}\right)$.

$$\forall \alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right).\exists \gamma \in \mathcal{A}\left(\mathbf{V}\right).\gamma \cong \alpha$$

The union for two sets of sources is equivalent to the meet of their corresponding atoms on the synergy lattice. Let $\alpha ,\beta \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ and $\gamma ,\delta \in \mathcal{A}\left(\mathbf{V}\right)$:

$$\begin{array}{c}\hfill \gamma \cong \alpha \mathit{and}\delta \cong \beta ⟹(\gamma \wedge \delta )\cong (\alpha \cup \beta )\end{array}$$

Proof. 

The used filter in the definition of an atom ($\mathcal{A}\left(\mathbf{V}\right)\subseteq \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$, Equation 8) only removes sets of cardinality $2\le \left|\alpha \right|$ and for any removed set of sources, we can construct an equivalent set which contains one less source by removing the subset ${\mathbf{S}}_a\subset {\mathbf{S}}_b$ as shown in Equation A3a. Therefore, all sets of sources $\alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ are equivalent to some atom $\gamma \in \mathcal{A}\left(\mathbf{V}\right)$ within the lattice (Equation A3b).

$$\begin{array}{ccc}\hfill {\mathbf{S}}_a\subset {\mathbf{S}}_b⟹\alpha & \cong (\alpha \setminus {\mathbf{S}}_a)\hfill & \hfill \mathrm{where}\text{:}{\mathbf{S}}_a,{\mathbf{S}}_b\in \alpha \end{array}$$

(A3a)

$$\begin{array}{cc}\hfill \forall \alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right),\exists \gamma \in \mathcal{A}\left(\mathbf{V}\right).\alpha & \cong \gamma \hfill \end{array}$$

(A3b)

The union of two sets of sources $\alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ is inferior to each individual set $\alpha$ and $\beta$:

$$\begin{array}{ccc}\hfill (\alpha \cup \beta )& ⪯\alpha \hfill & \hfill (\mathrm{by}\mathrm{Equation}10)\\ \hfill (\alpha \cup \beta )& ⪯\beta \hfill & \hfill (\mathrm{by}\mathrm{Equation}10)\end{array}$$

All sets of sources $\epsilon \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ that are inferior of both $\alpha$ and $\beta$ ($\epsilon ⪯\alpha$ and $\epsilon ⪯\beta$) are also inferior to their union.

$$\begin{array}{ccc}\hfill \epsilon ⪯\alpha \mathrm{and}\epsilon ⪯\beta & ⟹\epsilon ⪯(\alpha \cup \beta )\hfill & \hfill (\mathrm{by}\mathrm{Equation}10)\end{array}$$

Therefore, the union of $\alpha$ and $\beta$ is equivalent to the meet of their corresponding atoms on the synergy lattice. □

Proof of Lemma 1 from Section 3.2:

For any set of sources $\alpha ,\beta \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ and target variable T with state $t\in \mathcal{T}$, the function ${\kappa }_{\bigsqcup }$ (Equation 27) maintains the ordering from the synergy lattice under the Blackwell order.

$$\alpha ⪯\beta ⟹{\kappa }_{\bigsqcup }(\beta ,T,t)⊑{\kappa }_{\bigsqcup }(\alpha ,T,t)$$

(A4)

Proof. 

We consider two cases for $\beta$:

• If $\beta =\varnothing$, then the implication holds for any $\alpha$ since the bottom element ${\kappa }_{\bigsqcup }(\varnothing ,T,t)={\perp }_{\mathrm{BW}}$ is inferior (⊑) to any other channel.
• If $\beta \ne \varnothing$, then $\alpha$ is also a non-empty set since $\alpha ⪯\beta \prec {\top }_{\mathrm{SL}}=\varnothing$.

  $$\begin{array}{cccc}\hfill & \hfill & \hfill \alpha & ⪯\beta \hfill \\ \hfill \forall {\mathbf{S}}_b\in \beta ,\exists {\mathbf{S}}_a\in \alpha .& \hfill & \hfill {\mathbf{S}}_b& \subseteq {\mathbf{S}}_a\hfill & \hfill (\mathrm{by}\mathrm{Equation}10)\\ \hfill \forall {\mathbf{S}}_b\in \beta ,\exists {\mathbf{S}}_a\in \alpha .& \hfill & \hfill \kappa ({\mathbf{S}}_b,T,t)& ⊑\kappa ({\mathbf{S}}_a,T,t)\hfill & \hfill (\mathrm{by}\mathrm{Equation}2)\\ \hfill & \hfill & \hfill \underset{{\mathbf{S}}_b\in \beta }{⨆}\kappa ({\mathbf{S}}_b,T,t)& ⊑\underset{{\mathbf{S}}_a\in \alpha }{⨆}\kappa ({\mathbf{S}}_a,T,t)\hfill & \\ \hfill & \hfill & \hfill {\kappa }_{\bigsqcup }(\beta ,T,t)& ⊑{\kappa }_{\bigsqcup }(\alpha ,T,t)\hfill & \end{array}$$

Since the implication holds for both cases, the ordering is maintained. □

Corollary A2.

The defined cumulative loss measures ($i_{\cup ,f}$ of Equation 29a and $I_{\cup ,f}$ of Equation 30) maintain the ordering relation of the synergy lattice for any set of sources $\alpha ,\beta \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ and target variable T with state $t\in \mathcal{T}$:

$$\begin{array}{cc}\hfill \alpha ⪯\beta ⟹& i_{\cup ,f}(\alpha ,T,t)\le i_{\cup ,f}(\beta ,T,t)\hfill \\ \hfill \alpha ⪯\beta ⟹& I_{\cup ,f}(\alpha ;T)\le I_{\cup ,f}(\beta ;T)\hfill \end{array}$$

Proof. 

The pointwise monotonicity of the cumulative loss measure ($\alpha ⪯\beta ⟹i_{\cup ,f}(\alpha ,T,t)\le i_{\cup ,f}(\beta ,T,t)$) is obtained from Lemma 1 and A2 with Equation 29a. Sine all cumulative pointwise losses $i_{\cup ,f}$ are smaller for $\alpha$ than $\beta$, so will be their weighted sum ($\alpha ⪯\beta ⟹I_{\cup ,f}(\alpha ;T)\le I_{\cup ,f}(\beta ;T)$, see Equation 30). □

Corollary A3.

The cumulative pointwise loss of the meet from two atoms is equivalent to the cumulative pointwise loss of their union for any target variable T with state $t\in \mathcal{T}$:

$i_{\cup ,f}(\alpha \wedge \beta ,T,t)=i_{\cup ,f}(\alpha \cup \beta ,T,t)$.

Proof. 

The result follows from Lemma A4 and Corollary A2. □

Appendix B.2. Mapping Subsets of Even and Odd Cardinality

Let $\mathcal{P}\left(\mathbf{A}\right)$ represent the power-set of a non-empty set $\mathbf{A}\ne \varnothing$ and separate the subsets of even (${\mathcal{L}}_e$) and odd (${\mathcal{L}}_o$) cardinality as shown below. Additionally, let ${\mathcal{L}}_{\le 1}$ represent all subsets with cardinality less or equal to one and ${\mathcal{L}}_1$ all subsets of cardinality equal to one:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\le 1}& :=\{\mathbf{B}\in \mathcal{P}(\mathbf{A}):|\mathbf{B}|\le 1\}\hfill \\ \hfill {\mathcal{L}}_1& :=\{\mathbf{B}\in \mathcal{P}(\mathbf{A}):|\mathbf{B}|=1\}\hfill \\ \hfill {\mathcal{L}}_e& :=\{\mathbf{B}\in \mathcal{P}(\mathbf{A}):|\mathbf{B}\left|even\right\}\hfill \\ \hfill {\mathcal{L}}_o& :=\{\mathbf{B}\in \mathcal{P}(\mathbf{A}):|\mathbf{B}\left|odd\right\}\hfill \\ \hfill \mathcal{P}\left(\mathbf{A}\right)& ={\mathcal{L}}_e\cup {\mathcal{L}}_o\mathrm{and}\varnothing ={\mathcal{L}}_e\cap {\mathcal{L}}_o\hfill \end{array}$$

(A5)

The number of subsets with even cardinality is equal to the number of subsets with odd cardinality as shown in Equation A6.

$$|{\mathcal{L}}_e|=\sum _{i=0}^{⌊\frac{\left|\mathbf{A}\right|}{2}⌋}\left(\genfrac{}{}{0pt}{}{\left|\mathbf{A}\right|}{2i}\right)=2^{\left|\mathbf{A}\right|-1}=\sum _{i=0}^{⌊\frac{\left|\mathbf{A}\right|}{2}⌋}\left(\genfrac{}{}{0pt}{}{\left|\mathbf{A}\right|}{2i+1}\right)=\left|{\mathcal{L}}_o\right|$$

(A6)

Consider a function $g_e:{\mathcal{L}}_e\to {\mathcal{L}}_{\le 1}$, which takes an even subset $\mathbf{E}\in {\mathcal{L}}_e$ and returns a subset of cardinality $|g_e\left(\mathbf{E}\right)|=min(\left|\mathbf{E}\right|,1)$ according to Equation A7.

$$\forall \mathbf{E}\in {\mathbf{S}}_e:\left\{\begin{array}{cc}{g}_e\left(\mathbf{E}\right)=\varnothing \hfill & \mathrm{if}\mathbf{E}=\varnothing \hfill \\ g_e\left(\mathbf{E}\right)=\left\{e\right\}\mathrm{s}.\mathrm{t}.e\in \mathbf{E}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(A7)

Lemma A5.

For any function $g_e\in {\mathcal{G}}_e$, there exists a function $G:({\mathcal{L}}_e,{\mathcal{G}}_e)\to {\mathcal{L}}_o$ which satisfies the following two properties:

a)

For any subset with even cardinality, the function $g_e(·)$ returns a subset of function $G(·)$:

$$\forall g_e\in {\mathcal{G}}_e,\mathbf{E}\in {\mathcal{L}}_e:g_e\left(\mathbf{E}\right)\subseteq G(\mathbf{E},g_e).$$

(A8)

b)

The function $G(·)$ which satisfies Equation A8 has an inverse on its first argument $G^{-1}:({\mathcal{L}}_o,{\mathcal{G}}_e)\to {\mathcal{L}}_e$.

$$\forall g_e\in {\mathcal{G}}_e,\mathbf{E}\in {\mathcal{L}}_e,\exists G^{-1}:G^{-1}(G(\mathbf{E},g_e),g_e)=\mathbf{E}.$$

(A9)

Figure A1. Intuition for the definition of Equation A11. We can divide the set $\mathcal{P}\left({\mathbf{A}}_{n+1}\right)$ into $\mathcal{P}\left({\mathbf{A}}_n\right)$ and $\{\mathbf{B}\cup \left\{q\right\}:\mathbf{B}\in \mathcal{P}\left({\mathbf{A}}_n\right)\}$. The definition of function $G_{n+1}$ mirrors $G_n$ if $g_e(\mathbf{O}\cup \left\{q\right\})=\left\{q\right\}$ (blue) and otherwise breaks its mapping (orange).

Figure A1. Intuition for the definition of Equation A11. We can divide the set $\mathcal{P}\left({\mathbf{A}}_{n+1}\right)$ into $\mathcal{P}\left({\mathbf{A}}_n\right)$ and $\{\mathbf{B}\cup \left\{q\right\}:\mathbf{B}\in \mathcal{P}\left({\mathbf{A}}_n\right)\}$. The definition of function $G_{n+1}$ mirrors $G_n$ if $g_e(\mathbf{O}\cup \left\{q\right\})=\left\{q\right\}$ (blue) and otherwise breaks its mapping (orange).

Proof. 

We construct a function G for an arbitrary $g_e$ and demonstrate that it satisfies both properties (Equation A8 and A9) by induction on the cardinality of $\mathbf{A}$. We indicate the cardinality of $\mathbf{A}$ with $n=\left|\mathbf{A}\right|$ as subscript ${\mathbf{A}}_n$, ${\mathcal{L}}_{e,n}$, ${\mathcal{L}}_{o,n}$ and $G_n$:

• At the base case ${\mathbf{A}}_1=\left\{a\right\}$, the sets of subsets are ${\mathcal{L}}_{e,1}=\{\varnothing \}$ and ${\mathcal{L}}_{o,1}=\left\{\left\{a\right\}\right\}$. We define the function $G_1(\varnothing ,g_e)\left\{a\right\}$ for any $g_e$ to satisfy both required properties:

  a)

  The constraints of Equation A7 ensures that $g_e(\varnothing )=\varnothing$. Since the empty set is the only element in ${\mathbf{S}}_{e,1}$, the subset relation (requirement of Equation A8) is satisfied $g_e(\varnothing )=\varnothing \subseteq \left\{a\right\}=G_1(\varnothing ,g_e)$.

  b)

  The function $G_1:({\mathcal{L}}_{e,1},{\mathcal{G}}_e)\to {\mathcal{L}}_{o,1}$ is a bijection from ${\mathcal{L}}_{e,1}$ to ${\mathcal{L}}_{o,1}$ and therefore has an inverse on its first argument $G_1^{-1}:({\mathcal{L}}_{o,1},{\mathcal{G}}_e)\to {\mathcal{L}}_{e,1}$ (requirement of Equation A9).

• Assume there exists a function $G_n$, which satisfies both required properties (Equation A8 and A9) at sets of cardinality $1\le n=|{\mathbf{A}}_n|$.
• For the induction step, we show the definition of a function $G_{n+1}$ that satisfies both required properties. For sets ${\mathbf{A}}_{n+1}={\mathbf{A}}_n\cup \left\{q\right\}$, the subsets of even and odd cardinality can be expanded as shown in Equation A10.

  $$\begin{array}{cc}\hfill {\mathcal{L}}_{e,n+1}& ={\mathcal{L}}_{e,n}\cup \left\{\mathbf{O}\cup \left\{q\right\}:\mathbf{O}\in {\mathcal{L}}_{o,n}\right\},\hfill \\ \hfill {\mathcal{L}}_{o,n+1}& ={\mathcal{L}}_{o,n}\cup \left\{\mathbf{E}\cup \left\{q\right\}:\mathbf{E}\in {\mathcal{L}}_{e,n}\right\}.\hfill \end{array}$$

  (A10)
  We define $G_{n+1}$ for $\mathbf{E}\in {\mathcal{L}}_{e,n}$ and $\mathbf{O}\in {\mathcal{L}}_{o,n}$ at any $g_e$ as shown in Equation A11 using the function $G_n$ and its inverse $G_n^{-1}$ from the induction hypothesis. The function $G_{n+1}$ is defined for any subset in ${\mathcal{L}}_{e,n+1}$ as it can be seen from Equation A10.

  $$\begin{array}{cc}\hfill G_{n+1}(\mathbf{E},g_e):=& \left\{\begin{array}{cc}\mathbf{E}\cup \left\{q\right\}\hfill & \mathrm{if}{g}_e(G_n(\mathbf{E},g_e)\cup \left\{q\right\})\ne \left\{q\right\}\hfill \\ G_n(\mathbf{E},g_e)\hfill & \mathrm{if}{g}_e(G_n(\mathbf{E},g_e)\cup \left\{q\right\})=\left\{q\right\}\hfill \end{array}\right.\hfill \\ \hfill G_{n+1}(\mathbf{O}\cup \left\{q\right\},g_e):=& \left\{\begin{array}{cc}\mathbf{O}\hfill & \mathrm{if}{g}_e(\mathbf{O}\cup \left\{q\right\})\ne \left\{q\right\}\hfill \\ G_n^{-1}(\mathbf{O},g_e)\cup \left\{q\right\}\hfill & \mathrm{if}{g}_e(\mathbf{O}\cup \left\{q\right\})=\left\{q\right\}\hfill \end{array}\right.\hfill \end{array}$$

  (A11)
  Figure A1 provides an intuition for the definition of $G_{n+1}$: the outcome of $g_e(\mathbf{O}\cup \left\{q\right\})$ determines, if the function $G_{n+1}$ maintains or breaks the mapping of $G_n$.
  The function F as defined in Equation A11 satisfies both requirements (Equation A8 and A9) for any $g_e$:

  a)

  To demonstrate that the function satisfies the subset relation of Equation A8, we analyze the four cases for the return value of $G_{n+1}$ as defined in Equation A11 individually:

  • $g_e\left(\mathbf{E}\right)\subseteq \mathbf{E}\cup \left\{q\right\}$ holds, since the function $g_e$ always returns a subset of its input (Equation A7).
  • $g_e\left(\mathbf{E}\right)\subseteq G_n(\mathbf{E},g_e)$ holds by the induction hypothesis.
  • if $g_e(\mathbf{O}\cup \left\{q\right\})\ne \left\{q\right\}$ then $g_e(\mathbf{O}\cup \left\{q\right\})\subseteq \mathbf{O}$: Since the input to function $g_e$ is not the empty set, the function $g_e(\mathbf{O}\cup \left\{q\right\})$ returns a singleton subset of its input (Equation A7). If the element in the singleton subset is unequal to q, then it is a subset of $\mathbf{O}$.
  • if $g_e(\mathbf{O}\cup \left\{q\right\})=\left\{q\right\}$ then $g_e(\mathbf{O}\cup \left\{q\right\})\subseteq \left\{q\right\}\cup G_n^{-1}(\mathbf{O},g_e)$ holds trivially.

  b)

  To demonstrate that the function $G_{n+1}$ has an inverse (Equation A9), we show that the function $G_{n+1}$ is a bijection from ${\mathcal{L}}_{e,n+1}$ to ${\mathcal{L}}_{o,n+1}$. Since the function $G_{n+1}$ is defined for all elements in ${\mathcal{L}}_{e,n+1}$ and both sets have the same cardinality ($|{\mathcal{L}}_{e,n+1}|=|{\mathcal{L}}_{o,n+1}|$, Equation A6), it is sufficient to show that the function $G_{n+1}$ is distinct for all inputs.

  The return value of $G_{n+1}$ has four cases, two of which return a set containing q (case 1 and 4 in Equation A11), while the two others do not (case 2 and 3 in Equation A11). Therefore, we have to show that both of these cases cannot coincide for any input:

  -

  Case 2 and 3 in Equation A11: If the return value of both cases was equal, then $\mathbf{O}=G_n(\mathbf{E},g_e)$ and therefore $g_e(\mathbf{O}\cup \left\{q\right\})=g_e(G_n(\mathbf{E},g_e)\cup \left\{q\right\})$. This leads to a contradiction, since the condition of case 3 ensures $g_e(\mathbf{O}\cup \left\{q\right\})\ne \left\{q\right\}$, while the condition of case 2 ensures $g_e(G_n(\mathbf{E},g_e)\cup \left\{q\right\})=\left\{q\right\}$. Hence, the return values of case 2 and 3 are distinct.

  -

  Case 1 and 4 in Equation A11: If the return value of both cases was equal, then $\mathbf{E}=G_n^{-1}(\mathbf{O},g_e)$ and therefore $g_e(\mathbf{O}\cup \left\{q\right\})=g_e(G_n(\mathbf{E},g_e)\cup \left\{q\right\})$. This leads to a contradiction, since the condition of case 4 ensures $g_e(\mathbf{O}\cup \left\{q\right\})=\left\{q\right\}$, while the condition of case 1 ensures $g_e(G_n(\mathbf{E},g_e)\cup \left\{q\right\})\ne \left\{q\right\}$. Hence, the return values of case 1 and 4 are distinct.

  Since the function $G_{n+1}$ is a bijection, there exists an inverse $G_{n+1}^{-1}$.

□

Appendix B.3. The non-negativity of the decomposition

Lemma A6.

Consider a Non-Empty Set of of Binary Input Channel $\mathbf{A}\ne \varnothing$ and $0\le p\le 1$. Quantifying an inclusion-exclusion principle on the pointwise information of their Blackwell join is larger than the pointwise information of their Blackwell meet as shown in Equation A12.

$$i_f\left(p,\underset{\kappa \in \mathbf{A}}{\sqcap }\kappa \right)\le \sum _{\varnothing \ne \mathbf{B}\subseteq \mathbf{A}}{(-1)}^{\left|\mathbf{B}\right|-1}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)$$

(A12)

Proof. 

Consider a function $g_o:{\mathcal{L}}_o\to {\mathcal{L}}_1$, where $g_o\left(\mathbf{O}\right)\subseteq \mathbf{O}$ such that the function returns a singleton subset for a set of odd cardinality. Equation A13 can be obtained from the constraints on $g_e$ (Equation A7) and Lemma A5.

$$\forall g_e\in {\mathcal{G}}_e,\mathbf{E}\in {\mathcal{L}}_e,\exists g_o\in {\mathcal{G}}_o,G:\left\{\begin{array}{cc}{g}_e(\varnothing )\subseteq g_o\left(G(\varnothing )\right)\hfill & \mathrm{if}\mathbf{E}=\varnothing \hfill \\ g_e\left(\mathbf{E}\right)=g_o\left(G\left(\mathbf{E}\right)\right)\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(A13)

Equation A14a holds since we can replace $g_e(\varnothing )$ with $g_o\left(G(\varnothing )\right)$, meaning there exists a $\kappa \in \mathbf{A}$ for creating a (Minkowski) sum over the same set of channel zonogons on both sides of the quality. Equation A14b holds since Lemma A5 ensured that the existing function G is a bijection. Equation A14c holds since the intersection is a subset of each individual zonogon.

$$\begin{array}{cc}\hfill \forall g_e\in {\mathcal{G}}_e,\exists g_o\in {\mathcal{G}}_o,\kappa \in \mathbf{A},G& :Z\left(\kappa \right)+\sum _{\mathbf{E}\in {\mathcal{L}}_e\setminus \varnothing }Z\left(g_e\left(\mathbf{E}\right)\right)=\sum _{\mathbf{E}\in {\mathcal{L}}_e}Z\left(g_o\left(G\left(\mathbf{E}\right)\right)\right)\hfill \end{array}$$

(A14a)

$$\begin{array}{cc}\hfill \forall g_e\in {\mathcal{G}}_e,\exists g_o\in {\mathcal{G}}_o,\kappa \in \mathbf{A}& :Z\left(\kappa \right)+\sum _{\mathbf{E}\in {\mathcal{L}}_e\setminus \varnothing }Z\left(g_e\left(\mathbf{E}\right)\right)=\sum _{\mathbf{O}\in {\mathcal{L}}_o}Z\left(g_o\left(\mathbf{O}\right)\right)\hfill \end{array}$$

(A14b)

$$\begin{array}{cc}\hfill \forall g_e\in {\mathcal{G}}_e,\exists g_o\in {\mathcal{G}}_o& :\bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)+\sum _{\mathbf{E}\in {\mathcal{L}}_e\setminus \varnothing }Z\left(g_e\left(\mathbf{E}\right)\right)\subseteq \sum _{\mathbf{O}\in {\mathcal{L}}_o}Z\left(g_o\left(\mathbf{O}\right)\right)\hfill \end{array}$$

(A14c)

Equation A14c is parameterized by $g_e$ and subsets are closed under set union. Therefore, we can combine all choices for $g_e$ and $g_o$ using the set-theoretic union as shown below. For the notation, let $m=2^{\left|\mathbf{A}\right|-1}$ and we indicate subsets of $\mathbf{A}$ with even cardinality as ${\mathbf{E}}_i\in {\mathcal{L}}_e$, where $1\le i\le m$. We use the last index for the empty set ${\mathbf{E}}_m=\varnothing$. The subsets of $\mathbf{A}$ with odd cardinality are correspondingly noted as ${\mathbf{O}}_i\in {\mathcal{L}}_o$. For clarity, we note binary input channels from an even subset as $\lambda \in \mathbf{E}$ and binary input channels from an odd subset as $\nu \in \mathbf{O}$.

$$\begin{array}{ccc}\hfill \bigcup _{\begin{array}{c}{\lambda }_1\in {\mathbf{E}}_1\\ {\lambda }_2\in {\mathbf{E}}_2\\ \cdots \\ {\lambda }_{m-1}\in {\mathbf{E}}_{m-1}\end{array}}\left(\bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)+\sum _{i=1}^{m-1}Z\left({\lambda }_i\right)\right)& \subseteq \bigcup _{\begin{array}{c}{\nu }_1\in {\mathbf{O}}_1\\ {\nu }_2\in {\mathbf{O}}_2\\ \cdots \\ {\nu }_m\in {\mathbf{O}}_m\end{array}}\left(\sum _{j=1}^{m}Z\left({\nu }_j\right)\right)\hfill & \\ \hfill \bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)+\sum _{i=1}^{m-1}\bigcup _{\lambda \in {\mathbf{E}}_i}Z\left(\lambda \right)& \subseteq \sum _{j=1}^m\bigcup _{\nu \in {\mathbf{O}}_j}Z\left(\nu \right)\hfill & \hfill \left(\begin{array}{c}\mathrm{Minkowski}\mathrm{sum}\mathrm{dis}-\\ \mathrm{tributes}\mathrm{over}\mathrm{set}\mathrm{union}\end{array}\right)\\ \hfill Conv\left(\bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)+\sum _{i=1}^{m-1}\bigcup _{\lambda \in {\mathbf{E}}_i}Z\left(\lambda \right)\right)& \subseteq Conv\left(\sum _{j=1}^m\bigcup _{\nu \in {\mathbf{O}}_j}Z\left(\nu \right)\right)\hfill & \hfill \left(\begin{array}{c}\mathrm{if}\mathbf{X}\subseteq \mathbf{Y}\mathrm{then}\\ Conv\left(\mathbf{X}\right)\subseteq Conv\left(\mathbf{Y}\right)\end{array}\right)\\ \hfill \bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)+\sum _{i=1}^{m-1}Conv\left(\bigcup _{\lambda \in {\mathbf{E}}_i}Z\left(\lambda \right)\right)& \subseteq \sum _{j=1}^{m}Conv\left(\bigcup _{\nu \in {\mathbf{O}}_j}Z\left(\nu \right)\right)\hfill & \hfill \left(\begin{array}{c}\mathrm{Convex}\mathrm{hull}\mathrm{distributes}\\ \mathrm{over}\mathrm{Minkowski}\mathrm{sum}\end{array}\right)& \\ \hfill Z\left({\sqcap }_{\kappa \in \mathbf{A}}\kappa \right)+\sum _{i=1}^{m-1}Z\left(\underset{\lambda \in {\mathbf{E}}_i}{⨆}\lambda \right)& \subseteq \sum _{j=1}^{m}Z\left(\underset{\nu \in {\mathbf{O}}_j}{⨆}\nu \right)\hfill & \hfill \left(\mathrm{by}\mathrm{Equation}7\right)\\ \hfill i_f\left(p,{\sqcap }_{\kappa \in \mathbf{A}}\kappa \right)+\sum _{i=1}^{m-1}{i}_f\left(p,\underset{\lambda \in {\mathbf{E}}_i}{⨆}\lambda \right)& \le \sum _{j=1}^m{i}_f\left(p,\underset{\nu \in {\mathbf{O}}_j}{⨆}\nu \right)\hfill & \hfill \left(\mathrm{by}\mathrm{Lemma}\mathrm{A}3\right)\\ \hfill i_f\left(p,{\sqcap }_{\kappa \in \mathbf{A}}\kappa \right)+\sum _{\stackrel{\varnothing \ne \mathbf{B}\subseteq \mathbf{A}}{\left|\mathbf{B}\right|even}}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)& \le \sum _{\stackrel{\varnothing \ne \mathbf{B}\subseteq \mathbf{A}}{\left|\mathbf{B}\right|odd}}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)\hfill & \hfill (\mathrm{replace}\mathrm{notation})\\ \hfill i_f\left(p,{\sqcap }_{\kappa \in \mathbf{A}}\kappa \right)\le \sum _{\varnothing \ne \mathbf{B}\subseteq \mathbf{A}}& {(-1)}^{\left|\mathbf{B}\right|-1}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)\hfill \end{array}$$

□

Lemma A7.

The decomposition of f-information is non-negative on the pointwise and combined synergy lattice for any target variable T with state $t\in \mathcal{T}$:

$$\begin{array}{cc}\hfill \forall \alpha \in \mathcal{A}\left(\mathbf{V}\right).& 0\le \Delta i_{\cup ,f}(\alpha ,T,t),\hfill \\ \hfill \forall \alpha \in \mathcal{A}\left(\mathbf{V}\right).& 0\le \Delta I_{\cup ,f}(\alpha ;T).\hfill \end{array}$$

Proof. 

We show the non-negativity of pointwise partial information ($\Delta i_{\cup ,f}(\alpha ,T,t)$) in two cases . We write ${\alpha }^{-}$ to represent the cover set of $\alpha$ on the synergy lattice and use $p=P_T\left(t\right)$ as abbreviation:

• Let $\alpha ={\perp }_{\mathrm{SL}}=\left\{\mathbf{V}\right\}$. The bottom element of the synergy lattice is quantified to zero (by Equation 29a, $i_{\cup ,f}({\perp }_{\mathrm{SL}},T,t)=0$) and therefore also its partial contribution will be zero ($\Delta i_{\cup ,f}({\perp }_{\mathrm{SL}},T,t)=0$), which implies Equation A15.

  $$\alpha ={\perp }_{\mathrm{SL}}⟹0\le \Delta i_{\cup ,f}(\alpha ,T,t)$$

  (A15)
• Let $\alpha \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathrm{SL}}\right\}$, then its cover set is non-empty (${\alpha }^{-}\ne \varnothing$). Additionally, we know that no atom in the cover set is the empty set ($\forall \beta \in {\alpha }^{-}.\beta \ne \varnothing$), since the empty atom is the top element (${\top }_{\mathrm{SL}}=\varnothing$).
  Since it will be required later, note that the inclusion-exclusion principle of a constant is the constant itself as shown in Equation A16 since without the empty set there exists one more subset of odd cardinality than with even cardinality (see Equation A6).

  $$i_f(p,\kappa (\mathbf{V},T,t))=\sum _{\varnothing \ne \mathbf{B}\subseteq {\alpha }^{-}}{(-1)}^{\left|\mathbf{B}\right|-1}{i}_f(p,\kappa (\mathbf{V},T,t))$$

  (A16)
  We can re-write the Möbious inverse as shown in Equation A17.

  $$\begin{array}{ccc}\hfill \Delta i_{\cup ,f}(\alpha ,T,t)=& i_{\cup ,f}(\alpha ,T,t)-\sum _{\beta \in \dot{\downarrow }\alpha }\Delta i_{\cup ,f}(\beta ,T,t)\hfill & \hfill (\mathrm{by}\mathrm{Equation}29\mathrm{b})\end{array}$$

  (A17a)

  $$\begin{array}{ccc}\hfill =& i_{\cup ,f}(\alpha ,T,t)-\sum _{\varnothing \ne \mathbf{B}\subseteq {\alpha }^{-}}{(-1)}^{\left|\mathbf{B}\right|-1}·i_{\cup ,f}\left(\underset{\beta \in \mathbf{B}}{\bigwedge }\beta ,T,t\right)\hfill & \hfill (\mathrm{by}\left(\right[21,\mathrm{p}.15\left]\right)\end{array}$$

  (A17b)

  $$\begin{array}{ccc}\hfill =& i_{\cup ,f}(\alpha ,T,t)-\sum _{\varnothing \ne \mathbf{B}\subseteq {\alpha }^{-}}{(-1)}^{\left|\mathbf{B}\right|-1}·i_{\cup ,f}\left(\bigcup _{\beta \in \mathbf{B}}\beta ,T,t\right)\hfill & \hfill (\mathrm{by}\mathrm{Corollary}\mathrm{A}3)\end{array}$$

  (A17c)

  $$\begin{array}{ccc}\hfill =& -i_f(p,{\kappa }_{\bigsqcup }(\alpha ,T,t))+\sum _{\varnothing \ne \mathbf{B}\subseteq {\alpha }^{-}}{(-1)}^{\left|\mathbf{B}\right|-1}·i_f(p,{\kappa }_{\bigsqcup }(\bigcup _{\beta \in \mathbf{B}}\beta ,T,t))\hfill & \hfill (\mathrm{by}\mathrm{Eq}.29\mathrm{a},\mathrm{A}16)\end{array}$$

  (A17d)

  $$\begin{array}{ccc}\hfill =& -i_f(p,{\kappa }_{\bigsqcup }(\alpha ,T,t))+\sum _{\varnothing \ne \mathbf{B}\subseteq {\alpha }^{-}}{(-1)}^{\left|\mathbf{B}\right|-1}·i_f(p,\underset{\mathbf{S}\in \left({\bigcup }_{\beta \in \mathbf{B}}\beta \right)}{⨆}\kappa (\mathbf{S},T,t))\hfill & \hfill (\mathrm{by}\forall \beta \in {\alpha }^{-}.\beta \ne \varnothing )\end{array}$$

  (A17e)

  $$\begin{array}{cc}\hfill =& -i_f(p,{\kappa }_{\bigsqcup }(\alpha ,T,t))+\sum _{\varnothing \ne \mathbf{B}\subseteq {\alpha }^{-}}{(-1)}^{\left|\mathbf{B}\right|-1}·i_f(p,\underset{\beta \in \mathbf{B}}{⨆}\underset{\mathbf{S}\in \beta }{⨆}\kappa (\mathbf{S},T,t))\hfill \end{array}$$

  (A17f)

  $$\begin{array}{cc}\hfill =& -i_f(p,{\kappa }_{\bigsqcup }(\alpha ,T,t))+\sum _{\varnothing \ne \mathbf{B}\subseteq \{{\kappa }_{\bigsqcup }(\beta ,T,t):\beta \in {\alpha }^{-}\}}{(-1)}^{\left|\mathbf{B}\right|-1}·i_f(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa )\hfill \end{array}$$

  (A17g)
  Consider the non-empty set of channels $\mathbf{D}=\{{\kappa }_{\bigsqcup }(\beta ,T,t):\beta \in {\alpha }^{-}\}$, then we obtain Equation A18b from Lemma A6.

  $$\begin{array}{cc}\hfill i_f\left(p,{\sqcap }_{\kappa \in \{{\kappa }_{\bigsqcup }(\beta ,T,t):\beta \in {\alpha }^{-}\}}\kappa \right)& \le \sum _{\varnothing \ne \mathbf{B}\subseteq \{{\kappa }_{\bigsqcup }(\beta ,T,t):\beta \in {\alpha }^{-}\}}{(-1)}^{\left|\mathbf{B}\right|-1}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)\hfill \end{array}$$

  (A18a)

  $$\begin{array}{cc}\hfill i_f\left(p,{\sqcap }_{\beta \in {\alpha }^{-}}{\kappa }_{\bigsqcup }(\beta ,T,t)\right)& \le \sum _{\varnothing \ne \mathbf{B}\subseteq \{{\kappa }_{\bigsqcup }(\beta ,T,t):\beta \in {\alpha }^{-}\}}{(-1)}^{\left|\mathbf{B}\right|-1}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)\hfill \end{array}$$

  (A18b)
  We can construct an upper bound on $i_f(p,{\kappa }_{\bigsqcup }(\alpha ,T,t))$ based on the cover set ${\alpha }^{-}$ as shown in Equation A19.

  $$\begin{array}{cccc}\hfill & \forall \beta \in {\alpha }^{-}.\hfill & \hfill \beta & ⪯\alpha \hfill \end{array}$$

  (A19a)

  $$\begin{array}{ccccc}\hfill & \forall \beta \in {\alpha }^{-}.\hfill & \hfill {\kappa }_{\bigsqcup }(\alpha ,T,t)& ⊑{\kappa }_{\bigsqcup }(\beta ,T,t)\hfill & \hfill (\mathrm{by}\mathrm{Lemma}1)\end{array}$$

  (A19b)

  $$\begin{array}{cccc}\hfill & \hfill & \hfill {\kappa }_{\bigsqcup }(\alpha ,T,t)& ⊑\underset{\beta \in {\alpha }^{-}}{\sqcap }{\kappa }_{\bigsqcup }(\beta ,T,t)\hfill \end{array}$$

  (A19c)

  $$\begin{array}{ccccc}\hfill & \hfill & \hfill i_f\left(p,{\kappa }_{\bigsqcup }(\alpha ,T,t)\right)& \le i_f\left(p,\underset{\beta \in {\alpha }^{-}}{\sqcap }{\kappa }_{\bigsqcup }(\beta ,T,t)\right)\hfill & \hfill (\mathrm{by}\mathrm{Lemma}\mathrm{A}2)\end{array}$$

  (A19d)
  By transitivity of Equation A18b and A19d, we obtain Equation A20.

  $$i_f(p,{\kappa }_{\bigsqcup }(\alpha ,T,t))\le \sum _{\varnothing \ne \mathbf{B}\subseteq \{{\kappa }_{\bigsqcup }(\beta ,T,t):\beta \in {\alpha }^{-}\}}{(-1)}^{\left|\mathbf{B}\right|-1}{i}_f\left(p,\underset{\kappa \in \mathbf{B}}{⨆}\kappa \right)$$

  (A20)
  By Equation A17 and A20, we obtain the non-negativity of pointwise partial information as shown in Equation A21.

  $$\alpha \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathrm{SL}}\right\}.0\le \Delta i_{\cup ,f}(\alpha ,T,t)$$

  (A21)

From Equation A15 and A21, we obtain that pointwise partial information is non-negative for all atoms of the lattice:

$$\forall \alpha \in \mathcal{A}\left(\mathbf{V}\right).0\le \Delta i_{\cup ,f}(\alpha ,T)$$

(A22)

If all pointwise partial components are non-negative, then their expected value will also be non-negative (see Equation 31):

$$\forall \alpha \in \mathcal{A}\left(\mathbf{V}\right).0\le \Delta I_{\cup ,f}(\alpha ;T)$$

(A23)

□