Variations on the Expectation Due to Changes in the Probability Measure

1. Introduction

Let m be a positive integer and denote by $▵\left({\mathbb{R}}^m\right)$ the set of all probability measures on the measurable space $\left({\mathbb{R}}^m,\mathcal{B}{\mathbb{R}}^m\right)$, with $\mathcal{B}{\mathbb{R}}^m$ being the Borel $\sigma$-algebra on ${\mathbb{R}}^m$. Given a Borel measurable function $h:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$, consider the functional

$$\begin{array}{c}\hfill \{\begin{array}{cc}\hfill {\mathsf{G}}_h:{\mathbb{R}}^n\times ▵\left({\mathbb{R}}^m\right)\times ▵\left({\mathbb{R}}^m\right)& ⟶\mathbb{R}\hfill \\ \hfill (x,P_1,P_2)& ⟼\int h(x,y)\mathrm{d}{P}_1\left(y\right)-\int h(x,y)\mathrm{d}{P}_2\left(y\right),\hfill \end{array}.\end{array}$$

(1)

which quantifies the variation of the expectation of the measurable function h due to changing the probability measure from $P_2$ to $P_1$. These variations are often referred to as probability distribution drifts, in some application areas, see for instance [1,2] and [3]. The functional ${\mathsf{G}}_h$ is defined when both integrals exist and are finite.

In order to define the expectation of ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ when x is obtained by sampling a probability measure in $▵\left({\mathbb{R}}^n\right)$, the structure formalized below is required.

Definition 1. 

A family $P_{Y|X}\triangleq {\left(P_{Y|X=x}\right)}_{x\in {\mathbb{R}}^n}$ of elements of $▵\left({\mathbb{R}}^m\right)$ indexed by ${\mathbb{R}}^n$ is said to be a conditional probability measure if, for all sets $\mathcal{A}\in \mathcal{B}{\mathbb{R}}^m$, the map

$$\begin{array}{c}\hfill \{\begin{array}{cc}\hfill {\mathbb{R}}^n& ⟶[0,1]\hfill \\ \hfill x& ⟼P_{Y|X=x}\left(\mathcal{A}\right)\hfill \end{array}.\end{array}$$

is Borel measurable. The set of all such conditional probability measures is denoted by $▵\left({\mathbb{R}}^m|{\mathbb{R}}^n\right)$.

In this setting, consider the functional

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h:\{\begin{array}{cc}\hfill ▵\left({\mathbb{R}}^m|{\mathbb{R}}^n\right)\times ▵\left({\mathbb{R}}^m|{\mathbb{R}}^n\right)\times ▵\left({\mathbb{R}}^n\right)& ⟶\mathbb{R}\hfill \\ \hfill \left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)& ⟼\int {\mathsf{G}}_h\left(x,P_{Y|X=x}^{\left(1\right)},P_{Y|X=x}^{\left(2\right)}\right)\mathrm{d}{P}_X\left(x\right)\hfill \end{array}.\end{array}$$

(2)

This quantity can be interpreted as the variation of the integral (expectation) of the function h when the probability measure changes from the joint probability measure $P_{Y|X}^{\left(1\right)}{P}_X$ to another joint probability measure $P_{Y|X}^{\left(2\right)}{P}_X$, both in $▵\left({\mathbb{R}}^m\times {\mathbb{R}}^n\right)$. This follows from (2) by observing that

$$\begin{array}{ccc}\hfill {\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)& =\hfill & \hfill \int h(x,y)\mathrm{d}{P}_{Y|X}^{\left(1\right)}{P}_X(y,x)-\int h(x,y)\mathrm{d}{P}_{Y|X}^{\left(2\right)}{P}_X(y,x).\end{array}$$

(3)

Special attention is given to the quantity ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$, for some $P_{Y|X}\in ▵\left({\mathbb{R}}^m|{\mathbb{R}}^n\right)$, with $P_Y$ being the marginal of the joint probability measure $P_{Y|X}·P_X$. That is, for all sets $\mathcal{A}\in \mathcal{B}{\mathbb{R}}^m$,

$$\begin{array}{ccc}\hfill P_Y\left(\mathcal{A}\right)& =\hfill & \hfill \int P_{Y|X=x}\left(\mathcal{A}\right)\mathrm{d}{P}_X\left(x\right).\end{array}$$

(4)

Its relevance stems from the fact that it captures the variation of the expectation of the function h when the probability measure changes from the joint probability measure $P_{Y|X}{P}_X$ to the product of its marginals $P_Y{P}_X$. That is,

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)=\int \left(\int h(x,y)\mathrm{d}{P}_Y\left(y\right)-\int h(x,y)\mathrm{d}{P}_{Y|X=x}\left(y\right)\right)\mathrm{d}{P}_X\left(x\right)\end{array}$$

(5)

$$\begin{array}{c}\hfill =\int h(x,y)\mathrm{d}{P}_Y{P}_X(y,x)-\int h(x,y)\mathrm{d}{P}_{Y|X}{P}_X(y,x).\end{array}$$

(6)

1.1. Novelty and Contributions

This work makes two key contributions: First, it provides a closed-form expression for the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1) for a fixed $x\in {\mathbb{R}}^n$ and two arbitrary probability measures $P_1$ and $P_2$, formulated explicitly in terms of relative entropies. Second, it derives a closed-form expression for the expected variation ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ in (2), again in terms of information measures, for arbitrary conditional probability measures $P_{Y|X}^{\left(1\right)}$, $P_{Y|X}^{\left(2\right)}$, and an arbitrary probability measure $P_X$.

A further contribution of this work is the derivation of specific closed-form expressions for ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6), which reveal deep connections with both mutual information [4,5] and lautum information [6]. Notably, when $P_{Y|X}$ is a Gibbs conditional probability measure, this variation simplifies (up to a constant factor) to the sum of the mutual and lautum information induced by the joint distribution $P_{Y|X}{P}_X$.

Although these results were originally discovered in the analysis of generalization error of machine learning algorithms, see for instance [7,8,9,10,11], where the function h in (1) was assumed to represent an empirical risk, this paper presents such results in a comprehensive and general setting that is no longer tied to such assumptions. Also, strong connections with information projections and Pythagorean identities [12,13] are discussed. This new general presentation not only unifies previously scattered insights but also makes the results applicable across a broad range of domains in which changes in the expectation due to variations of the underlying probability measures are relevant.

1.2. Applications

The study of the variation of the integral (expectation) of h (for some fixed $x\in {\mathbb{R}}^n$) due to a measure change from $P_2$ to $P_1$, i.e., the value ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1), plays a central role in the definition of integral probability metrics (IPMs)[14,15]. Using the notation in (1), an IPM results from the optimization problem

$$\begin{array}{c}\hfill \underset{h\in \mathcal{H}}{sup}\left|{\mathsf{G}}_h\left(x,P_1,P_2\right)\right|,\end{array}$$

(7)

for some fixed $x\in {\mathbb{R}}^n$ and a particular class of functions $\mathcal{H}$. Note for instance that the maximum mean discrepancy is an IPM [16], as well as the Wasserstein distance of order one [17,18,19,20].

Other areas of mathematics in which the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1) plays a key role is distributionally robust optimization (DRO) [21,22] and optimization with relative entropy regularization [8,9]. In these areas, the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ is a central tool. See for instance, [7,23]. Variations of the form ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1) have also been studied in [10] and [11] in the particular case of statistical machine learning for the analysis of generalization error. The central observation is that the generalization error of machine learning algorithms can be written in the form ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6). This observation is the main building block of the method of gaps introduced in [11], which leads to a number of closed-form expressions for the generalization error involving mutual information, lautum information, among other information measures.

2. Preliminaries

The main results presented in this work involve Gibbs conditional probability measures. Such measures are parametrized by a Borel measurable function $h:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$; a $\sigma$-finite measure Q on ${\mathbb{R}}^m$; and a vector $x\in {\mathbb{R}}^n$. Note that the variable x will remain inactive until Section 4. Although it is introduced now for consistency, it could be removed altogether from all results presented in this section and Section 3.

Consider the following function:

$$\begin{array}{c}\hfill {\mathsf{K}}_{h,Q,x}:\{\begin{array}{cc}\hfill \mathbb{R}& ⟶\mathbb{R}\hfill \\ \hfill t& ⟼log\left(\int exp\left(th(x,y)\right)\mathrm{d}Q\left(y\right)\right).\hfill \end{array}.\end{array}$$

(8)

Under the assumption that Q is a probability measure, the function ${\mathsf{K}}_{h,Q,x}$ in (8) is the cumulant generating function of the random variable $h(x,Y)$, for some fixed $x\in {\mathbb{R}}^n$ and $Y\sim Q$. Using this notation, the definition of the Gibbs conditional probability measure is presented hereunder.

Definition 2 

(Gibbs Conditional Probability Measure). Given a Borel measurable function $h:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$; a σ-finite measure Q on ${\mathbb{R}}^m$; and a $\lambda \in \mathbb{R}$, the probability measure $P_{Y|X}^{(h,Q,\lambda )}\in ▵\left({\mathbb{R}}^m|{\mathbb{R}}^n\right)$ is said to be an $(h,Q,\lambda )$-Gibbs conditional probability measure if

$$\begin{array}{ccc}\hfill \forall x\in \mathcal{X},{\mathsf{K}}_{h,Q,x}\left(-\lambda \right)& <\hfill & \hfill +\infty ;\end{array}$$

(9)

for some set $\mathcal{X}\subseteq {\mathbb{R}}^n$; and for all $(x,y)\in \mathcal{X}\times \mathrm{supp}Q$,

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}{\mathrm{d}Q}}\left(y\right)& =\hfill & \hfill exp\left(-\lambda h\left(x,y\right)-{\mathsf{K}}_{h,Q,x}\left(-\lambda \right)\right),\end{array}$$

(10)

where the function ${\mathsf{K}}_{h,Q,x}$ is defined in (8); $\mathrm{supp}Q$ stands for the support of the σ-finite measure Q; and the function ${\displaystyle \frac{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}{\mathrm{d}Q}}$ is the Radon-Nikodym derivative [24,25] of the probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ with respect to Q.

Note that, while $P_{Y|X}^{(h,Q,\lambda )}$ is an $(h,Q,\lambda )$-Gibbs conditional probability measure, the measure $P_{Y|X=x}^{(h,Q,\lambda )}$, obtained by conditioning it upon a given vector $x\in \mathcal{X}$, is referred to as an $(h,Q,\lambda )$-Gibbs probability measure.

The condition in (9) is easily met under certain assumptions. For instance, if h is a nonnegative function and Q is a finite measure, then it holds for all $\lambda \in \left(0,+\infty \right)$. Let ${▵}_Q\left({\mathbb{R}}^m\right)\triangleq \left\{P\in ▵\left({\mathbb{R}}^m\right):P\ll Q\right\}$, with $P\ll Q$ standing for “P absolutely continuous with respect to Q”. The relevance of $(h,Q,\lambda )$-Gibbs probability measures relies on the fact that under some conditions, they are the unique solutions to problems of the form,

$$\begin{array}{c}\hfill \underset{P\in {▵}_Q\left({\mathbb{R}}^m\right)}{min}\int h(x,y)\mathrm{d}P\left(y\right)+{\displaystyle \frac{1}{\lambda }}D(\parallel Q),and\end{array}$$

(11)

$$\begin{array}{c}\hfill \underset{P\in {▵}_Q\left({\mathbb{R}}^m\right)}{max}\int h(x,y)\mathrm{d}P\left(y\right)+{\displaystyle \frac{1}{\lambda }}D(\parallel Q),\end{array}$$

(12)

where $\lambda \in \mathbb{R}\setminus \left\{0\right\}$, $x\in \mathbb{R}$, and $D(\parallel Q)$ denotes the relative entropy (or KL divergence) of P with respect to Q.

Definition 3 

(Relative Entropy). Given two σ-finite measures P and Q on the same measurable space, such that P is absolutely continuous with respect to Q, the relative entropy of P with respect to Q is

$$\begin{array}{c}\hfill D(\parallel Q)=\int {\displaystyle \frac{\mathrm{d}P}{\mathrm{d}Q}}\left(x\right)log\left({\displaystyle \frac{\mathrm{d}P}{\mathrm{d}Q}}\left(x\right)\right)\mathrm{d}Q\left(x\right),\end{array}$$

(13)

where the function ${\displaystyle \frac{\mathrm{d}P}{\mathrm{d}Q}}$ is the Radon-Nikodym derivative of P with respect to Q.

The connection between the optimization problems (11) and (12) and the Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ in (10) has been pointed out by several authors. See for instance, [8] and [26,27,28,29,30,31,32,33,34,35] for the former; and [10] together with [36,37,38] for the latter. In these references a variety of assumptions and proof techniques have been used to prove such connections. A general and unified statement of these observations is presented hereunder.

Lemma 1. 

Assume that the optimization problem in (11) (respectively, in (12)) admits a solution. Then, if $\lambda >0$ (respectively, if $\lambda <0$), the probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ in (10) is the unique solution.

Proof: 

For the case in which $\lambda >0$, the proof follows the same approach as the proof of [8]. Alternatively, for the case in which $\lambda <0$, the proof follows along the lines of the proof of [10]. □

The following lemma highlights a key property of $(h,Q,\lambda )$-Gibbs probability measures.

Lemma 2. 

Given an $(h,Q,\lambda )$-Gibbs probability measure, denoted by $P_{Y|X=x}^{(h,Q,\lambda )}$, with $x\in {\mathbb{R}}^n$,

$$\begin{array}{c}\hfill -{\displaystyle \frac{1}{\lambda }}{\mathsf{K}}_{h,Q,x}\left(-\lambda \right)=\int h(x,y)\mathrm{d}Q\left(y\right)-{\displaystyle \frac{1}{\lambda }}D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})\end{array}$$

(14)

$$\begin{array}{c}\hfill =\int h(x,y)\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}\left(y\right)+{\displaystyle \frac{1}{\lambda }}D(\parallel Q);\end{array}$$

(15)

moreover, if $\lambda >0$,

$$\begin{array}{ccc}\hfill -{\displaystyle \frac{1}{\lambda }}{\mathsf{K}}_{h,Q,x}\left(-\lambda \right)& =\hfill & \hfill \underset{P\in {▵}_Q\left({\mathbb{R}}^m\right)}{min}\int h(x,y)\mathrm{d}P\left(y\right)+{\displaystyle \frac{1}{\lambda }}D(\parallel Q);\end{array}$$

(16)

alternatively, if $\lambda <0$,

$$\begin{array}{ccc}\hfill -{\displaystyle \frac{1}{\lambda }}{\mathsf{K}}_{h,Q,x}\left(-\lambda \right)& =\hfill & \hfill \underset{P\in {▵}_Q\left({\mathbb{R}}^m\right)}{max}\int h(x,y)\mathrm{d}P\left(y\right)+{\displaystyle \frac{1}{\lambda }}D(\parallel Q),\end{array}$$

(17)

where the function ${\mathsf{K}}_{h,Q,x}$ is defined in (8).

Proof: 

The proof of (15) follows from taking the logarithm of both sides of (10) and integrating with respect to $P_{Y|X=x}^{(h,Q,\lambda )}$. As for the proof of (14), it follows by noticing that for all $(x,y)\in {\mathbb{R}}^n\times \mathrm{supp}Q$, the Radon-Nikodym derivative ${\displaystyle \frac{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}{\mathrm{d}Q}}\left(y\right)$ in (10) is strictly positive. Thus, from [39], it holds that ${\displaystyle \frac{\mathrm{d}Q}{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}}\left(y\right)={\left({\displaystyle \frac{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}{\mathrm{d}Q}}\left(y\right)\right)}^{-1}$. Hence, taking the negative logarithm of both sides of (10) and integrating with respect to Q leads to (14). Finally, the equalities in (16) and (17) follow from Lemma 1 and (15). □

Lemma 2, at least equalities (15), (16), and (17), can be seen as an immediate restatement of Donsker-Varadhan variational representation of the relative entropy [40]. Alternative interesting proofs for (14) have been presented by several authors including [10] and [35]. A proof for (15) appears in [29] in the specific case of $\lambda >0$.

The following lemma introduces the main building block of this work, which is a characterization of the variation from the probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ in (10) to an arbitrary measure $P\in {▵}_Q\left({\mathbb{R}}^m\right)$, i.e., ${\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)$. Such a result appeared for the first time in [7] for the case in which $\lambda >0$; and in [10] for the case in which $\lambda <0$, in different contexts of statistical machine learning. A general and unified statement of such results is presented hereunder.

Lemma 3. 

Consider an $(h,Q,\lambda )$-Gibbs probability measure, denoted by $P_{Y|X=x}^{(h,Q,\lambda )}\in ▵\left({\mathbb{R}}^m\right)$, with $\lambda \ne 0$ and $x\in \mathbb{R}$. For all $P\in {▵}_Q\left({\mathbb{R}}^m\right)$,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\left(D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})+D(\parallel Q)-D(\parallel Q)\right).\end{array}$$

(18)

Proof: 

The proof follows along the lines of the proofs of [7] for the case in which $\lambda >0$; and in [10] for the case in which $\lambda <0$. A unified proof is presented hereunder by noticing that for all $P\in {▵}_Q\left({\mathbb{R}}^m\right)$,

$$\begin{array}{ccc}\hfill D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})& =\hfill & \hfill \int log\left({\displaystyle \frac{\mathrm{d}P}{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}}\left(y\right)\right)\mathrm{d}P\left(y\right)\end{array}$$

(19)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill \int log\left({\displaystyle \frac{\mathrm{d}Q}{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}}\left(y\right){\displaystyle \frac{\mathrm{d}P}{\mathrm{d}Q}}\left(y\right)\right)\mathrm{d}P\left(y\right)\end{array}$$

(20)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill \int log\left({\displaystyle \frac{\mathrm{d}Q}{\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}}}\left(y\right)\right)\mathrm{d}P\left(y\right)+D(\parallel Q)\end{array}$$

(21)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill \lambda \int h(x,y)\mathrm{d}P\left(y\right)+{\mathsf{K}}_{h,Q,x}\left(-\lambda \right)+D(\parallel Q)\end{array}$$

(22)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill \lambda {\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)-D(\parallel Q)+D(\parallel Q),\end{array}$$

(23)

where (20) follows from [39]; (22) follows from [39] and (10); and (23) follows from (15). □

It is interesting to highlight that ${\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)$ in (18) characterizes the variation of the expectation of the function $h(x,·):{\mathbb{R}}^m\to \mathbb{R}$, when $\lambda >0$ (resp. $\lambda <0$) and the probability measure changes from the solution to the optimization problem (11) (resp. (12)) to an alternative measure P. This result takes another perspective if it is seen in the context of information projections [13]. Let Q be a probability measure and $\mathcal{S}\subseteq {▵}_Q\left({\mathbb{R}}^m\right)$ be a convex set. From [13], it holds that for all measures $P\in \mathcal{S}$,

$$\begin{array}{c}\hfill D(\parallel Q)⩾D(\parallel P^{★})+D(\parallel Q),\end{array}$$

(24)

where, $P^{★}$ satifies

$$\begin{array}{c}\hfill P^{★}\in arg\underset{P\in \mathcal{S}}{min}D(\parallel Q).\end{array}$$

(25)

In the particular case in which the set $\mathcal{S}$ in (24) satisfies

$$\begin{array}{ccc}\hfill \mathcal{S}& \triangleq \hfill & \hfill \left\{P\in {▵}_Q\left({\mathbb{R}}^m\right):\int h(x,y)\mathrm{d}P\left(y\right)=c\right\},\end{array}$$

(26)

for some real c, with the vector x and the function h defined in Lemma 3, the optimal measure $P^{★}$ in (25) is the Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ in (10), with $\lambda >0$ chosen to satisfy

$$\begin{array}{c}\hfill \int h(x,y)\mathrm{d}{P}_{Y|X=x}^{(h,Q,\lambda )}\left(y\right)=c.\end{array}$$

(27)

The case in which the measure Q in (25) is a $\sigma$-finite measure, for instance, either the Lebesgue measure or the counting measure, respectively leads to the classical framework of differential entropy maximization or discrete entropy maximization, which have been studied under particular assumptions on the set $\mathcal{S}$ in [36,37] and [38].

When the reference measure Q is a probability measure, under the assumption that (27) holds, it follows from [13] that for all $P\in \mathcal{S}$, with $\mathcal{S}$ in (26),

$$\begin{array}{ccc}\hfill D(\parallel Q)& =\hfill & \hfill D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})+D(\parallel Q),\end{array}$$

(28)

which is known as the Pythagorean theorem for relative entropy. Such a geometric interpretation follows from admitting relative entropy as an analog of squared Euclidean distance. The first appearance of such a “Pythagorean theorem” was in [12] and was later revisited in [13]. Interestingly, the same result can be obtained from Lemma 3 by noticing that for all $P\in \mathcal{S}$, with $\mathcal{S}$ in (26),

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)& =\hfill & \hfill 0.\end{array}$$

(29)

The converse of the Pythagorean theorem [41] together with Lemma 3, lead to the geometric construction shown in Figure 1. A similar interpretation was also presented in [11] and [11] in the context of the generalization error of machine learning algorithms. The former considers $\lambda >0$, while the latter considers $\lambda <0$. Nonetheless, the interpretation in Figure 1 is general and independent of such an application.

The relevance of Lemma 3, with respect to this body of literature on information projections, follows from the fact that Q might be a $\sigma$-finite measure, which is a class of measures that includes the class of probability measures, and thus, unifies the results separately obtained in the realm of maximum entropy methods and information-projection methods. More importantly, when $P\notin \mathcal{S}$, with $\mathcal{S}$ in (26), it might hold that ${\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)<0$ or ${\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)>0$, with ${\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)$ in (18), which resonates with the fact that $(h,Q,\lambda )$-Gibbs conditional probability measures are also related to another class of optimization problems, as shown by the following lemma.

Lemma 4. 

Assume that the following optimization problems possess at least one solution for some $x\in {\mathbb{R}}^n$,

$$\begin{array}{ccc}\hfill \underset{P\in {▵}_Q\left({\mathbb{R}}^m\right)}{min}& \hfill & \hfill \int h(x,y)\mathrm{d}P\left(y\right)\end{array}$$

(30a)

$$\begin{array}{ccc}\hfill \mathrm{s}.\mathrm{t}.& & \hfill D(\parallel Q)⩽\rho .\end{array}$$

(30b)

and

$$\begin{array}{ccc}\hfill \underset{P\in {▵}_Q\left({\mathbb{R}}^m\right)}{max}& \hfill & \hfill \int h(x,y)\mathrm{d}P\left(y\right)\end{array}$$

(31a)

$$\begin{array}{ccc}\hfill \mathrm{s}.\mathrm{t}.& & \hfill D(\parallel Q)⩽\rho .\end{array}$$

(31b)

Consider the $(h,Q,\lambda )$-Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ in (10), with $\lambda \in \mathbb{R}\setminus \left\{0\right\}$ such that $\rho =D(\parallel Q)$. Then, the $(h,Q,\lambda )$-Gibbs probability measure $P_{Y|X=x}^{(h,Q,\lambda )}$ is a solution to (4) if $\lambda >0$; or to (4) if $\lambda <0$.

Proof: 

Note that if $\lambda >0$, then, ${\displaystyle \frac{1}{\lambda }}D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})⩾0$. Hence, from Lemma 3, it holds that for all probability mesures P such that $D(\parallel Q)⩽\rho$,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)& ⩾\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\left(D(\parallel Q)-D(\parallel Q)\right)\end{array}$$

(32)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\left(\rho -D(\parallel Q)\right)\end{array}$$

(33)

$$\begin{array}{ccc}\hfill & ⩾\hfill & \hfill 0,\end{array}$$

(34)

with equality if $D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})=0$. This implies that $P_{Y|X=x}^{(h,Q,\lambda )}$ is a solution to (4). Note also that if $\lambda <0$, from Lemma 3, it holds that for all probability mesures P such that $D(\parallel Q)⩽\rho$,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P,P_{Y|X=x}^{(h,Q,\lambda )}\right)& ⩽\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\left(D(\parallel Q)-D(\parallel Q)\right)\end{array}$$

(35)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\left(\rho -D(\parallel Q)\right)\end{array}$$

(36)

$$\begin{array}{ccc}\hfill & ⩽\hfill & \hfill 0,\end{array}$$

(37)

with equality if $D(\parallel P_{Y|X=x}^{(h,Q,\lambda )})=0$. This implies that $P_{Y|X=x}^{(h,Q,\lambda )}$ is a solution to (4). □

3. Characterization of ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1)

The main result of this section is the following theorem.

Theorem 1. 

For all probability measures $P_1$ and $P_2$, both absolutely continuous with respect to a given σ-finite measure Q on ${\mathbb{R}}^m$, the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1) satisfies,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P_1,P_2\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}(D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})+D(\parallel Q)-D(\parallel Q)),\end{array}$$

(38)

where the probability measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$, is an $(h,Q,\lambda )$-Gibbs probability measure.

Proof: 

The proof follows from Lemma 3 and by observing that

$${\mathsf{G}}_h\left(x,P_1,P_2\right)={\mathsf{G}}_h\left(x,P_1,P_{Y|X=x}^{(h,Q,\lambda )}\right)-{\mathsf{G}}_h\left(x,P_2,P_{Y|X=x}^{(h,Q,\lambda )}\right).$$

□

Theorem 1 might be particularly simplified in the case in which the reference measure Q is a probability measure. Consider for instance the case in which $P_1\ll P_2$ (or $P_2\ll P_1$). In such a case, the reference measure might be chosen as $P_2$ (or $P_1$), as shown hereunder.

Corollary 1. 

Consider the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1). If the probability measure $P_1$ is absolutely continuous with respect to $P_2$, then,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P_1,P_2\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}(D(\parallel P_{Y|X=x}^{\left(h,P_2,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,P_2,\lambda \right)})-D(\parallel P_2)).\end{array}$$

(39)

Alternatively, if the probability measure $P_2$ is absolutely continuous with respect to $P_1$, then,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P_1,P_2\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}(D(\parallel P_{Y|X=x}^{\left(h,P_1,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,P_1,\lambda \right)})+D(\parallel P_1)),\end{array}$$

(40)

where the probability measures $P_{Y|X=x}^{\left(h,P_1,\lambda \right)}$ and $P_{Y|X=x}^{\left(h,P_2,\lambda \right)}$ are respectively $(h,P_1,\lambda )$- and $(h,P_2,\lambda )$-Gibbs probability measures, with $\lambda \ne 0$.

In the case in which neither $P_1$ is absolutely continuous with respect to $P_2$; nor $P_2$ is absolutely continuous with respect to $P_1$, the reference measure Q in Theorem 1 can always be chosen as a convex combination of $P_1$ and $P_2$. That is, for all Borel sets $\mathcal{A}\in \mathcal{B}{\mathbb{R}}^m$, $Q\left(\mathcal{A}\right)=\alpha P_1\left(\mathcal{A}\right)+(1-\lambda )P_2\left(\mathcal{A}\right)$, with $\alpha \in (0,1)$.

Theorem 1 can be specialized to the specific cases in which Q is the Lebesgue or the counting measure.

If Q is the Lebesgue measure the probability measures $P_1$ and $P_2$ in (38) admit probability density functions $f_1$ and $f_2$, respectively. Moreover, the terms $-D(\parallel Q)$ and $-D(\parallel Q)$ are Shannon’s differential entropies [4] induced by $P_1$ and $P_2$, denoted by $\mathsf{h}\left(P_1\right)$ and $\mathsf{h}\left(P_2\right)$, respectively. That is, for all $i\in \{1,2\}$,

$$\begin{array}{ccc}\hfill \mathsf{h}\left(P_i\right)& \triangleq \hfill & \hfill -\int f_i\left(x\right)log{f}_i\left(x\right)\mathrm{d}x.\end{array}$$

(41)

The probability measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$, $x\in {\mathbb{R}}^n$, and Q the Lebesgue measure, possesses a probability density function, denoted by $f_{Y|X=x}^{\left(h,Q,\lambda \right)}:{\mathbb{R}}^m\to (0,+\infty )$, which satisfies

$$\begin{array}{ccc}\hfill f_{Y|X=x}^{\left(h,Q,\lambda \right)}\left(y\right)& =\hfill & \hfill {\displaystyle \frac{exp\left(-\lambda h(x,y)\right)}{\int exp\left(-\lambda h(x,y)\right)\mathrm{d}y}}.\end{array}$$

(42)

If Q is the counting measure the probability measures $P_1$ and $P_2$ in (38) admit probability mass functions $p_1:\mathcal{Y}\to [0,1]$ and $p_2:\mathcal{Y}\to [0,1]$, with $\mathcal{Y}$ a countable subset of ${\mathbb{R}}^m$. Moreover, $-D(\parallel Q)$ and $-D(\parallel Q)$ are respectively Shannon’s discrete entropies [4] induced by $P_1$ and $P_2$, denoted by $\mathsf{H}\left(P_1\right)$ and $\mathsf{H}\left(P_2\right)$, respectively. That is, for all $i\in \{1,2\}$,

$$\begin{array}{ccc}\hfill \mathsf{H}\left(P_i\right)& \triangleq \hfill & \hfill -\sum _{y\in \mathcal{Y}}{p}_i\left(y\right)log{p}_i\left(y\right).\end{array}$$

(43)

The probability measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$ and Q the counting measure, possesses a conditional probability mass function, denoted by $p_{Y|X=x}^{\left(h,Q,\lambda \right)}:\mathcal{Y}\to (0,+\infty )$, which satisfies

$$\begin{array}{ccc}\hfill p_{Y|X=x}^{\left(h,Q,\lambda \right)}\left(y\right)& =\hfill & \hfill {\displaystyle \frac{exp\left(-\lambda h(x,y)\right)}{{\sum }_{y\in \mathcal{Y}}exp\left(-\lambda h(x,y)\right)}}.\end{array}$$

(44)

These observations lead to the following corollary of Theorem 1.

Corollary 2. 

Given two probability measures $P_1$ and $P_2$, with probability density functions $f_1$ and $f_2$ respectively, the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1) satisfies,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P_1,P_2\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}(D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-\mathsf{h}\left(P_2\right)+\mathsf{h}\left(P_1\right)),\end{array}$$

(45)

where the probability density function of the measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$ and Q the Lebesgue measure, is defined in (42); and the entropy functional $\mathsf{h}$ is defined in (41). Alternatively, given two probability measures $P_1$ and $P_2$, with probability mass functions $p_1$ and $p_2$ respectively, the variation ${\mathsf{G}}_h\left(x,P_1,P_2\right)$ in (1) satisfies,

$$\begin{array}{ccc}\hfill {\mathsf{G}}_h\left(x,P_1,P_2\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}(D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-\mathsf{H}\left(P_2\right)+\mathsf{H}\left(P_1\right)),\end{array}$$

(46)

where the probability mass function of the measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$ and Q the counting measure, is defined in (44); and the entropy functional $\mathsf{H}$ is defined in (43).

4. Characterizations of ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ in (2)

The main result of this section is a characterization of ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ in (2).

Theorem 2. 

Consider the variation ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ in (2) and assume that for all $x\in \mathrm{supp}{P}_X$, the probability measures $P_{Y|X=x}^{\left(1\right)}$ and $P_{Y|X=x}^{\left(2\right)}$ are both absolutely continuous with respect to a σ-measure Q. Then,

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)={\displaystyle \frac{1}{\lambda }}\int (D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})\end{array}$$

$$\begin{array}{c}\hfill +D(\parallel Q)-D(\parallel Q))\mathrm{d}{P}_X\left(x\right),\end{array}$$

(47)

where the probability measure $P_{Y|X}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$, is an $(h,Q,\lambda )$-Gibbs conditional probability measure.

Proof: 

The proof follows from (2) and Theorem 1. □

Two special cases are particularly noteworthy.

When the reference measure Q is the Lebesgue measure

both $-\int D(\parallel Q)\mathrm{d}{P}_X\left(x\right)$ and $-\int D(\parallel Q)\mathrm{d}{P}_X\left(x\right)$ in (47) become Shannon’s differential conditional entropies, denoted by $\mathsf{h}\left(P_{Y|X}^{\left(1\right)}|P_X\right)$ and $\mathsf{h}\left(P_{Y|X}^{\left(2\right)}|P_X\right)$, respectively. That is, for all $i\in \{1,2\}$,

$$\begin{array}{ccc}\hfill \mathsf{h}\left(P_{Y|X}^{\left(i\right)}|P_X\right)& \triangleq \hfill & \hfill \int \mathsf{h}\left(P_{Y|X=x}^{\left(i\right)}\right)\mathrm{d}{P}_X\left(x\right),\end{array}$$

(48)

where $\mathsf{h}$ is the entropy functional in (41).

When the reference measure Q is the counting measure

both $-\int D(\parallel Q)\mathrm{d}{P}_X\left(x\right)$ and $-\int D(\parallel Q)\mathrm{d}{P}_X\left(x\right)$ in (47) become Shannon’s discrete conditional entropies, denoted by $\mathsf{H}\left(P_{Y|X}^{\left(1\right)}|P_X\right)$ and $\mathsf{H}\left(P_{Y|X}^{\left(2\right)}|P_X\right)$, respectively. That is, for all $i\in \{1,2\}$,

$$\begin{array}{ccc}\hfill \mathsf{H}\left(P_{Y|X}^{\left(i\right)}|P_X\right)& \triangleq \hfill & \hfill \int \mathsf{H}\left(P_{Y|X=x}^{\left(i\right)}\right)\mathrm{d}{P}_X\left(x\right),\end{array}$$

(49)

where $\mathsf{H}$ is the entropy functional in (43).

These observations lead to the following corollary of Theorem 2.

Corollary 3. 

Consider the variation ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ in (2) and assume that for all $x\in \mathrm{supp}{P}_X$, the probability measures $P_{Y|X=x}^{\left(1\right)}$ and $P_{Y|X=x}^{\left(2\right)}$ possess probability density functions. Then,

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)={\displaystyle \frac{1}{\lambda }}\int (D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)}))\mathrm{d}{P}_X\left(x\right)\end{array}$$

$$\begin{array}{c}\hfill -{\displaystyle \frac{1}{\lambda }}\mathsf{h}\left(P_{Y|X}^{\left(2\right)}|P_X\right)+{\displaystyle \frac{1}{\lambda }}\mathsf{h}\left(P_{Y|X}^{\left(1\right)}|P_X\right),\end{array}$$

(50)

where the probability density function of the measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$ and Q the Lebesgue measure, is defined in (42); and for all $i\in \{1,2\}$, the conditional entropy $\mathsf{h}\left(P_{Y|X}^{\left(i\right)}|P_X\right)$ is defined in (48). Alternatively, assume that for all $x\in \mathrm{supp}{P}_X$, the probability measures $P_{Y|X=x}^{\left(1\right)}$ and $P_{Y|X=x}^{\left(2\right)}$ possess probability mass functions. Then,

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)={\displaystyle \frac{1}{\lambda }}\int (D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,Q,\lambda \right)}))\mathrm{d}{P}_X\left(x\right)\end{array}$$

$$\begin{array}{c}\hfill -{\displaystyle \frac{1}{\lambda }}\mathsf{H}\left(P_{Y|X}^{\left(2\right)}|P_X\right)+{\displaystyle \frac{1}{\lambda }}\mathsf{H}\left(P_{Y|X}^{\left(1\right)}|P_X\right),\end{array}$$

(51)

where the probability mass function of the measure $P_{Y|X=x}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$ and Q the counting measure, is defined in (44); and for all $i\in \{1,2\}$, the conditional entropy $\mathsf{H}\left(P_{Y|X}^{\left(i\right)}|P_X\right)$ is defined in (49).

Note that, from (2), it follows that the general expression for the expected variation ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ might be simplified according to Corollary 1. For instance, if for all $x\in \mathrm{supp}{P}_X$, the probability measure $P_{Y|X=x}^{\left(1\right)}$ is absolutely continuous with respect to $P_{Y|X=x}^{\left(2\right)}$, the measure $P_{Y|X=x}^{\left(2\right)}$ can be chosen to be the reference measure in the calculation of ${\mathsf{G}}_h\left(x,P_{Y|X=x}^{\left(1\right)},P_{Y|X=x}^{\left(2\right)}\right)$ in (2). This observation leads to the following corollary of Theorem 2.

Corollary 4. 

Consider the variation ${\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)$ in (2) and assume that for all $x\in \mathrm{supp}{P}_X$, $P_{Y|X=x}^{\left(1\right)}\ll P_{Y|X=x}^{\left(2\right)}$. Then,

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)={\displaystyle \frac{1}{\lambda }}\int (D(\parallel P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(2\right)},\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(2\right)},\lambda \right)})\end{array}$$

$$\begin{array}{c}\hfill -D(\parallel P_{Y|X=x}^{\left(2\right)}))\mathrm{d}{P}_X\left(x\right).\end{array}$$

(52)

Alternatively, if for all $x\in \mathrm{supp}{P}_X$, the probability measure $P_{Y|X=x}^{\left(2\right)}$ is absolutely continuous with respect to $P_{Y|X=x}^{\left(1\right)}$, then,

$$\begin{array}{c}\hfill {\overline{\mathsf{G}}}_h\left(P_{Y|X}^{\left(1\right)},P_{Y|X}^{\left(2\right)},P_X\right)={\displaystyle \frac{1}{\lambda }}\int (D(\parallel P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(1\right)},\lambda \right)})-D(\parallel P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(1\right)},\lambda \right)})\end{array}$$

$$\begin{array}{c}\hfill +D(\parallel P_{Y|X=x}^{\left(1\right)}))\mathrm{d}{P}_X\left(x\right),\end{array}$$

(53)

where the measures $P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(1\right)},\lambda \right)}$ and $P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(2\right)},\lambda \right)}$ are respectively $(h,P_{Y|X=x}^{\left(1\right)},\lambda )$- and $(h,P_{Y|X=x}^{\left(2\right)},\lambda )$-Gibbs probability measures.

The Gibbs probability measures $P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(1\right)},\lambda \right)}$ and $P_{Y|X=x}^{\left(h,P_{Y|X=x}^{\left(2\right)},\lambda \right)}$ in Corollary 4 are particularly interesting as their reference measures depend on x. Gibbs measures of this form appear, for instance, in [8].

5. Characterizations of ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6)

The main result of this section is a characterization of ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6), which describes the variation of the expectation of the function h when the probability measure changes from the joint probability measure $P_{Y|X}{P}_X$ to the product of its marginals $P_Y·P_X$. This result is presented hereunder and involves the mutual information $I\left(P_{Y|X};P_X\right)$ and lautum information $L\left(P_{Y|X};P_X\right)$, defined as follows:

$$\begin{array}{ccc}\hfill I\left(P_{Y|X};P_X\right)& \triangleq \hfill & \hfill \int D(\parallel P_Y)\mathrm{d}{P}_X\left(x\right);and\end{array}$$

(54)

$$\begin{array}{ccc}\hfill L\left(P_{Y|X};P_X\right)& \triangleq \hfill & \hfill \int D(\parallel P_{Y|X=x})\mathrm{d}{P}_X\left(x\right).\end{array}$$

(55)

Theorem 3. 

Consider the expected variation ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6) and assume that, for all $x\in \mathrm{supp}{P}_X$:

• The probability measures $P_Y$ and $P_{Y|X=x}$ are both absolutely continuous with respect to a given σ-finite measure Q; and
• The probability measures $P_Y$ and $P_{Y|X=x}$ are mutually absolutely continuous.

Then, it follows that

$$\begin{array}{ccc}\hfill & \hfill & \hfill {\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)={\displaystyle \frac{1}{\lambda }}(I\left(P_{Y|X};P_X\right)+L\left(P_{Y|X};P_X\right)\\ \hfill & \hfill & \hfill +\int \int log\left({\displaystyle \frac{\mathrm{d}{P}_{Y|X=x}}{\mathrm{d}{P}_{Y|X=x}^{\left(h,Q,\lambda \right)}}}\left(y\right)\right)\mathrm{d}{P}_Y\left(y\right)\mathrm{d}{P}_X\left(x\right)-\int \int log\left({\displaystyle \frac{\mathrm{d}{P}_{Y|X=x}}{\mathrm{d}{P}_{Y|X=x}^{\left(h,Q,\lambda \right)}}}\left(y\right)\right)\mathrm{d}{P}_{Y|X=x}\left(y\right)\mathrm{d}{P}_X\left(x\right)),\end{array}$$

(56)

where the probability measure $P_{Y|X}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$, is an $(h,Q,\lambda )$-Gibbs conditional probability measure.

Proof: 

The proof is presented in Appendix A. □

An alternative expression for ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6) involving only relative entropies is presented by the following theorem.

Theorem 4. 

Consider the expected variation ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6) and assume that, for all $x\in \mathrm{supp}{P}_X$, the probability measure $P_{Y|X=x}$ is absolutely continuous with respect to a given σ-finite measure Q. Then, it follows that

$$\begin{array}{ccc}\hfill & \hfill & \hfill {\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)\\ \hfill & =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\int \int (D(\parallel P_{Y|X=x_1}^{\left(h,Q,\lambda \right)})-D(\parallel P_{Y|X=x_2}^{\left(h,Q,\lambda \right)}))\mathrm{d}{P}_X\left(x_1\right)\mathrm{d}{P}_X\left(x_2\right),\end{array}$$

(57)

where $P_{Y|X}^{\left(h,Q,\lambda \right)}$, with $\lambda \ne 0$, is an $(h,Q,\lambda )$-Gibbs conditional probability measure.

Proof: 

The proof is presented in Appendix B. □

Theorem 4 expresses the variation ${\overline{\mathsf{G}}}_h\left(P_Y,P_{Y|X},P_X\right)$ in (6) as the expectation (w.r.t. $P_X{P}_X$) of a comparison of the conditional probability measure $P_{Y|X}$ with a Gibbs conditional probability measure $P_{Y|X}^{\left(h,Q,\lambda \right)}$ via relative entropy. More specifically, the expression consists in the expectation of the difference of two relative entropies. The former compares $P_{Y|X=x_1}$ with $P_{Y|X=x_2}^{\left(h,Q,\lambda \right)}$, where $(x_1,x_2)\in \mathcal{X}\times \mathcal{X}$ are independently sampled from the same probability measure $P_X$. The latter compares these two conditional measures conditioning on the same element of $\mathcal{X}$. That is, it compares $P_{Y|X=x_2}$ with $P_{Y|X=x_2}^{\left(h,Q,\lambda \right)}$.

An interesting observation from Theorem 3 and Theorem 4 is that the last two terms in the right-hand side of (56) are both zero in the case in which $P_{Y|X}$ is an $(h,Q,\lambda )$-Gibbs conditional probability measure. Similarly, in such a case, the second term in the right-hand side of (57) is also zero. This observation is highlighted by the following corollary.

Corollary 5. 

Consider an $(h,Q,\lambda )$-Gibbs conditional probability measure, denoted by $P_{Y|X}^{(h,Q,\lambda )}\in ▵\left({\mathbb{R}}^m|{\mathbb{R}}^n\right)$, with $\lambda \ne 0$; and a probability measure $P_X\in ▵\left({\mathbb{R}}^n\right)$. Let the measure $P_Y^{(h,Q,\lambda )}\in ▵\left({\mathbb{R}}^m\right)$ be such that for all sets $\mathcal{A}\in \mathcal{B}{\mathbb{R}}^m$,

$$\begin{array}{ccc}\hfill P_Y^{(h,Q,\lambda )}\left(\mathcal{A}\right)& =\hfill & \hfill \int P_{Y|X=x}^{(h,Q,\lambda )}\left(\mathcal{A}\right)\mathrm{d}{P}_X\left(x\right).\end{array}$$

(58)

Then,

$$\begin{array}{ccc}\hfill {\overline{\mathsf{G}}}_h\left(P_Y^{(h,Q,\lambda )},P_{Y|X}^{(h,Q,\lambda )},P_X\right)& =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\left(I\left(P_{Y|X}^{(h,Q,\lambda )};P_X\right)+L\left(P_{Y|X}^{(h,Q,\lambda )};P_X\right)\right)\end{array}$$

(59)

$$\begin{array}{ccc}\hfill & =\hfill & \hfill {\displaystyle \frac{1}{\lambda }}\int \int D(\parallel P_{Y|X=x_1}^{\left(h,Q,\lambda \right)})\mathrm{d}{P}_X\left(x_1\right)\mathrm{d}{P}_X\left(x_2\right).\end{array}$$

(60)

Note that mutual information and lautum information are both nonnegative information measures, which from Corollary 5, implies that ${\overline{\mathsf{G}}}_h\left(P_Y^{(h,Q,\lambda )},P_{Y|X}^{(h,Q,\lambda )},P_X\right)$ in (59) might be either positive or negative depending exclusively on the sign of the regularization factor $\lambda$. The following corollary exploits such an observation to present a property of Gibbs conditional probability measures and their corresponding marginal probability measures.

Corollary 6. 

Given a probability measure $P_X\in ▵\left({\mathbb{R}}^n\right)$, the $(h,Q,\lambda )$-Gibbs conditional probability measure $P_{Y|X}^{\left(h,Q,\lambda \right)}$ in (10) and the probability measure $P_Y^{(h,Q,\lambda )}$ in (58) satisfy

$$\begin{array}{ccc}\hfill \int \int h(x,y)\mathrm{d}{P}_Y^{\left(h,Q,\lambda \right)}\left(y\right)\mathrm{d}{P}_X\left(x\right)& ⩾\hfill & \hfill \int \int h(x,y)\mathrm{d}{P}_{Y|X=x}^{\left(h,Q,\lambda \right)}\left(y\right)\mathrm{d}{P}_X\left(x\right)if\lambda >0;\end{array}$$

(61)

or

$$\begin{array}{ccc}\hfill \int \int h(x,y)\mathrm{d}{P}_Y^{\left(h,Q,\lambda \right)}\left(y\right)\mathrm{d}{P}_X\left(x\right)& ⩽\hfill & \hfill \int \int h(x,y)\mathrm{d}{P}_{Y|X=x}^{\left(h,Q,\lambda \right)}\left(y\right)\mathrm{d}{P}_X\left(x\right)if\lambda <0.\end{array}$$

(62)

Corollary 6 highlights the fact that a deviation from the joint probability measure $P_{Y|X}^{\left(h,Q,\lambda \right)}{P}_X\in ▵\left(\mathcal{Y}\times \mathcal{X}\right)$ to the product of its marignals $P_Y^{\left(h,Q,\lambda \right)}{P}_X\in ▵\left(\mathcal{Y}\times \mathcal{X}\right)$ might increase or decrease the expectation of the function h depending on the sign of $\lambda$.

6. Final Remarks

A simple re-formulation of Varandan’s variational representation of relative entropy (Lemma 2) has been harnessed to provide an explicit expression of the variation of the expectation of a multi-dimensional real function when the probability measure changes from a Gibbs probability measure to an arbitrary measure (Lemma 3). This result reveals strong connections with information projection methods, Pythagorean identities involving relative entropy, and mean optimization problems constrained to a neighborhood around a reference measure, where the neighborhood is defined via an upper bound on relative entropy with respect to the reference measure (Lemma 4). An algebraic manipulation on Lemma 3 leads to an explicit expression for the variation of the expectation under study when the probability measure changes from an arbitrary measure to another arbitrary measure (Theorem 1). The astonishing simplicity in the proof, which is straight forward from Lemma 3, contrasts with the generality of the result. In particular, the only assumption is that both measures, before and after the variation, are absolutely continuous with a reference measure. The underlying observation is the central role played by Gibbs probability distributions in this analysis. In particular, such a variation is expressed in terms of comparisons, via relative entropy, of the initial and final probability measures with respect to the Gibbs probability measure specifically built for the function under study. Interestingly, the reference measure of such Gibbs probability measures can be freely chosen beyond probability measures. When such a reference is a $\sigma$-finite measure, e.g., Lebesgue measure or a counting measure, the expressions mentioned above include Shannon’s fundamental information measures, e.g., entropy and conditional entropy (Corollary 2).

Using these initial results, the variations of expectations of multi-dimensional functions due to variations of joint probability measures has been studied. In this case, the focus has been on two particular measure changes, which unveil connections with both mutual and lautum information. First, one of the marginal probability measures remains the same after the change (Theorem 2); and second, the joint probability measure changes to the product of its marginals (Theorem 3 and Theorem 4). In the case of Gibbs joint probability measures, these expressions involve exclusively well known information measures: mutual information; lautum information; and relative entropy. These expressions reveal general connections between the variation in the expectation of arbitrary functions, induced by changes in the underlying probability measure, and both mutual and lautum information. These connections extend beyond those previously established in the analysis of generalization error in machine learning algorithms; see, for instance, [26] and [8].