Information Geometry Description of Inferential Scattering

1. Introduction

The notion of Inferential Scattering was exposed by E.T. Jaynes in a couple of papers [1,2] whose content has remained little known in the scientific literature with respect to more celebrated works of this author. Jaynes’ original exposition refers to equilibrium statistical mechanics. It is well known that the probabilistic formalism of equilibrium statistical mechanics is described by an exponential family manifold which is an instance of a dually flat manifold [3]; the main aim of this work is to rephrase the content of the notion of inferential scattering in this enlarged setting of dually flat manifolds to better appreciate its geometric structure and possibly expand its range of applications. Dually flat manifolds are a generalization of Riemannian manifolds and they occupy a central place in Information Geometry. The have found multiple applications in the description of neural networks [4], machine learning (especially Restricted Boltzmann Machines) [5] and the description of hierarchical systems [6].

In a dually flat manifold M there are two local coordinate systems $\theta =\theta \left(p\right)$ and $\eta =\eta \left(p\right)$ which describe the same point $p\in M$. The transition maps $\eta \circ {\theta }^{-1}$ and $\theta \circ {\eta }^{-1}$ are related by a Legendre transform so that $\eta \left(\theta \right)=\nabla \psi \left(\theta \right)$ with inverse $\theta \left(\eta \right)=\nabla \phi \left(\eta \right)$ where ∇ denotes the gradient and $\phi ,\psi$ are called potentials. In the case of an exponential manifold, there is a natural statistical mechanic interpretation of $\theta$ as control variables and $\eta$ as effect variables, see Section 4. The tangents of the above introduced gradient maps give the linear variation $d\eta$ in the effect variables due to a variation $d\theta$ in control variables and vice-versa.

The phenomenon of inferential scattering arises when particular kinds of variations $d\eta$ and $d\theta$ are considered. In this work it is shown that its underlying geometric structure can be understood in terms of horizontal lift with respect to a couple of integrable connections (dually orthogonal foliations) which are characteristics of the dually flat manifold structure. The notion of inferential scattering comes into play because the variation $d\eta$ in effect variables can be written as an infinite sum of contributions. According to Jaynes [1] this perturbative expansion can be interpreted using the multiple scattering phenomenon of a wave emanating from a point source A in presence of an obstacle C: the intensity of the wave at a point B in space is the sum of the contribution of the wave arriving from A plus the contribution of the wave reflected by C plus the one of the wave reflected from C back in A and then in B and so on with the higher order multiple reflections (Born series, see [10]). The main interest of this analysis of Jaynes’ result is to show that the scattering phenomenon is present also in this abstract setting of a dually flat manifold, where it establishes a relation between the dual coordinate systems $\theta$ and $\eta$.

We conclude with the plan of the paper: In Section 2 we give a cursory view of the differential geometry notion of dually flat manifold and mutually dual foliations. In Section 3 we describe the paradigmatic case of the exponential family manifold and in Section 4 we investigate the geometric structure of the notion of direct and inverse control and in Section 5 the one of inferential scattering.

2. Dually Flat Manifolds

We briefly recall the notions of differential geometry useful for introducing the definition of a dually flat manifold, a central object in information geometry. We keep the exposition to a minimum, without proofs, essentially following [3]. Other important introductory references are [5,7,9].

Let $(M,g)$ be a n-dimensional smooth manifold with g a Riemannian metric. In a local chart $\zeta$ the coordinates of a point $p\in M$ are $\zeta \left(p\right)=(x_1,\dots x_n)$ and the canonical basis is ${\partial }_i={\displaystyle \frac{\partial }{\partial x_i}}$. We equip M with a linear connection ∇ whose expression in the local chart is given by the Christoffel symbols ${\Gamma }_{ij}^k$ as

$${\nabla }_{{\partial }_i}{\partial }_j={\Gamma }_{ij}^k{\partial }_k,{\Gamma }_{ij,k}=g({\nabla }_{{\partial }_i}{\partial }_j,{\partial }_k)=g_{lk}{\Gamma }_{ij}^l.$$

Throughout the paper we adopt the convention that repeated indices are summed. The torsion $\tau$ of the connection ∇ is defined as

$$\tau (X,Y)={\nabla }_{X}Y-{\nabla }_{Y}X-[X,Y]$$

where $X,Y\in \mathfrak{X}\left(M\right)$ are vector fields on M and $[X,Y]$ is the Lie bracket. The connection ∇ is said to be compatible with the metric g or simply metric if

$$X·g(X,Y)=g({\nabla }_{X}Y,Z)+g(X,{\nabla }_{Y}Z)\forall X,Y,Z\in \mathfrak{X}\left(M\right).$$

(1)

If the connection is metric, the parallel transport defined by ∇ is an isometry, preserving the length and the angle between vectors. A celebrated theorem in differential geometry states that in a Riemannian manifold there exists a unique metric connection which is torsion-free $\tau =0$, the so-called Levi-Civita connection $\widehat{\nabla }$. For later use we introduce the curvature defined by the connection ∇

$$R(X,Y,Z)={\nabla }_X\left({\nabla }_{Y}Z\right)-{\nabla }_Y\left({\nabla }_{X}Z\right)-{\nabla }_{[X,Y]}Z.$$

The connection ∇ is flat if the associated curvature is identically zero. This is a coordinate-free condition and the Christoffel symbols are zero in every coordinate chart. Moreover, there exists a local affine coordinate system with the property that every canonical vector $\partial /\partial x_i$ of the basis can be parallel transported along the coordinate line $x_i$ so each coordinate line is a geodesic.

The above notion (1) of compatibility between the metric g and a connection ∇ is generalized in Information Geometry using the following

Definition 1. 

The connections ∇ and ${\nabla }^{*}$ are called conjugate or dual with respect to the metric in $(M,g)$ if

$$X·g(X,Y)=g({\nabla }_{X}Y,Z)+g(X,{\nabla }_Y^{*}Z)\forall X,Y,Z\in \mathfrak{X}\left(M\right).$$

One can show [3] that the connection ${\nabla }^{*}$ dual to ∇ is unique. Moreover, in [3] it is proved that if $\nabla ,{\nabla }^{*}$ are dual and both torsion-free, then $R=0$ if and only if $R^{*}=0$. This motivate the following

Definition 2. 

A Riemannian manifold $(M,g,\nabla ,{\nabla }^{*})$ equipped with two connections which are dual and torsion-free is called dually flat when $R=R^{*}=0$, i.e. both connections are flat.

As a consequence, there exists on M two affine local coordinate systems $\theta$ and $\eta$ with the property that the vectors ${\partial }^i={\displaystyle \frac{\partial }{\partial {\eta }^i}}$ of the basis are ∇-parallel transported along the coordinate lines ${\eta }^i$ and the vectors ${\partial }_i={\displaystyle \frac{\partial }{\partial {\theta }_i}}$ are ${\nabla }^{*}$-parallel transported along the coordinate lines ${\theta }_i$. The coordinates $\theta$ and $\eta$ are called dual or bi-orthogonal because it holds that $g({\partial }_i,{\partial }^j)={\delta }_i^j$ where $\delta$ is the Kronecker symbol. Note that the two basis are orthogonal each other but the vectors of each base need not to be orthogonal. More in detail, if we consider the representation of the metric g in the two basis

$$g^{ij}=g({\partial }^i,{\partial }^j),g_{ij}=g({\partial }_i,{\partial }_j)$$

it holds that the two matrices are in the relation $g^{ij}{g}_{jk}={\delta }_k^i$. Finally, let $\theta =\theta \left(\eta \right)$ and $\eta =\eta \left(\theta \right)$ be the transition maps connecting the representation of a point $p\in M$ in the two coordinate charts. Note that they are one the inverse of the other. Using the orthogonality condition $g^{ij}{g}_{jk}={\delta }_k^i$ one can prove that their tangent maps coincide with the representation of the metric g in the two coordinate basis, that is

$$g^{ij}\left(\eta \right)={\displaystyle \frac{\partial {\theta }_i\left(\eta \right)}{\partial {\eta }^j}},g_{ij}\left(\theta \right)={\displaystyle \frac{\partial {\eta }^i\left(\theta \right)}{\partial {\theta }_j}}.$$

(2)

We now show that

Proposition 1. 

The change of coordinate $\eta =\eta \left(\theta \right)$ and $\theta =\theta \left(\eta \right)$ can be represented using the so-called potential scalar functions $\phi \left(\eta \right)$ and $\psi \left(\theta \right)$ in the sense that

$${\theta }_i\left(\eta \right)={\displaystyle \frac{\partial \phi \left(\eta \right)}{\partial {\eta }^i}},{\eta }^i\left(\theta \right)={\displaystyle \frac{\partial \psi \left(\theta \right)}{\partial {\theta }_i}},i=1,\dots ,n.$$

(3)

Proof. 

We prove the first relation of (3), the proof for the second follows the same pattern. We need to prove that the one form $\omega ={\theta }_i\left(\eta \right)d{\eta }^i$ is exact, that is there exists a scalar function $\phi \left(\eta \right)$ such that $d\phi =\omega$. A necessary (and sufficient condition if the manifold is simply connected) is that $d\omega =0$. Then we need to check that

$$d\omega =d\left({\theta }_i\left(\eta \right)d{\eta }^i\right)={\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}d{\eta }^j\wedge d{\eta }^i=\left({\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}-{\displaystyle \frac{\partial {\theta }_j}{\partial {\eta }^i}}\right)d{\eta }^j\otimes d{\eta }^i=0$$

that is

$${\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}-{\displaystyle \frac{\partial {\theta }_j}{\partial {\eta }^i}}=0\forall i,j=1,\dots n$$

Now, using (2), the above relation becomes $g^{ij}=g^{ji}$ which is true. □

Note that the relations (2) and (3) together imply that the potentials $\phi \left(\eta \right)$ and $\psi \left(\theta \right)$ are convex function. Indeed we have

$${\displaystyle \frac{{\partial }^2\phi \left(\eta \right)}{\partial {\eta }^j\partial {\eta }^i}}={\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}=g^{ij}\in Sy{m}^{+}\left(n\right)$$

i.e. that the Hessian matrix of $\phi$ is positive definite. Since the metric g is the Hessian of a function, the dually flat manifold is an example of an Hessian manifold. We do not pursue this line of investigation, see [9] for further details.

Note that the existence of a dual coordinate system induces stringent conditions on the metric g: indeed by deriving (2) and using Schwartz condition on mixed second order derivative we have that

$${\displaystyle \frac{\partial g^{ij}}{\partial {\eta }^k}}={\displaystyle \frac{\partial g^{ik}}{\partial {\eta }^j}}\mathrm{and}{\displaystyle \frac{\partial g_{ij}}{\partial {\theta }_k}}={\displaystyle \frac{\partial g_{ik}}{\partial {\theta }_j}}.$$

Another remarkable consequence is the following, which is a classical result of Legendre transform theory: since the change of coordinate maps are gradient maps, their inverse are also gradient maps and they are related by the Legendre condition

$$\phi \left(\eta \right)=\theta \left(\eta \right)·\eta -\psi \left(\theta \right(\eta \left)\right).$$

Indeed, using the one-form notation, we have

$$d\phi \left(\eta \right)=d\theta ·\eta +\theta ·d\eta -d\psi =d\theta ·\eta +\theta ·d\eta -\eta ·d\theta =\theta ·d\eta .$$

2.1. Mutually Dual Foliations of a Dually Flat Space

A remarkable fact of dually flat manifolds is that they can admit a couple of orthogonal foliations. Let us consider an n-dimensional dually flat manifold $(M,g,\nabla ,{\nabla }^{*})$ and let $\eta \left(p\right)=\eta$, $\theta \left(p\right)=\theta$, $p\in M$, be a couple of bi-orthogonal affine local coordinate systems. Given a k with $1<k<n$ we partition the coordinates in two subsets

$$\eta =({\eta }^A,{\eta }^B)=({\eta }^1,\dots ,{\eta }^k,{\eta }^{k+1},\dots ,{\eta }^n)\mathrm{and}\theta =({\theta }_A,{\theta }_B)=({\theta }_1,\dots ,{\theta }_k,{\theta }_{k+1},\dots ,{\theta }_n)$$

and we consider the coordinate projections ${\pi }_A\left(\eta \right)={\eta }^A$ and ${\pi }_B\left(\theta \right)={\theta }_B$. For every ${\overline{\eta }}^A$ and ${\overline{\theta }}_B$ we define the submanifolds of M

$$E\left({\overline{\eta }}^A\right)=\{p\in M:{\pi }_A\circ \eta \left(p\right)={\overline{\eta }}^A)\}Q\left({\overline{\theta }}_B\right)=\{p\in M:{\pi }_B\circ \theta \left(p\right)={\overline{\theta }}_B)\}$$

(4)

In [3] is is proved that since M is dually flat and the subsets $E\left({\overline{\eta }}^A\right)$ and $Q\left({\overline{\theta }}_B\right)$ are defined as affine subspaces of an affine coordinate system , they are totally geodesic submanifolds with respect to the ∇ and ${\nabla }^{*}$ connection respectively, that is it holds that a ∇-geodesic (respectively ${\nabla }^{*}$-geodesic) which is tangent to the submanifold at a point remains tangent to the submanifold. Moreover since their tangent spaces can be described as

$$T_{p}E\left({\overline{\eta }}_A\right)=<{\partial }^{k+1},\dots ,{\partial }^n>,T_{p}Q\left({\overline{\theta }}_B\right)=<{\partial }_1,\dots ,{\partial }_k>$$

the two submanifolds are mutually orthogonal by the condition $g({\partial }_i,{\partial }^j)={\delta }_i^j$ and complementary in the sense that their direct sum of subspaces gives the tangent space $T_{p}M$. See Figure 1 below. If we introduce the so called mixed coordinate system

$$m\left(p\right)=({\eta }^A\left(p\right),{\theta }_B\left(p\right))=({\pi }_A\left(\eta \left(p\right)\right),{\pi }_B\left(\theta \left(p\right)\right))$$

the metric g admits the block decomposition

$$g\left(m\left(p\right)\right)=g^{ab}d{\eta }^a\otimes d{\eta }^b+g_{cd}d{\theta }_c\otimes d{\theta }_d,a,b\in \{1,\dots ,k\},c,d\in \{k+1,\dots ,n\}.$$

So the complementary projections ${\pi }_A$ and ${\pi }_B$ define a couple of mutually dual foliation of M whose leaves are orthogonal submanifolds of constant dimension (respectively $\dim E\left({\overline{\eta }}^A\right)=n-k$, and $\dim Q\left({\overline{\theta }}_B\right)=k$ ) and totally geodesic with respect to the ∇ and ${\nabla }^{*}$ connections. This partition of the space is called k-cut of a dual coordinate system in [5]. The notion of mutually dual foliations has important applications on Information Geometry due to the fact that a version of the Pithagorean Theorem applies (see again [3]).

3. Exponential Family Manifold

We describe an example of dually flat manifold which occupies a central place in information geometry. We start with the definition of statistical model. Let $(X,\mathcal{B},d\mu )$ be a measure space, where the set X can be numerable or not, $\mathcal{B}$ is a $\sigma -$algebra over X and $d\mu$ is a measure on X. The infinite dimensional space of all the positive probability densities over X is

$$\mathcal{P}\left(X\right)=\{p\left(x\right)\ge 0a.e.\mathrm{on}X,\mathrm{and}{\int }_{X}p\left(x\right)d\mu =1\}.$$

Given an open convex set $U\subset {\mathbb{R}}^n$ with coordinates $\theta =({\theta }_1,\dots ,{\theta }_n)$ we consider the subset of $\mathcal{P}\left(X\right)$ of probability densities parameterized by the parameters $\theta \in U$

$$S=\{p_{\theta }=p(x,\theta ),\theta \in U\}\subset \mathcal{P}\left(X\right).$$

The set S has the structure of a n-dimensional sub-manifold of $\mathcal{P}\left(X\right)$ if: 1) the map $\theta \mapsto p_{\theta }$ is injective and 2) the n base functions $p_i=\partial p(x,\theta )/\partial {\theta }_i$ are linearly independent as functions over X. In this case we call S a n-dimensional statistical model. Usually, linear independence condition is checked using the equivalent score function basis functions $l_i=\partial l/\partial {\theta }_i$, where $l=lnp$ is the likelihood of p.

An exponential family is the n-dimensional statistical manifold

$$\mathcal{E}=\{p(x,\theta )=e^{\theta ·h\left(x\right)+k\left(x\right)-\psi \left(\theta \right)},\theta \in U\}\subset \mathcal{P}\left(X\right)$$

(5)

where $h:X\to {\mathbb{R}}^n$ is a map of maximum rank n whose components $h_i$, $i=1,\dots ,n$ are called feature functions or observables, k is a positive function such that $e^{k\left(x\right)}d\mu$ is a probability density over X and the function $\psi \left(\theta \right)$

$$\psi \left(\theta \right)=ln{\int }_X{e}^{\theta ·h\left(x\right)+k\left(x\right)}d\mu$$

is the cumulant or moment generating function, called free energy in statistical mechanics. A direct computation shows that the averages of $h_i$ can be written as

$${\eta }_i={\mathbb{E}}_{p_{\theta }}\left(h_i\right)={\displaystyle \frac{\partial \psi \left(\theta \right)}{\partial {\theta }_i}},i=1,\dots ,n$$

(6)

and that the covariances are

$$cov(h_i,h_j)={\mathbb{E}}_{p_{\theta }}\left(h_i{h}_j\right)-{\mathbb{E}}_{p_{\theta }}\left(h_i\right){\mathbb{E}}_{p_{\theta }}\left(h_j\right)={\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_i\partial {\theta }_j}}.$$

Using the score vectors $l_i=h_i-\partial \psi /\partial {\theta }_i$ associated to the exponential model one can show that the Hessian of the cumulant function coincides with the Fisher Riemannian metric $g_{ij}$ [3] that is

$$g_{ij}\left(\theta \right)={\mathbb{E}}_{p_{\theta }}\left(l_i{l}_j\right)={\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_i\partial {\theta }_j}}\left(\theta \right),$$

which is positive definite by the maximum rank hypothesis on h. So $(\mathcal{E},g)$ is a Riemannian manifold. To show that $\mathcal{E}$ is dually flat manifold one needs to introduce Amari $\alpha$-connection with Christoffel symbols

$${\Gamma }_{ij,k}^{\left(\alpha \right)}=\mathbb{E}\left[({\partial }_i{\partial }_{j}l+{\displaystyle \frac{1-\alpha }{2}}{\partial }_{i}l{\partial }_{j}l){\partial }_{k}l\right],\alpha \in [0,1]$$

where $l=lnp(x,\theta )$. By direct computation one shows that ${\Gamma }^{\left(1\right)}$ (resp. ${\Gamma }^{(-1)}$) vanish in $\theta$ (resp. $\eta )$ coordinates, so $\theta$ and $\eta$ are global affine charts and the curvature $R^{\left(1\right)}$ and $R^{(-1)}$ both vanish, making $\mathcal{E}$ dually flat. Note that a connection can be locally flat (zero curvature) without admitting a global affine coordinate system. For an exponential family manifold the Christoffel symbols vanish on the entire domains of the $\theta$ and $\eta$ coordinates, which are convex open sets, so the flat structure is global. For more details see [3].

4. Controlled Evolution in a Dually Flat Manifold

In this Section we show how the dual coordinate system $\eta ,\theta$ can be used to describe the controlled evolution of a point on a dually flat manifold. It is well known that the statistical description of a thermodynamic system is given by an exponential family (5) whose feature functions $h\left(x\right)\in {\mathbb{R}}^n$ are the physical observables of interest for the macroscopic description of the system given in terms of their average values $\eta \in {\mathbb{R}}^n$. The associated natural parameters $\theta =\theta \left(\eta \right)$ are interpreted as generalized inverse temperatures. This is a generalization of the fact that if $h\left(x\right)\in \mathbb{R}$ represents the energy of the system in the microscopic phase-space configuration $x\in X$, its average $\eta$ is the internal energy of the system in contact with an heat bath at the inverse temperature $\theta \left(\eta \right)$. The heat bath allows one to control the (inverse) temperature $\theta$ of the system. This brief review of equilibrium statistical mechanics terminology motivates the name of (thermal) control or input coordinates for $\theta \in {\mathbb{R}}^n$ and of effect or output coordinate for $\eta \in {\mathbb{R}}^n$.

The equilibrium statistical mechanics formalism is an example of an exponential family manifold which is an instance of a dually flat manifold; we will be interested in the following two problems formulated for a dually flat n-dimensional Riemannian manifold $(M,g)$ equipped with a dual coordinate systems $(\eta ,\theta )$:

1) forward of direct problem: given a path in control variables $\theta \left(t\right)\in {\mathbb{R}}^n$ to determine the corresponding evolution $\eta \left(t\right)\in {\mathbb{R}}^n$ in effect space;

2) backward or inverse problem: given an observed evolution $\eta \left(t\right)$ in effect space, to determine the corresponding control or cause path $\theta \left(t\right)$.

Since $\eta$ and $\theta$ are local affine coordinates of a dually flat manifold, the above problems have a solution using (3)

$$\mathrm{direct}\mathrm{problem}:\mathrm{given}\theta \left(t\right),\mathrm{the}\mathrm{evolution}\mathrm{is}\eta \left(t\right)=\nabla \psi \left(\theta \right(t\left)\right)$$

(7)

and

$$\mathrm{inverse}\mathrm{problem}:\mathrm{given}\eta \left(t\right),\mathrm{the}\mathrm{evolution}\mathrm{is}\theta \left(t\right)=\nabla \phi \left(\eta \right(t\left)\right).$$

(8)

In the following we will consider two important classes of direct and inverse problems.

Direct Problem: System in a Heat Bath

In [1] the following instance of a direct problem for a control path referred to a k-cut of a mutually dual foliation (see Section 2.1) is considered: let $t\mapsto ({\theta }_A\left(t\right),{\overline{\theta }}_B$) be given, that is the system is contained in a heath bath which realize the constraint ${\theta }_B\left(t\right)={\overline{\theta }}_B$. Using (7) we have, adopting for simplicity the shorthand block notation ${\psi }_{AB}={\partial }_{AB}\psi$

$$\left\{\begin{array}{c}{\dot{\eta }}^A\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }_A\psi ({\theta }_A\left(t\right),{\overline{\theta }}_B)\right)={\partial }_{AA}\psi {\dot{\theta }}_A={\psi }_{AA}{\dot{\theta }}_A,\hfill \\ \\ {\dot{\eta }}_B\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }_B\psi ({\theta }_A\left(t\right),{\overline{\theta }}_B)\right)={\psi }_{BA}{\dot{\theta }}_A\hfill \end{array}\right.$$

and since ${\psi }_{AA}\in Mat\left(k\right)$ is invertible, we can write ${\dot{\theta }}_A={\psi }_{AA}^{-1}{\dot{\eta }}_A$ and obtain the result

$${\dot{\eta }}^B={\psi }_{BA}{\psi }_{AA}^{-1}{\dot{\eta }}_A={\Gamma }_{BA}({\theta }_A\left(t\right),{\overline{\theta }}_B){\dot{\eta }}^A,{\dot{\eta }}^A={\psi }_{AA}{\dot{\theta }}_A.$$

(9)

So, if the system is contained in a B-type heath bath ${\theta }_B\left(t\right)={\overline{\theta }}_B$, then the evolution in the ${\eta }^B$ effect variables can be reconstructed from the evolution of the ${\eta }^A$ variables using (9)₁.

This result has an interesting geometrical interpretation: if we consider the local fibration $\eta \circ {\pi }_A\left(p\right)={\eta }^A$, with fibres $E\left({\overline{\eta }}^A\right)$ see (4), the orthogonal submanifold $Q\left({\overline{\theta }}_B\right)$ defines an orthogonal complement $H(\eta \circ {\pi }_A)$ to the tangent space to the fibres $V(\eta \circ {\pi }_A)=kerT(\eta \circ {\pi }_A)$ and hence $Q\left({\overline{\theta }}_B\right)$ is an integrable orthogonal Ehresmann connection, i.e a section of the fibre map $\eta \circ {\pi }_A$ and the ${\Gamma }_{BA}$ are the coefficients of the Ehresmann connection, see e.g. [11,12]. Therefore (9) can be seen as the horizontal lift of the path ${\eta }_A\left(t\right)$ in base space

$${\dot{\eta }}^B=hor\left({\eta }^A\left(t\right)\right)={\Gamma }_{BA}\left(\theta \left(t\right)\right){\dot{\eta }}^A\left(t\right)\otimes {\partial }^B$$

where $hor:{\mathbb{R}}^k\to H(\eta \circ {\pi }_A)=TQ\left({\theta }_B\right)\subset T_{p}M$ is the horizontal lift operator.

In the case of an exponential family manifold, the connection coefficients ${\Gamma }_{BA}$ have an important interpretation in statistical theory

$${\Gamma }_{BA}\left(\theta \right)={\psi }_{BA}{\psi }_{AA}^{-1}=cov(h_A,h_B)cov{(h_A,h_A)}^{-1}$$

where $cov(h_A,h_B)$ is the matrices of the covariances between $h_A$ and $h_B$ observables and $cov(h_A,h_A)$ is the square matrix of the variance and covariance between $h_A$ type observables. This shows that when the ${\theta }_B$ parameter are kept constant, the inference on the ${\eta }^B$ variables given the evolution of the ${\eta }^A$ variables is the linear predictor of the B random variables in terms of the A random variables.

The dual of the above system evolution is given by the following:

Inverse Problem: System Insensitive to the Control in the A Variables

Given an evolution in the effect space of the type $\eta \left(t\right)=({\overline{\eta }}^A,{\eta }^B\left(t\right))$, to determine the corresponding time evolution in the cause parameter $\theta \left(t\right)$. Using the same argument as above we have from (7)

$$\left\{\begin{array}{c}{\dot{\eta }}^A\left(t\right)={\displaystyle \frac{d}{dt}}({\partial }_A\psi ({\theta }_A\left(t\right),{\theta }_B\left(t\right))={\psi }_{AA}{\dot{\theta }}_A+{\psi }_{AB}{\dot{\theta }}_B=0,\hfill \\ \\ {\dot{\eta }}^B\left(t\right)={\displaystyle \frac{d}{dt}}({\partial }_B\psi ({\theta }_A\left(t\right),{\theta }_B\left(t\right))={\psi }_{BA}{\dot{\theta }}_A+{\psi }_{BB}{\dot{\theta }}_B.\hfill \end{array}\right.$$

Therefore the constraint in effect space induces the following constraint in control space

$${\dot{\theta }}_A=-{\psi }_{AA}^{-1}{\psi }_{AB}{\dot{\theta }}_B=-{\Gamma }_{AB}^T({\theta }_A\left(t\right),{\theta }_B\left(t\right)){\dot{\theta }}_B$$

(10)

where ${\Gamma }_{AB}^T$ is the transpose of the matrix ${\Gamma }_{BA}$. Note that if ${\dot{\theta }}_B\left(t\right)\ne 0$ the above equation (10) can be given the simpler form of an ODE for the function ${\theta }_A={\theta }_A\left({\theta }_B\right)$ as

$${\displaystyle \frac{d{\theta }_A}{d{\theta }_B}}=-{\Gamma }_{AB}^T({\theta }_A,{\theta }_B).$$

For the sake of completeness we derive the above result (10) using the inverse problem formulation in (8). Adopting the shorthand notation ${\partial }^{AB}\phi ={\phi }^{AB}$ we have

$$\left\{\begin{array}{c}{\dot{\theta }}_A\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }^A\phi ({\overline{\eta }}^A\left(t\right),{\eta }^B)\right)={\partial }^{AB}\phi {\dot{\eta }}^B={\phi }^{AB}{\dot{\eta }}^B,\hfill \\ \\ {\dot{\theta }}_B\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }^B\phi ({\overline{\eta }}^A\left(t\right),{\eta }_B)\right)={\phi }^{BB}{\dot{\eta }}^B\hfill \end{array}\right.$$

and since ${\phi }^{BB}\in Mat(n-k)$ is invertible, we have that

$${\dot{\theta }}_A={\phi }^{AB}{\left({\phi }^{BB}\right)}^{-1}{\dot{\theta }}_B={\Gamma }^{AB}{\dot{\theta }}_B$$

which coincides with the above result (10) because

$${\Gamma }^{AB}={\phi }^{AB}{\left({\phi }^{BB}\right)}^{-1}=-{\psi }_{AA}^{-1}{\psi }_{AB}=-{\Gamma }_{AB}^T.$$

This last equality can be easily checked by expanding in block representation the identity

$$g^{ij}{g}_{jk}={\phi }^{ij}{\psi }_{jk}={\delta }_k^i.$$

Example: Direct Problem for the Gaussian Exponential Manifold

Before turning to the mixed control problem we give a simple example of the theory developed so far. It is well known that the gaussian probability density is the maximal entropy density under the constraint of fixed value for the first two moments. Therefore, we can consider for $x\in X=\mathbb{R}$ the feature functions $h_1\left(x\right)=x$ and $h_2\left(x\right)=x^2$. The probability density defines an exponential family of the form

$$p({\theta }_1,{\theta }_2,x)=e^{{\theta }_{1}x+{\theta }_2{x}^2-\psi ({\theta }_1,{\theta }_2)}$$

where

$$\psi ({\theta }_1,{\theta }_2)=ln{\int }_{\mathbb{R}}{e}^{{\theta }_{1}x+{\theta }_2{x}^2}dx=-{\displaystyle \frac{{\theta }_1^2}{4{\theta }_2}}+{\displaystyle \frac{1}{2}}ln(-{\theta }_2)+{\displaystyle \frac{1}{2}}ln\pi ,{\theta }_2<0.$$

We consider the following forward control problem: to determine the evolution of ${\eta }^B={\eta }^2={\mathbb{E}}_{p_{\theta }}\left[x^2\right]$ given the evolution ${\eta }^A={\eta }^1={\mathbb{E}}_{p_{\theta }}\left[x\right]$ under the control path $({\theta }_A\left(t\right),{\overline{\theta }}_B)=({\theta }_1\left(t\right),{\overline{\theta }}_2)$. This can be done using (9) by computing the Ehresmann connection coefficient

$${\Gamma }_{BA}={\Gamma }_{21}={\displaystyle \frac{{\psi }_{12}}{{\psi }_{11}}}={\displaystyle \frac{{\psi }_{{\theta }_1{\theta }_2}}{{\psi }_{{\theta }_1{\theta }_1}}}=-{\displaystyle \frac{{\theta }_1}{{\theta }_2}}=-{\displaystyle \frac{{\theta }_1\left(t\right)}{{\overline{\theta }}_2}}$$

therefore

$${\dot{\eta }}^2\left(t\right)={\Gamma }_{21}\left(t\right){\dot{\eta }}^1\left(t\right)=-{\displaystyle \frac{{\theta }_1\left(t\right)}{{\overline{\theta }}_2}}{\dot{\eta }}^1\left(t\right)$$

(11)

which can directly integrated

$${\eta }^2\left(t\right)=-{\displaystyle \frac{1}{{\overline{\theta }}_2}}\int {\theta }_1\left(t\right){\dot{\eta }}^1\left(t\right)dt+\mathrm{const}.$$

If ${\dot{\eta }}^1\left(t\right)\ne 0$, ${\eta }^1\left(t\right)$ can be locally inverted in $t=t\left({\eta }^1\right)$ and we can express the horizontal lift (11) in the form

$${\displaystyle \frac{d{\eta }^2}{d{\eta }^1}}={\Gamma }_{21}\left(t\right)={\Gamma }_{21}\left(t\left({\eta }^1\right)\right)$$

so that

$${\eta }^2=\int {\Gamma }_{21}\left(t\left({\eta }^1\right)\right)d{\eta }^1=-{\displaystyle \frac{1}{{\overline{\theta }}_2}}\int {\theta }_1\left(t\left({\eta }^1\right)\right)d{\eta }^1+\mathrm{const}.$$

As an example, consider the evolution ${\eta }^1\left(t\right)=t$, with inverse $t\left({\eta }^1\right)={\eta }^1$. We want to determine the control path ${\theta }_1\left(t\left({\eta }^1\right)\right)={\theta }_1\left({\eta }^1\right)$ so that the variance is a given function $var\left(h_1\right)={\eta }^2-{\left({\eta }^1\right)}^2=f\left({\eta }^1\right)$. We have the functional equation

$$f\left({\eta }^1\right)={\eta }^2-{\left({\eta }^1\right)}^2=-{\displaystyle \frac{1}{{\overline{\theta }}_2}}\int {\theta }_1\left({\eta }^1\right)d{\eta }^1+\mathrm{const}-{\left({\eta }^1\right)}^2$$

which gives by derivation

$${\theta }_1\left({\eta }^1\right)=(-{\overline{\theta }}_2)(2{\eta }^1+f^{\prime }\left({\eta }^1\right)).$$

5. Mixed Control Problem and Inferential Scattering

The notion of Inferential scattering was exposed in a couple of papers by E.T. Jaynes [1,2]. Jaynes’ original exposition refers to equilibrium statistical mechanics. In this setting, the results of the above Section describe the inference on the evolution of the average value of a group of random variables (B) given the evolution of the average value of another group (A) of random variables under the hypothesis that the random variables are described by an exponential family and a constraint is in force. It can be seen as an inference result.

To generalize the notion of inferential scattering we need to introduce the notion of $(k,l)$-cut in a n-dimensional dual coordinate system of a dually flat manifold: let $(l,k)$ with $1<l<k<n$ and consider the splitting of the coordinates

$$\eta =({\eta }^a,{\eta }^c,{\eta }^B),\theta =({\theta }_a,{\theta }_c,{\theta }_B),a=1,\dots ,l,c=l+1,\dots ,k,B=k+1,\dots ,n.$$

In practice, we split the A block variables in $A=(a,c)$. Now we can consider the following instance of mixed control problem

Mixed Control Problem for a System in a B-Type Heat Bath and Insensitive to the Control in the c Variables

This is a combination of the two above cases. We want to determine the evolution in the ${\eta }^B$ variables under the controlled evolution ${\theta }_a\left(t\right)$ knowing that we have the constraint ${\eta }^c\left(t\right)={\overline{\eta }}^c$ in the effect space and ${\theta }_B\left(t\right)={\overline{\theta }}_B$ in cause space. Reasoning as above we consider the system of evolution equations

$$\left\{\begin{array}{c}{\dot{\eta }}^a={\displaystyle \frac{d}{dt}}({\partial }_a\psi ({\theta }_a\left(t\right),{\theta }_c\left(t\right),{\overline{\theta }}_B)={\psi }_{aa}{\dot{\theta }}_a+{\psi }_{ac}{\dot{\theta }}_c,\hfill \\ {\dot{\eta }}^c={\displaystyle \frac{d}{dt}}({\partial }_c\psi ({\theta }_a\left(t\right),{\theta }_c\left(t\right){\overline{\theta }}_B)={\psi }_{ca}{\dot{\theta }}_a+{\psi }_{cc}{\dot{\theta }}_c=0,\hfill \\ {\dot{\eta }}^B={\displaystyle \frac{d}{dt}}({\partial }_B\psi ({\theta }_a\left(t\right),{\theta }_c\left(t\right),{\overline{\theta }}_B)={\psi }_{Ba}{\dot{\theta }}_a+{\psi }_{Bc}{\dot{\theta }}_c.\hfill \end{array}\right.$$

From the second equation we get

$${\dot{\theta }}_c=-{\psi }_{cc}^{-1}{\psi }_{ca}{\dot{\theta }}_a$$

(12)

and we substitute ${\dot{\theta }}_c$ in the first and third equation to obtain

$$\left\{\begin{array}{c}{\dot{\eta }}^a={\psi }_{aa}{\dot{\theta }}_a-{\psi }_{ac}{\psi }_{cc}^{-1}{\psi }_{ca}{\dot{\theta }}_a={\psi }_{aa}(\mathbb{I}-{\psi }_{aa}^{-1}{\psi }_{ac}{\psi }_{cc}^{-1}{\psi }_{ca}){\dot{\theta }}_a={\psi }_{aa}(\mathbb{I}-R(a,c)){\dot{\theta }}_a\hfill \\ {\dot{\eta }}^B={\psi }_{Ba}{\dot{\theta }}_a-{\psi }_{Bc}{\psi }_{cc}^{-1}{\psi }_{ca}{\dot{\theta }}_a=({\psi }_{Ba}-{\psi }_{Bc}{\psi }_{cc}^{-1}{\psi }_{ca}){\dot{\theta }}_a\hfill \end{array}\right.$$

(13)

where $\mathbb{I}$ is the identity matrix and we have introduced the correlation coeffcient matrix

$$R(a,c)={\psi }_{aa}^{-1}{\psi }_{ac}{\psi }_{cc}^{-1}{\psi }_{ca}.$$

(14)

The last equality in (13)₁ can be inverted

$${\dot{\theta }}_a={(\mathbb{I}-R(a,c))}^{-1}{\psi }_{aa}^{-1}{\dot{\eta }}^a$$

and substituted in (13)₂ to finally obtain

$${\dot{\eta }}^B=({\psi }_{Ba}-{\psi }_{Bc}{\psi }_{cc}^{-1}{\psi }_{ca}){(\mathbb{I}-R(a,c))}^{-1}{\psi }_{aa}^{-1}{\dot{\eta }}^a={\mathcal{G}}_{Ba}{\dot{\eta }}^a.$$

(15)

This is our inference result on the evolution of the ${\eta }^B$ variables given the mixed (forward and backward) control. Note that ${\mathcal{G}}_{Ba}={\mathcal{G}}_{Ba}({\theta }_a\left(t\right),{\theta }_c\left(t\right),{\overline{\theta }}_B))$ but the evolution in the controlled variables ${\theta }_a,{\theta }_c$ is subject to the equation (12) which enforces the constraint in effect space ${\eta }^c={\overline{\eta }}^c$.

5.1. The Inferential Scattering Formula

Since $R(a,c)$ in (14) is the analogous of the correlation coefficient of two random variables

$$\rho (A,B)={\displaystyle \frac{cov{(A,B)}^2}{var\left(A\right)var\left(B\right)}},0<\rho (A,B)<1$$

it reasonable to assume that the matrix $R(a,c)$ has spectral radius less than one. Under this assumption, the following Taylor expansion hold

$${(\mathbb{I}-R(a,c))}^{-1}=\mathbb{I}+R(a,c)+R{(a,c)}^2+R{(a,c)}^3+\dots$$

(16)

If we substitute (16) in (15) we readily obtain the following expansion

$${\dot{\eta }}^B=({\Gamma }_{Ba}-{\Gamma }_{Bc}{\Gamma }_{ca}+{\Gamma }_{Ba}{\Gamma }_{ac}{\Gamma }_{ca}-{\Gamma }_{Bc}{\Gamma }_{ca}{\Gamma }_{ac}{\Gamma }_{ca}+\dots ){\dot{\eta }}^a$$

(17)

where we have set

$${\Gamma }_{Ba}={\psi }_{Ba}{\psi }_{aa}^{-1},{\Gamma }_{Bc}={\psi }_{Bc}{\psi }_{cc}^{-1},{\Gamma }_{ca}={\psi }_{ca}{\psi }_{aa}^{-1},{\Gamma }_{ac}={\psi }_{ac}{\psi }_{cc}^{-1}.$$

The above expansion formula (17) has the following interpretation, see [1]: the term ${\Gamma }_{Ba}d{\eta }^a$ is the inferred (infinitesimal) variation of the B observable induced by a variation $d{\eta }^a$ of the a observables. The term ${\Gamma }_{Bc}{\Gamma }_{ca}d{\eta }^a$ is the inferred variation in B as induced by a variation $d{\eta }_c={\Gamma }_{ca}d{\eta }^a$ of the c observables which is in turn induced by the variation $d{\eta }^a$. The other terms in (17) have a similar interpretation. This result has been called Inferential Scattering formula [2] by analogy with the Born series expansion of the operator which describes the multiple scattering phenomenon [10]: a wave can reach a point B in space directly from source a (${\Gamma }_{Ba}$ term) or after having been reflected by an obstacle in c (${\Gamma }_{Bc}{\Gamma }_{ca}$ term) or after having been reflected back in a from c and then arriving at B (${\Gamma }_{Ba}{\Gamma }_{ac}{\Gamma }_{ca}$ term). See diagram (18) below. Note that the first term in (17) ${\Gamma }_{Ba}$ coincides with the one derived before in (9) in case of forward control. Of course the size of the higher order correction terms after ${\Gamma }_{Ba}$ depends on the norm of the correlation matrix $R(a,c)$.

(18)

5.2. A Mixed Control Example: Neural Network of Two Neurons

The following toy model (see e.g. [5]) describes the neural activity of a set of two neurons using two discrete random variables which can take the value 1 (active) or 0 (silent). The state space of the network is $X=\left\{\right(0,0),(0,1),(1,0),(1,1\left)\right\}$. Given $x=(x_1,x_2)\in X$, let $p\left(x\right)$ be the probability that the network is in the state x. As observables we consider $h_i\left(x\right)=x_i$, $i=1,2$ which describe the state of the two neurons and $h_3\left(x\right)=x_1{x}_2$. The exponential manifold is comprised of discrete probability distributions of the form

$$p\left(x\right)=e^{\theta ·h-\psi \left(\theta \right)}=e^{{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_1{x}_2-\psi \left(\theta \right)}$$

where

$$\psi \left(\theta \right)=ln\left(\sum _{x\in X}{e}^{{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_1{x}_2}\right)=ln\left(1+e_1^{\theta }+e_2^{\theta }+e^{{\theta }_1+{\theta }_2+{\theta }_3}\right)=lnZ\left(\theta \right)$$

The associated effect space coordinates

$${\eta }^i={\mathbb{E}}_{p_{\theta }}\left[x_i\right]={\displaystyle \frac{\partial \psi }{\partial {\theta }_i}},i=1,2{\eta }^3={\mathbb{E}}_{p_{\theta }}\left[x_1{x}_2\right]={\displaystyle \frac{\partial \psi }{\partial {\theta }_3}}$$

are called firing rate of neurons $i=1,2$ and joint firing rate. A direct computation shows that

$${\eta }^i={\displaystyle \frac{e^{{\theta }_i}+e^{{\theta }_1+{\theta }_2+{\theta }_3}}{Z\left(\theta \right)}}=\mathrm{Prob}(x_i=1),i=1,2,{\eta }^3={\displaystyle \frac{e^{{\theta }_1+{\theta }_2+{\theta }_3}}{Z\left(\theta \right)}}=\mathrm{Prob}(x_1{x}_2=1).$$

The Riemannian metric matrix $g\in Mat\left(3\right)$ can also be computed directly as

$$g_{ij}\left(\theta \right)={\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_i\partial {\theta }_j}}.$$

We investigate the mixed control problem (see Section 5) for this system. We consider the $(2,1)$-cut in the 3-dimensional dual coordinate system defined by the following identification

$$({\eta }^1,{\eta }^2,{\eta }^3)=({\eta }^a,{\eta }^c,{\eta }^B),({\theta }_1,{\theta }_2,{\theta }_3)=({\theta }_a,{\theta }_c,{\theta }_B).$$

We want to determine the evolution in the ${\eta }^B={\eta }^3$ variable (joint firing rate of neuron 1 and 2) in terms of the evolution of ${\eta }^a={\eta }^1$ (firing rate of neuron 1) under the control ${\theta }_B\left(t\right)={\theta }_3\left(t\right)$ knowing that we have the constraint ${\eta }^c\left(t\right)={\eta }^2\left(t\right)={\overline{\eta }}^2$ (firing rate of neuron 2 is insensitive to the control) in the effect space. To this we can use the perturbation expansion (17)

$$d{\eta }^3=({\Gamma }_{31}-{\Gamma }_{32}{\Gamma }_{21}+{\Gamma }_{31}{\Gamma }_{12}{\Gamma }_{21}-{\Gamma }_{32}{\Gamma }_{21}{\Gamma }_{12}{\Gamma }_{21}+\dots )d{\eta }^1$$

where

$${\Gamma }_{31}={\displaystyle \frac{{\psi }_{31}}{{\psi }_{11}}},{\Gamma }_{32}={\displaystyle \frac{{\psi }_{32}}{{\psi }_{22}}},{\Gamma }_{21}={\displaystyle \frac{{\psi }_{21}}{{\psi }_{11}}}{\Gamma }_{12}={\displaystyle \frac{{\psi }_{12}}{{\psi }_{22}}}$$

and

$$R(1,2)={\Gamma }_{12}{\Gamma }_{21}={\displaystyle \frac{{\psi }_{12}{\psi }_{21}}{{\psi }_{11}{\psi }_{22}}}={\displaystyle \frac{{\psi }_{12}^2}{{\psi }_{11}{\psi }_{22}}}={\displaystyle \frac{cov{(x_1,x_2)}^2}{var\left(x_1\right)var\left(x_2\right)}}={\rho }^2(x_1,x_2)$$

where $\rho (x_1,x_2)\in [0,1]$ is Pearson’s correlation coefficient which is a measure of the strength of the linear correlation (normalized covariance) between the two variables. So the perturbation parameter in the Born series expansion (16) is the correlation coefficient. The explicit expression (setting ${\theta }_1=x$, ${\theta }_2=y$ and ${\theta }_3=z$ for simplicity) is

$$R(1,2)(x,y,z)=({\displaystyle \frac{1}{1+e^x}}-{\displaystyle \frac{1}{e^{x+z}}})({\displaystyle \frac{1}{1+e^y}}-{\displaystyle \frac{1}{e^{y+z}}}).$$

Note that the two control variables $(x,y)$ are not completely free because relation (12) which enforces the constraint ${\eta }^3={\eta }^c={\overline{\eta }}^c$ holds. In our notation after a straightforward computation (12) reads

$$\dot{y}=-{\displaystyle \frac{{\psi }_{12}}{{\psi }_{22}}}\dot{x}=\left({\displaystyle \frac{1}{1+e^x}}-{\displaystyle \frac{1}{1+e^{x+\overline{z}}}}\right)\dot{x}=-{\Gamma }_{12}(x,\overline{z})\dot{x}$$

where we have set $z={\theta }_3={\overline{\theta }}_3=\overline{z}$. If $\dot{x}\ne 0$ the above equation can be readily integrated

$${\displaystyle \frac{dy}{dx}}={\Gamma }_{12}(x,\overline{z})\mathrm{hence}y\left(x\right)=\int {\Gamma }_{12}(x,\overline{z})dx+\mathrm{const}.$$

to give

$$y(x,\overline{z})=2[artanh(1+2e^{x+\overline{z}})-artanh(1+2e^x)].$$

Therefore, in the mixed control case, the correlation coefficient $R(1,2)$ is a function of ${\theta }_1=x$ only, of the form

$$R_{\overline{z}}(1,2)\left(x\right)=R(1,2)(x,y(x,\overline{z}),\overline{z}).$$

A plot of $R(1,2)$ for $\overline{z}={\overline{\theta }}_3=5$ is in Figure 2. The figure helps to visualize the size of the perturbation expansion parameter $R(1,2)$. This simple example is only suggestive of the possible applications of the notion of Inferential Scattering.

6. Discussion

The primary aim of this paper was to embed the notion of Inferential Scattering as exposed b E.T. Jaynes for a thermodynamical system into the language and the concepts of Information Geometry. The key geometrical object turns out to be a dually flat manifold with its coupled bi-orthogonal coordinate systems $\eta$ and $\theta$. These coordinates have a simple and appealing interpretation in the statistical mechanics framework as (thermal) control and effect space coordinates.

We have also shown that the forward and backward control problems can be seen as a horizontal lift (parallel transport) with respect to an integrable orthogonal Ehresmann connection. In this way the controlled evolution of the system in effect space is globally defined providing a generalization of the original formulation for infinitesimal displacements contained in the Jaynes papers. We stress that this notion of controlled evolution has a potentially wide application range being a form of inference, hence applicable to every statistical system with insufficient information. The geometrical analysis presented in this paper could be further investigated and we plan to do it in a subsequent paper: for example, it remains to be seen its formulation in the larger setting of Hessian manifolds and the possible developments using a weaker setting for the Legendre transform. We hope that the material presented here can renew interest in the scientific community for the original contribution of E.T. Jaynes and its potential applications.