Fitting Copulas with Maximal Entropy

1. Formulation of the task

1.1. Motivation

Copulas are a successful tool for the description of any dependence of continuous random variables. Based on empirical data, we may try to find a copula best describing the underlying distribution.

The standard approach to fitting the parametric copula classes is widely used across various copula applications. The hidden assumption behind this is the knowledge of the dependence model for which we try to estimate the parameters. However, there are many applications where this approach is not desirable because of the lack of knowledge, insufficient sample size, or a high dimension of the task. Then we want to find a model based on general copulas. When there is no other information than the sample, the empirical or discrete copulas and their continuous extensions can be used. Because of the incomplete information of copula values in between the known values, the maximum entropy principle can also be applied.

We study the problem of finding a joint probability distribution from partial information, when we are given

(I1)

the marginals,

(I2)

the values of the joint cumulative distribution function (joint cdf) at finitely many points.

This refers to the situation when we have (or assume to have) complete information about the marginals, but we do not know their dependence. The sample is discrete and it can possibly be discretised to a coarser scale. Thus we know the desired values of the copula only on a 2D rectangular grid of points resulting from the discretization. This task has a standard solution (a checkerboard copula, see below). However, some of these constraints may be missing for various reasons. In particular, we may have too few elements of the sample (possibly none) in some 2D intervals, so we do not consider it a reliable support for modelling the whole copula. We investigate this case.

1.2. Criteria of Optimality

Suppose that we have two continuous random variables, $X,Y$, with known cumulative distribution functions, $F_X,F_Y$. They need not be independent and their dependence can be specified, e.g., by their joint cdf, $F_{X,Y}$. According to Sklar’s theorem (see [1,2]), it can be determined also by the respective copula, $C_{X,Y}$. This is the joint cumulative distribution function of transformed random variables $F_X\left(X\right)$, $F_Y\left(Y\right)$, which have the uniform distribution on the interval $\left]0,1\right[$,1

$$C_{X,Y}(x,y)=F_{F_X\left(X\right),F_Y\left(Y\right)}(x,y)=F_{X,Y}(s,t),$$

where $s,t$ are such that $F_X\left(s\right)=x$, $F_Y\left(t\right)=y$. The copula $C_{X,Y}$ gives “pure” information of the dependence of $X,Y$, independently of the marginal distributions. It allows the expression of an arbitrary dependence between two continuous random variables.2

Definition 1. 

If there is a function $c_{X,Y}$ such that the copula can be expressed as its integral,

$$C_{X,Y}(\overline{x},\overline{y})={\int }_0^{\overline{x}}{\int }_0^{\overline{y}}{c}_{X,Y}(x,y)\mathrm{d}x\mathrm{d}y,$$

it is called a density of the copula $C_{X,Y}$. The differential entropy of the copula $C_{X,Y}$ is the (Shannon) entropy [3] of its density, given by

$$Ent\left(c_{X,Y}\right)={\int }_0^1{\int }_0^1\eta \left(c_{X,Y}(x,y)\right)\mathrm{d}x\mathrm{d}y,$$

(1)

where

$$\eta \left(w\right):=\left\{\begin{array}{cc}-wlnw\hfill & \mathrm{if}w>0,\hfill \\ 0\hfill & \mathrm{if}w=0.\hfill \end{array}\right.$$

Remark 1. 

The differential entropy of a copula does not exist if the copula is not absolutely continuous, i.e., it does not have a density. Our proposed solution avoids this problem.

1.3. Related Work and History of the Problem

One of the earliest pioneers in studying multivariate distributions was Claude E. Shannon, who introduced his groundbreaking work on information theory in [3]. Later, in 1959, Abe Sklar introduced copulas in his seminal work, which laid the foundation for the study of dependency structures in multivariate distributions [2].

The principle of maximum entropy in the choice of a distribution was well defended by Jaynes [4]. Several approaches to this aim have been proposed.

The study of copulas in relation to entropy began to gain attention around 2010, with foundational work by Pougaza et al. [5,6]. They give a nice overview and motivation of the maximum entropy principle.3 They maximize the differential entropy of the original joint cdf, while we propose to maximize the differential entropy of the corresponding copula.

The early contributions were primarily theoretical. Later on, Ma and Sun [7] introduced the use of the maximum entropy principle to estimate mutual information via copulas. Subsequently, Singh and Zhang [8] expanded the application of copulas connected to entropy for multivariate stochastic modeling in water engineering.

Piantadosi et al [9] (resp. [10]) found bivariate (resp. multivariate) copulas with maximal differential entropy under the knowledge of

(I1)

the marginals,

(I3)

the grade correlation coefficients.

Their solutions are checkerboard copulas.

Recently Lin et al. [11] applied the principle of maximum entropy and found checkerboard copulas as solutions. Their approach differs in the aspect that they start from distributions whose marginals are not continuous; the discontinuities determine the checkerboard structure. The difficulties of extension of the copula approach to distributions with discontinuous marginals are presented by Genest et al. [12].

While the literature on this topic remains relatively limited, this leaves considerable room for further theoretical advancements and potential new applications.

None of the preceding approaches restricts solutions by prescribed values at some points (I2). Sometimes the values of the joint cdf (and hence also of the copula) at some points are known. Such constraints can originate, e.g., from a discretization of a continuous scale. We want to choose a distribution fitting to these constraints.

Let us choose the distribution whose copula has maximal differential entropy (among those fitting to the constraints). The principle of maximal entropy was successfully used in many other tasks. It expresses our intention “not to include more information than that contained in the constraints.” Choosing this option, we try not to introduce any other dependence than that which follows from the given restrictions. This seems to be a natural criterion for the choice of the model fitting to constraints.

The result can be equivalently described by the joint cdf or by the copula. The differential entropy of the joint cdf suffers some drawbacks, e.g., it is not invariant to the change of scale; the entropies of random variables X and $cX$, $c\ne 1$, are different. In contrast to it, the copula remains the same even if the random variables are transformed by arbitrary increasing functions $f,g$,

$$C_{X,Y}(x,y)=C_{f\left(X\right),g\left(Y\right)}(x,y).$$

Thus the differential entropy of a copula is a well-defined notion which can be a criterion for the choice of the model.4

In this paper, we formulate the task of finding a copula with maximal differential entropy fitting the constraints. We convert it to a finite-dimensional optimization problem. As one of the main contributions, we prove that the result is independent of some inputs. This further simplifies the task. It leads to a system of higher-order polynomial equations which allows for a numerical solution using convex optimization. We show that in the simplest cases, an analytical solution is also feasible.

1.4. Notation and Formulation of the Problem

Before the formulation of the task solved in this paper, let us introduce our notation. We restrict attention to 2-copulas, i.e., binary functions describing the dependence of two random variables. We are trying to find a copula, $C_{X,Y}:{[0,1]}^2\to [0,1]$. As we shall not deal with another copula in the sequel, we shall denote it briefly by C. For the readers not familiar with the notion of a copula, it can be any joint cdf with marginals uniformly distributed on $\left]0,1\right[$.5 They are characterized by the following necessary and sufficient conditions:

$$\begin{array}{cc}\hfill C(w,0)=C(0,w)& =0,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill C(w,1)=C(1,w)& =w,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill C(x_i,y_j)-C(x_i,y_{\ell })-C(x_k,y_j)+C(x_k,y_{\ell })& \ge 0\mathrm{for}{x}_k<x_i,y_{\ell }<y_j.\hfill \end{array}$$

(4)

Inequality (4) means that the probability

$$P(x_k<X<x_i,y_{\ell }<Y<y_j)=C(x_i,y_j)-C(x_i,y_{\ell })-C(x_k,y_j)+C(x_k,y_{\ell })$$

(5)

is non-negative.

The constraints are finitely many points in ${\left]0,1\right[}^2$ where the value of the copula is given. Their first (resp. second) coordinates can be organized in an increasing sequence $(x_1,\dots ,x_{I-1})$, resp. $(y_1,\dots ,y_{J-1})$. These values determine a rectangular grid. At some of the grid points, $(x_i,y_j)$, the values $z_{i,j}$ are given,

$$C(x_i,y_j)=z_{i,j}.$$

(6)

We denote by $G_0$ the set of all indices $(i,j)\in \{1,\dots ,I-1\}\times \{1,\dots ,J-1\}$ for which the values $z_{i,j}$ of the copula are given by (6). To simplify the notation, we define additionally $x_0=y_0=0$, $x_I=y_J=1$. The corresponding new grid points are at the boundary of the domain of the copula and form the set

$$B_{I,J}=\{(x_i,0),(x_i,1)\mid i=0,\dots ,k\}\cup \{(0,y_j),(1,y_j)\mid j=0,\dots ,J\}.$$

The values of the copula at the boundary are given by (2), (3), so we may define $z_{i,j}$ for all $(i,j)\in B_{I,J}$ by

$$\begin{array}{cccc}\hfill z_{i,0}& =C(x_i,0)=0,\hfill & \hfill z_{0,j}& =C(0,y_j)=0,\hfill \end{array}$$

(7)

$$\begin{array}{cccc}\hfill z_{i,J}& =C(x_i,1)=x_i,\hfill & \hfill z_{I,j}& =C(1,y_j)=y_j.\hfill \end{array}$$

(8)

In total, (6) is required for all $(i,j)$ from the set

$$G=G_0\cup B_{I,J}$$

and $G_0$ can be expressed as

$$G_0=G\setminus B_{I,J}=G\cap {\left]0,1\right[}^2.$$

In figures, we use a filled disk to denote a grid point at which the value is given (an element of G) and an empty circle to denote a grid point where the value is not restricted. Figure 1 and Figure 2 demonstrate our notation.

Problem 1. 

Let $0=x_0<x_1<\cdots <x_I=1$, $0=y_0<y_1<\cdots <y_J=1$, and let there be given values $z_{i,j}\in [0,1]$ for all $(i,j)$ in some set G such that $B_{I,J}\subseteq G\subseteq \{0,\dots ,I\}\times \{0,\dots ,J\}$. Find a copula C satisfying (6) for all $(i,j)\in G$. Moreover, among all such copulas, we want to find one with maximum differential entropy.

The necessary and sufficient conditions for a copula, in particular (4), imply necessary conditions for solvability of Problem 1 which later will appear also sufficient.

Proposition 1. 

A necessary condition for the existence of a solution to Problem 1 is the conjunction of (7), (8), and

(N)

If $(i,j),(i,\ell ),(k,j),(k,\ell )\in G$, $k<i$, $\ell <j$, then $z_{i,j}-z_{i,\ell }-z_{k,j}+z_{k,\ell }\ge 0.$

In the sequel, we assume that the constraints satisfy these conditions.

2. General Results

2.1. Reformulation to a Finite-Dimensional Optimization

As formulated, Problem 1 looks like a task from variational calculus. We shall transform it into a finite-dimensional optimization problem.

It is well-known that among all absolutely continuous distributions on a bounded set, the uniform distribution has the highest differential entropy (if the integral of the density over the whole domain is fixed). We shall derive a consequence of this principle. To simplify its formulation, we denote the lengths of intervals between neighboring grid points:

$$\begin{array}{cc}\hfill a_i& =x_i-x_{i-1},\hfill \\ \hfill b_j& =y_j-y_{j-1}\hfill \end{array}$$

for all $i\in \{1,\dots ,I\}$, $j\in \{1,\dots ,J\}$ (see Figure 1).

Proposition 2. 

Let $i\in \{1,\dots ,I\}$, $j\in \{1,\dots ,J\}$. The restriction of the copula C to the two-dimensional open interval $\left]x_{i-1},x_i\right[\times \left]y_{j-1},y_j\right[$ contributes to the copula differential entropy by

$${\int }_{x_{i-1}}^{x_i}{\int }_{y_{j-1}}^{y_j}\eta \left(c_{X,Y}(x,y)\right)\mathrm{d}x\mathrm{d}y.$$

(9)

Among all copulas with a given value

$$\begin{array}{cc}\hfill n_{i,j}& :=C(x_i,y_j)-C(x_i,y_{j-1})-C(x_{i-1},y_j)+C(x_{i-1},y_{j-1})\hfill \\ \hfill & =P(x_{i-1}<X<x_i,y_{j-1}<Y<y_j),\hfill \end{array}$$

(10)

the contribution (9) is maximal iff its density is constant,

$$c_{X,Y}(x,y)={\displaystyle \frac{n_{i,j}}{(x_i-x_{i-1})(y_j-y_{j-1})}}={\displaystyle \frac{n_{i,j}}{a_i{b}_j}},$$

(11)

almost everywhere on $\left]x_{i-1},x_i\right[\times \left]y_{j-1},y_j\right[$. Without loss of generality, we may choose the copula with density (11) for all $(x,y)\in \left]x_{i-1},x_i\right[\times \left]y_{j-1},y_j\right[$.

Proposition 2 determines the solution to Problem 1 (almost everywhere) when all values

$$C(x_i,y_j),C(x_i,y_{j-1}),C(x_{i-1},y_j),C(x_{i-1},y_{j-1})$$

are given, i.e., $(i,j)$, $(i,j-1)$, $(i-1,j)$, $(i-1,j-1)\in G$. Even when this is not the case, there are some optimal values $n_{i,j}$, $(i,j)\in \{1,\dots ,I\}\times \{1,\dots ,J\}$, and the copula density on $(x_{i-1},x_i)\times (y_{j-1},y_j)$ can be chosen according to (11) because there are no other restrictions on its values. This observation allows us to restrict attention to copulas with piecewise constant densities, so-called checkerboard copulas. Lin et al. [11] also found checkerboard copulas by maximization of the differential entropy, although they solved a different task.

It remains to find the optimal values of finitely many densities on intervals (rectangles) $\left]x_{i-1},x_i\right[\times \left]y_{j-1},y_j\right[$ for $(i,j)\in \{1,\dots ,I\}\times \{1,\dots ,J\}$. Equivalently, we look for the values $n_{i,j}$ defined by (10). From them, the values of the copula at grid points can be computed as sums

$$C(x_i,y_j)=\sum _{u=1}^i\sum _{v=1}^j{n}_{u,v}.$$

(12)

In particular, for $i=I$, resp. $j=J$, we obtain

$$\begin{array}{cc}\hfill \sum _{u=1}^I\sum _{v=1}^j{n}_{u,v}& =C(x_I,y_j)=C(1,y_j)=y_j,\hfill \\ \hfill \sum _{u=1}^i\sum _{v=1}^J{n}_{u,v}& =C(x_i,y_J)=C(x_i,1)=x_i.\hfill \end{array}$$

These formulas can be equivalently expressed using row and column sums:

$$\begin{array}{cc}\hfill \sum _{u=1}^I{n}_{u,j}& =y_j-y_{j-1},\hfill \\ \hfill \sum _{v=1}^J{n}_{i,v}& =x_i-x_{i-1}.\hfill \end{array}$$

Remark 2. 

We do not deal with the values of density on the boundaries of these intervals. The boundaries are of measure zero, so they do not influence the results. In fact, the densities at these boundaries are not uniquely defined.

Copulas that do not have densities or entropies (see Remark 1) do not seem to be good candidates for approximation and cannot compete with our proposed solution.

The differential entropy of the checkerboard copula is

$$\begin{array}{cc}\hfill Ent\left(c_{X,Y}\right)& ={\int }_0^1{\int }_0^1\eta \left(c_{X,Y}(x,y)\right)\mathrm{d}x\mathrm{d}y\hfill \\ \hfill & =\sum _{i=1}^I\sum _{j=1}^J{a}_i{b}_j\eta \left({\displaystyle \frac{n_{i,j}}{a_i{b}_j}}\right)\hfill \\ \hfill & =-\sum _{i=1}^I\sum _{j=1}^J{n}_{i,j}ln\left({\displaystyle \frac{n_{i,j}}{a_i{b}_j}}\right),\hfill \end{array}$$

provided that all $n_{i,j}\ne 0$ (otherwise, the corresponding summand of the differential entropy is 0, i.e., minimal). We shall deal with the following reformulation:

Problem 2. 

Let $0=x_0<x_1<\cdots <x_I=1$, $0=y_0<y_1<\cdots <y_J=1$, and let there be given values $z_{i,j}\in [0,1]$ for all $(i,j)$ in some set G such that $B_{I,J}\subseteq G\subseteq \{0,\dots ,I\}\times \{0,\dots ,J\}$. Suppose that values $z_{i,j}$, $(i,j)\in G$, satisfy the necessary conditions from Proposition 1. Find values $n_{i,j}\in [0,1]$, $(i,j)\in \{1,\dots ,I\}\times \{1,\dots ,J\}$, satisfying

$$\begin{array}{cc}\hfill \sum _{u=1}^I{n}_{u,j}& =y_j-y_{j-1},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \sum _{v=1}^J{n}_{i,v}& =x_i-x_{i-1},\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \sum _{u=1}^i\sum _{v=1}^j{n}_{u,v}& =z_{i,j}\mathrm{for}\mathrm{all}(i,j)\in G,\hfill \end{array}$$

(15)

and such that the differential entropy

$$\begin{array}{cc}\hfill Ent\left(c_{X,Y}\right)& =-\sum _{i=1}^I\sum _{j=1}^J{n}_{i,j}ln\left({\displaystyle \frac{n_{i,j}}{(x_i-x_{i-1})(y_j-y_{j-1})}}\right)\hfill \end{array}$$

(16)

is maximal.

Remark 3. 

If $G=\{0,\dots ,I\}\times \{0,\dots ,J\}$, i.e., if all values $z_{i,j}$, $(i,j)\in \{1,\dots ,I\}\times \{1,\dots ,J\}$, are given, then (10) and (6) determine all $n_{i,j}$ and the solution.

2.2. Decomposition of the Problem

We concentrate on rectangles with the following property:

(B)

The values of the copula at all grid points at the boundary of the rectangle $[x_k,x_K]\times [y_{\ell },y_L]$ are given, i.e.,

$$\{(x_i,y_{\ell }),(x_i,y_L)\mid i=k,\dots ,K\}\cup \{(x_k,y_j),(x_K,y_j)\mid j=\ell ,\dots ,L\}\subseteq G.$$

Then the optimization of the copula inside such a rectangle is independent of the rest of the domain and can be solved separately. E.g., in Figure 2, rectangles satisfying (B) are drawn by thick lines. The whole domain $[0,1]\times [0,1]$ has the values on the entire boundary given, so it satisfies (B).

Let us consider a rectangle satisfying (B). If all its grid points in some row or column (not at the boundary) are in G, then we can split it into two disjoint rectangles satisfying (B). (The choice need not be unique.) If this is not the case (i.e., there is a missing value in each row and each column6), we call the rectangle irreducible. Irreducible rectangles can be equivalently characterized as minimal rectangles satisfying (B). The whole domain can be covered by disjoint irreducible rectangles. This may allow us to decompose the problem into simpler ones, dealing with each irreducible rectangle separately. Figure 2 shows eight irreducible rectangles. Each component of the decomposed task has the following formulation. The active restrictions refer to grid points with indices from the set $G\cap \{k,\dots ,K\}\times \{\ell ,\dots ,L\}$. Notice that the monotonicity of the values $z_{i,j}$ needs to be formulated explicitly, while in Problem 2 it followed from (7).

Problem 3. 

Let $0\le x_k<\cdots <x_K\le 1$, $0\le y_{\ell }<\cdots <y_L\le 1$, and let $G\subseteq \{k,\dots ,K\}\times \{\ell ,\dots ,L\}$ be a set such that

$$\begin{array}{c}\hfill \{(x_i,\ell ),(x_i,L)\mid i=0,\dots ,I\}\cup \{(k,y_j),(K,y_j)\mid j=0,\dots ,J\}\subseteq G.\end{array}$$

Suppose that values $z_{i,j}\in [0,1]$ are given for all $(i,j)\in G$ so that sequences ${\left(z_{i,\ell }\right)}_{i=k,\dots ,K}$, ${\left(z_{k,j}\right)}_{j=\ell ,\dots ,L}$ are nondecreasing and(N)is satisfied. Find values $n_{i,j}\in [0,1]$, $(i,j)\in \{k+1,\dots ,K\}\times \{\ell +1,\dots ,L\}$, satisfying

$$\begin{array}{cc}\hfill \sum _{u=k+1}^i\sum _{v=\ell +1}^j{n}_{u,v}& =z_{i,j}-z_{i,\ell }-z_{k,j}+z_{k,\ell }\hfill \end{array}$$

(17)

for all $(i,j)\in G$ and such that the contribution to the differential entropy

$$\begin{array}{c}\hfill -\sum _{i=k+1}^K\sum _{j=\ell +1}^L{n}_{i,j}ln\left({\displaystyle \frac{n_{i,j}}{(x_i-x_{i-1})(y_j-y_{j-1})}}\right)\end{array}$$

(18)

is maximal.

In the sequel, we analyze special cases of irreducible rectangles. Before that, we simplify indexing by shifting the rectangle to the origin.

2.3. Shifted Indexing

If a rectangle $[x_k,x_K]\times [y_{\ell },y_L]$ satisfies (B), the corresponding values $n_{i,j}$,

$(i,j)\in \{k+1,\dots ,K\}\times \{\ell +1,\dots ,L\}$, are bounded only by the row and column sums

$$\begin{array}{cc}\hfill r_j& =\sum _{i=k+1}^K{n}_{i,j}=z_{K,j}-z_{K,j-1}-z_{k,j}+z_{k,j-1},j\in \{\ell +1,\dots ,L\},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill s_i& =\sum _{j=\ell +1}^L{n}_{i,j}=z_{i,L}-z_{i-1,L}-z_{i,\ell }+z_{i-1,\ell },i\in \{k+1,\dots ,K\},\hfill \end{array}$$

(20)

which are non-negative values satisfying the equality

$$\sum _{j=\ell }^L{r}_j=\sum _{i=k}^K{s}_i=\sum _{j=\ell +1}^L\sum _{i=k+1}^K{n}_{i,j}=z_{K,L}-z_{k,L}-z_{K,\ell }+z_{k,\ell }.$$

(21)

The same task is obtained if we take a rectangle of the same size with the lower left vertex at the origin and impose the same requirements on the row and column sums. We shall apply this simplification, and instead of the original task for a general rectangle, we shall, without loss of generality, solve the case when $k=\ell =0$ in the above notation (with the new upper bounds still denoted by $K,L$; the above general indexing will not be used anymore). The boundary grid points form the set

$$B_{K,L}=\{(x_i,0),(x_i,L)\mid i=0,\dots ,I\}\cup \{(0,y_j),(K,y_j)\mid j=0,\dots ,J\}.$$

The situation simplifies not only in indexing (starting from 0, resp. 1), but also the values at the bottom and left edge become zero. The values at the top and right bound must be modified to general values $z_{i,L}$, $z_{K,j}$, which are not determined by $x_i$, $y_j$ like they were in (8). The modified task is as follows.

Problem 4. 

Let $0=x_0<\cdots <x_K\le 1$, $0=y_0<\cdots <y_L\le 1$, and let and let there be given values $z_{i,j}\in [0,1]$ for all $(i,j)$ in some set G such that $B_{K,L}\subseteq G\subseteq \{0,\dots ,K\}\times \{0,\dots ,L\}$.7 Suppose that values $z_{i,j}$, $(i,j)\in G$, satisfy the necessary conditions(N)and (7) for $i\in \{0,\dots ,K\}$ and $j\in \{0,\dots ,L\}$. Find values $n_{i,j}\in [0,1]$, $(i,j)\in \{1,\dots ,K\}\times \{1,\dots ,L\}$, satisfying

$$\begin{array}{cc}\hfill \sum _{u=1}^i\sum _{v=1}^j{n}_{u,v}& =z_{i,j}\mathrm{for}\mathrm{all}(i,j)\in G\cap \left]0,K\right]\times \left]0,L\right]\hfill \end{array}$$

(22)

and such that the contribution to the differential entropy

$$\begin{array}{c}\hfill -\sum _{i=1}^K\sum _{j=1}^L{n}_{i,j}ln\left({\displaystyle \frac{n_{i,j}}{(x_i-x_{i-1})(y_j-y_{j-1})}}\right)\end{array}$$

(23)

is maximal.

As in [11], this is an optimization of a concave function that has a maximum that can be found numerically (applying standard convex optimization with the opposite sign). For numerical solutions of convex tasks, standard references are, e.g., [15,16,17,18]. We shall show that analytical solutions exist in some cases and we demonstrate them to support the intuition about the task.

3. Less General Tasks

In this section, we collect results related to Problem 4. Throughout this section, we consider an irreducible rectangle $[0,x_K]\times [0,y_L]$.

3.1. Rectangle with no Given Values Inside

Suppose that the set $G=B_{K,L}$ has an empty intersection with the interior of the rectangle $\left]0,x_K\right[\times \left]0,y_L\right[$. This is an instance of Problem 4; equations (22) reduce to monotonicity.

Problem 5. 

Let $0=x_0<\cdots <x_K\le 1$, $0=y_0<\cdots <y_L\le 1$, and let

$$G=\{(x_i,0),(x_i,L)\mid i=0,\dots ,I\}\cup \{(0,y_j),(K,y_j)\mid j=0,\dots ,J\}.$$

Suppose that values $z_{i,j}\in [0,1]$ are given for all $(i,j)\in G$ so that (7) is satisfied for all $i\in \{0,\dots ,K\}$ and $j\in \{0,\dots ,L\}$ and sequences ${\left(z_{i,L}\right)}_{i=0}^K$, ${\left(z_{K,j}\right)}_{j=0}^L$ are nondecreasing. Find values $n_{i,j}\in [0,1]$, $(i,j)\in \{1,\dots ,K\}\times \{1,\dots ,L\}$, satisfying

$$\begin{array}{cc}\hfill \sum _{u=1}^K{n}_{u,j}& =z_{K,j},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill \sum _{v=1}^L{n}_{i,v}& =z_{i,L},\hfill \end{array}$$

(25)

and such that the contribution to the differential entropy

$$\begin{array}{c}\hfill -\sum _{i=1}^K\sum _{j=1}^L{n}_{i,j}ln\left({\displaystyle \frac{n_{i,j}}{(x_i-x_{i-1})(y_j-y_{j-1})}}\right)\end{array}$$

(26)

is maximal.

The solution need not be a copula with a constant density at $\left]0,x_K\right[\times \left]0,y_L\right[$ because we must keep the correct row and column sums.

We use Lagrange multipliers ${\lambda }_1,\dots ,{\lambda }_K$, ${\mu }_1,\dots ,{\mu }_L$; the Lagrange function is

$$L(n_{i,j},{\lambda }_i,{\mu }_j)=\sum _{i=1}^K\sum _{j=1}^L-n_{i,j}ln{\displaystyle \frac{n_{i,j}}{a_i{b}_j}}+\sum _{i=1}^K{\lambda }_i\left(\sum _{j=1}^L{n}_{i,j}-s_i\right)+\sum _{j=1}^L{\mu }_j\left(\sum _{i=1}^K{n}_{i,j}-r_j\right).$$

We put all partial derivatives equal to zero and obtain a system of equations

$${\displaystyle \frac{\partial L}{\partial n_{i,j}}}=-1-ln{\displaystyle \frac{n_{i,j}}{a_i{b}_j}}+{\lambda }_i+{\mu }_j=0,$$

where $i\in \{1,2\cdots ,K\}$, $j\in \{1,2\cdots ,L\}$. Equivalently,

$$ln{\displaystyle \frac{n_{i,j}}{a_i{b}_j}}=-1+{\lambda }_i+{\mu }_j.$$

Subtracting the last ones, we obtain

$$\begin{array}{cc}\hfill ln{\displaystyle \frac{n_{i,j}}{a_i{b}_j}}-ln{\displaystyle \frac{n_{K,j}}{a_K{b}_j}}& ={\lambda }_i-{\lambda }_K,\hfill \\ \hfill ln{\displaystyle \frac{n_{i,j}}{a_i{b}_j}}-ln{\displaystyle \frac{n_{i,L}}{a_i{b}_L}}& ={\mu }_i-{\mu }_L,\hfill \end{array}$$

and express $n_{i,j}$ using $n_{K,j}$ or $n_{i,L}$ as

$$\begin{array}{cc}\hfill n_{i,j}& =n_{K,j}{\displaystyle \frac{a_i}{a_K}}exp({\lambda }_i-{\lambda }_K),\hfill \\ \hfill n_{i,j}& =n_{i,L}{\displaystyle \frac{b_j}{b_L}}exp({\mu }_j-{\mu }_L).\hfill \end{array}$$

We see that all rows (resp. columns) are multiples of the last one and the matrix composed of all $n_{i,j}$ is a dyad (a product of two vectors),

$$\left[\begin{array}{ccc}{n}_{1,1}& \cdots & n_{1,K}\\ ⋮& \ddots & ⋮\\ n_{L,1}& \cdots & n_{L,K}\end{array}\right]=c\left[\begin{array}{c}{r}_1\\ ⋮\\ r_m\end{array}\right]\left[\begin{array}{ccc}{s}_1& \cdots & s_t\end{array}\right],$$

(27)

where $r_j$ are the row sums, $s_i$ the column sums, and the constant c can be determined from the known total sum

$$\begin{array}{cc}\hfill c\sum _{j=1}^L\sum _{i=1}^K{s}_i{r}_j& =\sum _{j=1}^L{r}_j=\sum _{i=1}^K{s}_i,\hfill \\ \hfill c& ={\displaystyle \frac{{\displaystyle \sum _{j=1}^L{r}_j}}{{\displaystyle \sum _{i=1}^L\sum _{j=1}^K{s}_j{r}_i}}}={\displaystyle \frac{1}{{\displaystyle \sum _{i=1}^K{s}_i}}}.\hfill \end{array}$$

This is an expected result that says that the probability density of a result in an elementary rectangle is the product of the probability densities of the marginal distributions.

3.2. Methodology of Solving Particular Cases

Here we collect ideas used to solve particular cases described in the appendices. We consider Problem 4. This leads to a system of linear equations; after the Gauss–Jordan elimination, it could look like

$$M=\left[\begin{array}{ccccccccc}1& 0& 1& 0& 0& \cdots & 0& 1& C_1\\ 0& 1& 1& 0& 0& \cdots & 0& 0& C_2\\ 0& 0& 0& 1& 0& \cdots & 0& 1& C_3\\ 0& 0& 0& 0& 1& \cdots & 0& 1& C_4\\ ⋮& ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮\\ 0& 0& 0& 0& 0& \cdots & 1& 1& C_N\end{array}\right],$$

where $C_1,\dots ,C_N$ are some constants computed from the values $z_{i,j}$. Until we consider the maximized differential entropy, some variables $n_{i,j}$ can be chosen and the rest can be computed from them. This describes all possible solutions to the system of linear equations (ignoring their bounds). To maximize the differential entropy, we compute the partial derivatives of the contribution to the differential entropy and put them equal to zero. We obtain

$$N=KL-cardG=KL-K-L+1-o$$

equations, where $o=card(G\cap \left]0,x_K\right[\times \left]0,y_L\right[)$ is the number of given values inside the rectangle.

Theorem 1. 

The optimal solution to Problem 4 is independent of the values $a_1,\dots ,a_K$, $b_1,\dots ,b_L$.

Proof. 

Recall that

$$n_{i,j}=z_{i,j}-z_{i-1,j}-z_{i,j-1}+z_{i-1,j-1}.$$

(28)

This leads to an optimization in those variables $z_{i,j}=C(x_i,y_j)$ where the copula values are not given ($(i,j)\notin G$).

In the criterion, each variable $z_{i,j}$, $(i,j)\in G\setminus B_{K,L}$, occurs in four equations; for $n_{i,j}$ and $n_{i+1,j+1}$ with coeefficient 1 and for $n_{i+1,j}$ and $n_{i,j+1}$ with coeefficient $-1$.

We solve this task by computing the partial derivatives w.r.t. unknown values $z_{i,j}$ and putting them zero,

$$\begin{array}{cc}\hfill & ln{\displaystyle \frac{z_{i,j}-z_{i-1,j}-z_{i,j-1}+z_{i-1,j-1}}{a_i{b}_j}}+1-ln{\displaystyle \frac{z_{i,j+1}-z_{i-1,j+1}-z_{i,j}+z_{i-1,j}}{a_i{b}_{j+1}}}-1\hfill \\ \hfill & -ln{\displaystyle \frac{z_{i+1,j}-z_{i,j}-z_{i+1,j-1}+z_{i,j-1}}{a_{i+1}{b}_j}}-1+ln{\displaystyle \frac{z_{i+1,j+1}-z_{i+1,j}-z_{i,j+1}+z_{i,j}}{a_{i+1}{b}_{j+1}}}+1=0.\hfill \end{array}$$

(29)

This can be expressed as

$$\begin{array}{cc}\hfill & {\displaystyle \frac{z_{i,j}-z_{i-1,j}-z_{i,j-1}+z_{i-1,j-1}}{a_i{b}_j}}·{\displaystyle \frac{z_{i+1,j+1}-z_{i+1,j}-z_{i,j+1}+z_{i,j}}{a_{i+1}{b}_{j+1}}}\hfill \\ \hfill =& {\displaystyle \frac{z_{i,j+1}-z_{i-1,j+1}-z_{i,j}+z_{i-1,j}}{a_i{b}_{j+1}}}·{\displaystyle \frac{z_{i+1,j}-z_{i,j}-z_{i+1,j-1}+z_{i,j-1}}{a_{i+1}{b}_j}}.\hfill \end{array}$$

Multiplying this by $a_i{b}_j{a}_{i+1}{b}_{j+1}$, we obtain an equation independent of $a_i$, $b_j$, $a_{i+1}$, and $b_{j+1}$. This holds for all $z_{i,j}$ and $n_{i,j}$, $(i,j)\in \{1,\dots ,K-1\}\times \{1,\dots ,L-1\}\setminus G$. Due to (28), also the values $n_{i,j}$ are independent of $a_1,\dots ,a_K$, $b_1,\dots ,b_L$. □

Besides the independence of the result on interval lengths $a_1,\dots ,a_K$, $b_1,\dots ,b_L$, we proved that each equation in the system describing entropy maximization is of the form

$$\prod _{{\kappa }_1\in Z_{+}}(n_{i,j}+B_{{\kappa }_1})=\prod _{{\kappa }_2\in Z_{-}}(-n_{i,j}+B_{{\kappa }_2}),$$

where $Z_{+}$ is the set of indices $(i,j)$ for which $n_{i,j}$ occurs with the positive sign and $Z_{-}$ is the set of indices for which $n_{i,j}$ occurs with the negative sign. Apparently, $card{Z}_{+}=card{Z}_{-}$, therefore the units in (29) cancelled.

The only problem is that $n_{i,j}$ may depend also on other variables. Thus the summands $B_{*}$ contain not only constants $C_{*}$, but also other independent variables.

In the appendices, we present explicit solutions to some specific instances of Problem 4. They lead to higher-order algebraic equations.

4. Conclusion

We formulated the task of fitting a continuous copula to finitely many given values in such a way that the entropy of the copula density is maximal. This is motivated by situations when some of the values are prescribed and the rest of the copula should be “as independent as possible”, with the intention not to include any other dependence than that contained in the constraints. We simplified the task with several hints, showed that it has a unique solution (because it is equivalent to a convex optimization problem), and demonstrated that a closed-form solution can be found analytically in some simple cases, although it may require solving higher-order polynomials. We propose this concept as an alternative to current approaches, which also maximize the entropy, but use different restrictions of the admitted copulas.