Chordal Metric Formula Between Generalized Singular Values of Grassmann Matrix Pairs by Riemannian Optimization Models

0.1. GSVD and Its Applications

More precisely, GSVD induces a linear transformation of two data sets from the two genes × arrays spaces to two reduced and diagonalized “genelets” × “arraylets” spaces. A single microarray probes the relative expression levels of p genes in a single sample. A series of $q_1$ arrays probes the genome-scale expression levels in $q_1$ different samples, i.e., under $q_1$ different experimental conditions. Let matrix A of size p-genes ×$q_1$-arrays, denote the full expression data, whose k-th row is the expression of the k-th gene across the different samples that correspond to the different arrays. Let the matrix B of size t-genes ×$q_2$-arrays, denote the relative expression levels of t genes under $q_2=q_1=q<max\{p,t\}$ experimental conditions that correspond one to one to the $q_1$ conditions underlying A. GSVD induces simultaneous linear transformation of two expression data sets A and B from two p-genes ×q-arrays and t-genes ×q-arrays spaces to two reduced q-genelets ×q-arraylets spaces.

Let A and B have the following GSVD: $A=U{\mathrm{\Sigma }}_A{R}^{-1},B=V{\mathrm{\Sigma }}_B{R}^{-1}$. In these spaces the data is represented by the diagonal non-negative matrices ${\mathrm{\Sigma }}_A$ and ${\mathrm{\Sigma }}_B$ and denote $\langle k|{\mathrm{\Sigma }}_A|l\rangle ={\epsilon }_{1,l}{\delta }_{kl},\langle k|{\mathrm{\Sigma }}_B|l\rangle ={\epsilon }_{2,l}{\delta }_{kl}$ for all $1\le k,l\le q$, where ${\delta }_{kl}=1$ if $k=l$ and ${\delta }_{kl}=0$ if $k\ne l$. By $\langle k|{\mathrm{\Sigma }}_A$ we denote the k-th row of the matrix ${\mathrm{\Sigma }}_A$, which lists the expression of the k-th gene across the different samples that correspond to the different arrays. By ${\mathrm{\Sigma }}_A|l\rangle$ we denote the l-th column of the matrix ${\mathrm{\Sigma }}_A$, which lists the genome-scale expression measured by the l-th array. The antisymmetric angular distance between the data sets [3], ${\theta }_l=arctan({\epsilon }_{1,l}/{\epsilon }_{2,l})-\pi /4$ indicates the relative significance of the l-th genelet, i.e., its significance in the first data set relative to that in the second in terms of the ratio of the expression information captured by this genelet in the first data set to that in the second. An angular distance of 0 indicates a genelet of equal significance in both data sets, with ${\epsilon }_{1,l}={\epsilon }_{2,l}$.

Alter et al. [1] made comparative analysis of gene mRNA expression data sets by transforming those into GSVs and analyzed the degree of change for reduced and diagonalized “genelets” × “arraylets” spaces in gene expression levels from samples under different experimental conditions. This behavior is significant for guiding further experimental research. Motivated by practical applications proximity and range size between GSVs, Xu et al. [22] considered the upper bound of sum of chordal metric between generalized singular values of Grassmann matrix pairs and applied this result in comparative analysis of gene mRNA expression data.

0.2. Riemannian Optimization

Optimization on Riemannian manifolds is called Riemannian optimization. Unconstrained optimization methods on the Euclidean space, such as the steepest descent, the conjugate gradient, and Newton’s methods are generalized to those on a Riemannian manifold. The term “optimization on manifolds” is applied to algorithms that preserve manifold constraints during iterations. Generally speaking, preserving constraints has advantages in many situations, for example, when the cost functions are undefined or of little use outside the feasible region and when the algorithm is terminated prior to convergence yet a feasible solution is required. Theories and methods for optimization on manifolds date back to the 1970s, and algorithms for specific problems, e.g., various eigenvalue problems, appeared even earlier. Most of the general-purpose algorithms, however, did not appear until the 1990s [30]. While it is trivial to generate trial points in Rn along straight search lines, it is not as easy to do so in the curved manifold. A natural choice is the geodesic, which is the analogof straightline: it has the shortest length between two points in the manifold. For more details we may refer reders to [29,30,31,32].

In this paper we provide explicit formula of sum of chordal metric between generalized singular values of Grassmann matrix pairs with Riemannian optimization models, which absolutely improves the existing upper bounds in [22] in theory. The new formula involves two small-size unitary variable matrices from Riemannian optimization models. Then we use Newton methods on Riemannian manifolds to solve the involved Riemannian optimization models for deducing these two variable matrices solutions efficiently. The contributions of this paper are summarized as follows.

• On theoretic aspect we provide a new explicit formula of sum of chordal metric between generalized singular values of Grassmann matrix pairs with Riemannian optimization models instead of the existing upper bounds.
• On algorithm aspect the new expression formula involves only two small-size unitary variable matrices from Riemannian optimization models and Newton methods on Riemannian manifolds are given, which reduces the required computational cost.

0.3. Notations

Throughout this paper, by $\mathbb{R}$, $\mathbb{C}$, ${\mathbb{R}}^n$, ${\mathbb{C}}^{m\times n},{\mathbb{U}}_n$ and ${\mathbb{OR}}_n$ we denote the sets of real numbers, complex numbers, real vectors of order n, $m\times n$ matrices with entries in $\mathbb{C}$, and $n\times n$ unitary matrices and real orthogonal matrices, respectively. The symbols $|·|$ and $\mathfrak{Re}(·)$ stand for absolute value and real part of a complex number, respectively. The symbols $I_n$ and $O_{m\times n}$ represent the identity matrix of order n and $m\times n$ zero matrix, respectively. For a square matrix $A=\left(a_{ij}\right)\in {\mathbb{C}}^{n\times n}$, by $A^T$, $A^H$, $A^{-1},$ and $\mathrm{tr}\left(A\right)$ we denote the transpose, conjugate transpose, inverse, and trace of matrix A, respectively. We use ${\parallel ·\parallel }_2$ to denote the spectral norm of a matrix. The singular value set of A is denoted by $\sigma \left(A\right)$. The complex number $a+bı(a,b\in \mathbb{R})$ and the real number pair $(a,b)$ can establish a one-to-one correspondence with a point on the coordinate plane, so that the coordinate plane that establishes a one-to-one correspondence with the whole complex number is called Gauss plane, which is denoted by $\mathfrak{P}(1,1)$. For given matrices $A,B\in {\mathbb{C}}^{n\times n}$, $A<(\le )B$ means $B-A$ is a positive (semi-)definite matrix. As in [9,15,16,17], we also use the chordal metric on the Riemann sphere to measure the difference between two GSVs. The pair $(\alpha ,\beta )$ can be regarded as a point in Gauss plane $\mathfrak{P}(1,1)$. For two points $(\alpha ,\beta )\ne (0,0),(\gamma ,\delta )\ne (0,0)$ in Gauss plane $\mathfrak{P}(1,1),$ their chordal distance is defined by

$$\begin{array}{c}\hfill \rho ((\alpha ,\beta ),(\gamma ,\delta ))={\displaystyle \frac{|\alpha \delta -\beta \gamma |}{\sqrt{{\left(\right|\alpha |}^2+{\left|\beta \right|}^2{\left)\right(\left|\gamma \right|}^2+{\left|\delta \right|}^2)}}}\end{array}$$

(1)

to measure the difference between two points $(\alpha ,\beta )$ and $(\gamma ,\delta )$. Next, we outline some definitions related to GSVD. Readers may refer to [7,9,16] for these definitions.

Definition 1. 

The matrix pair $\{A,B\}$ with $A,B\in {\mathbb{C}}^{n\times n}$ is said to be regular if there exists $\zeta \in \mathbb{C}$ such that $\det (A+\zeta B)\ne 0.$ Denote by

$${\mathcal{G}}_{1,2}=\{(\alpha ,\beta )\ne (0,0):\alpha ,\beta \in \mathbb{C}\}.$$

The pair $(\alpha ,\beta )\in {\mathcal{G}}_{1,2}$ is called a generalized eigenvalue of a regular pair $\{A,B\}$, if $\det (\beta A-\alpha B)=0$.

The set of generalized eigenvalues of the regular pair $\{A,B\}$ is denoted by $\lambda (A,B)$.

Definition 2. 

Let $A\in {\mathbb{C}}^{m\times n}$ and $B\in {\mathbb{C}}^{p\times n}$. A matrix pair $\{A,B\}$ is an $(m,p,n)$-Grassman matrix pair (GMP) if rank ${(A^T,B^T)}^T=n$.

For GSVD, it has several formulations in the literature. In this paper we adopt the following form as in [9,16,19].

Definition 3. 

Let {A,B} be an (m,p,n)-GMP. Then there exist unitary matrices $U\in {\mathbb{C}}^{m\times m},V\in {\mathbb{C}}^{p\times p},$ and a nonsingular matrix $R\in {\mathbb{C}}^{n\times n}$ such that

$$U^{H}AR={\mathrm{\Sigma }}_A,V^{H}BR={\mathrm{\Sigma }}_B,$$

(2)

$${\mathrm{\Sigma }}_A=\left(\begin{array}{cc}\mathrm{\Lambda }& \\ & O_{(m-r-s)\times (n-r-s)}\end{array}\right),{\mathrm{\Sigma }}_B=\left(\begin{array}{cc}{O}_{(p+r-n)\times r}& \\ & \Omega \end{array}\right),$$

(3)

where $O_{(m-r-s)\times (n-r-s)}$ and $O_{(p+r-n)\times r}$ are zero matrices, and

$$\mathrm{\Lambda }=\mathrm{diag}({\alpha }_1,\dots ,{\alpha }_{r+s}),\Omega =\mathrm{diag}({\beta }_{r+1},\dots ,{\beta }_n),$$

with

$$1={\alpha }_1=\cdots ={\alpha }_r>{\alpha }_{r+1}\ge \cdots \ge {\alpha }_{r+s}>{\alpha }_{r+s+1}=\cdots ={\alpha }_n=0,$$

$$0={\beta }_1=\cdots ={\beta }_r<{\beta }_{r+1}\le \cdots \le {\beta }_{r+s}<{\beta }_{r+s+1}=\cdots ={\beta }_n=1,$$

and

$${\alpha }_i^2+{\beta }_i^2=1,1\le i\le n.$$

We note that by Definition 3, given another $(m,p,n)$-GMP $\{\tilde{A},\tilde{B}\}$, there also exist unitary matrices $\tilde{U}\in {\mathbb{C}}^{m\times m},\tilde{V}\in {\mathbb{C}}^{p\times p}$, nonsingular $\tilde{R}\in {\mathbb{C}}^{n\times n}$, and nonnegative diagonal matrices ${\mathrm{\Sigma }}_{\tilde{A}}\in {\mathbb{C}}^{m\times n},{\mathrm{\Sigma }}_{\tilde{B}}\in {\mathbb{C}}^{p\times n}$ for the GSVD of $\{\tilde{A},\tilde{B}\}$.

For GSVD of a real matrix pair $\{A,B\}$ we adopt the following definition form as in [9,16,19].

Definition 4. 

Let $A\in {\mathbb{R}}^{m\times n},B\in {\mathbb{R}}^{p\times n}$ have rank $(A^T,B^T)=n.$ Then there exist two orthogonal matrices $U\in {\mathbb{R}}^{m\times m},V\in {\mathbb{R}}^{p\times p}$ and a nonsingular matrix $R\in {\mathbb{R}}^{n\times n}$ such that

$$U^{T}AR={\mathrm{\Sigma }}_A,V^{T}BR={\mathrm{\Sigma }}_B,$$

(4)

$${\mathrm{\Sigma }}_A=\left(\begin{array}{cc}\mathrm{\Lambda }& \\ & O_{(m-r-s)\times (n-r-s)}\end{array}\right),{\mathrm{\Sigma }}_B=\left(\begin{array}{cc}{O}_{(p+r-n)\times r}& \\ & \Omega \end{array}\right),$$

(5)

where $\mathrm{\Lambda }=\mathrm{diag}({\alpha }_1,\dots ,{\alpha }_{r+s})$ and $\Omega =\mathrm{diag}({\beta }_{r+1},\dots ,{\beta }_n)$ with

$$\begin{array}{c}1={\alpha }_1=\cdots ={\alpha }_r>{\alpha }_{r+1}\ge \cdots \ge {\alpha }_{r+s}>{\alpha }_{r+s+1}=\cdots ={\alpha }_n=0,\\ 0={\beta }_1=\cdots ={\beta }_r<{\beta }_{r+1}\le \cdots \le {\beta }_{r+s}<{\beta }_{r+s+1}=\cdots ={\beta }_n=1,\end{array}$$

and ${\alpha }_i^2+{\beta }_i^2=1$ for $1\le i\le n$.

0.4. Overview of Existing Results about Chordal Metric Between GSVs of GMP

There have been a large amount of papers in various topics related to GSVD and its applications, e.g., see [5,6,8,9,10,11,12,16,17,18,19,23]. One of interesting and important topics is about chordal metric between GSVs of Grassmann matrix pairs (GMPs), which was first analyzed by Sun [16] in 1983. In [16], Sun presented the Hoffman-Wielandt theorem for GSVs of GMPs and gave bounds on perturbations of GSVs, which generalized several well-known results for the standard singular value problem. Since then, Paige and Saunders [15] in 1981 gave a bound for GSV variations. Li [9] in 1993 presented several perturbation bounds of GSVs and its associated subspace. Recently Xu et al. [22] considered sharp upper bounds of the chordal metric between generalized singular values of Grassmann matrix pairs. The following results provide perturbation bounds based on the chordal metric of GSVs.

Theorem 1 

(The Hoffman-Wielandt type theorem for GSV of GMP[16]). Let $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ be two $(m,p,n)$-GMPs, $\sigma \{A,B\}={\left\{({\alpha }_i,{\beta }_i)\right\}}_{i=1}^n$ and $\sigma \{\tilde{A},\tilde{B}\}={\left\{({\tilde{\alpha }}_i,{\tilde{\beta }}_i)\right\}}_{i=1}^n$. Then

$$\prod _{i=1}^n\left(1-{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))\right)\ge 1-d^2(Y,\tilde{Y}),$$

(6)

$$\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))\le n\left(1-\sqrt[n]{1-d^2(Y,\tilde{Y})}\right),$$

(7)

where

$$Y=\left(\begin{array}{c}A\\ B\end{array}\right),\tilde{Y}=\left(\begin{array}{c}\tilde{A}\\ \tilde{B}\end{array}\right),$$

$\rho (({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ is given by (1) and $d(Y,\tilde{Y})={\left\{{\displaystyle 1-\frac{|\det \left(Y^H\tilde{Y}\right){|}^2}{\det \left(Y^{H}Y\right)\det \left({\tilde{Y}}^H\tilde{Y}\right)}}\right\}}^{{\displaystyle \frac{1}{2}}}$.

Theorem 2 

(Explicit expression and sharp upper bound[22]). Let $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ be two $(m,p,n)$-GMPs, $\sigma \{A,B\}={\left\{({\alpha }_i,{\beta }_i)\right\}}_{i=1}^n$ and $\sigma \{\tilde{A},\tilde{B}\}={\left\{({\tilde{\alpha }}_i,{\tilde{\beta }}_i)\right\}}_{i=1}^n$. Then

$$\begin{array}{cc}\hfill \prod _{i=1}^n\left(1-{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))\right)& \equiv \underset{{\mathrm{\Psi }}_1\in {\mathbb{U}}_m,{\mathrm{\Psi }}_2\in {\mathbb{U}}_p}{max}(1-d^2(Y,{\tilde{Y}}_{\mathrm{\Psi }}))\hfill \\ \hfill & =1-d^2(Y,{\tilde{Y}}_{{\mathrm{\Psi }}_0}),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))& \le n\left(1-\underset{{\mathrm{\Psi }}_1\in {\mathbb{U}}_m,{\mathrm{\Psi }}_2\in {\mathbb{U}}_p}{max}\sqrt[n]{1-d^2(Y,{\tilde{Y}}_{\mathrm{\Psi }})}\right)\hfill \\ \hfill & =n\left(1-\sqrt[n]{1-d^2(Y,{\tilde{Y}}_{{\mathrm{\Psi }}_0})}\right),\hfill \end{array}$$

(9)

where

$$Y=\left(\begin{array}{c}A\\ B\end{array}\right),\tilde{Y}=\left(\begin{array}{c}\tilde{A}\\ \tilde{B}\end{array}\right),{\tilde{Y}}_{\mathrm{\Psi }}=\mathrm{\Psi }\tilde{Y},{\tilde{Y}}_{{\mathrm{\Psi }}_0}={\mathrm{\Psi }}_0\tilde{Y},$$

$$\mathrm{\Psi }=\left(\begin{array}{cc}{\mathrm{\Psi }}_1& O_{m\times p}\\ O_{p\times m}& {\mathrm{\Psi }}_2\end{array}\right),{\mathrm{\Psi }}_0=\left(\begin{array}{cc}U{\tilde{U}}^H& O_{m\times p}\\ O_{p\times m}& V{\tilde{V}}^H\end{array}\right),$$

and ${\mathrm{\Psi }}_1\in {\mathbb{U}}_m,{\mathrm{\Psi }}_2\in {\mathbb{U}}_p;U,\tilde{U},V,\tilde{V}$ are the unitary matrices in GSVD of $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ (refer to (2)-(3)). The inequality (9) is an equality if and only if $\rho (({\alpha }_1,{\beta }_1),({\tilde{\alpha }}_1,{\tilde{\beta }}_1))=\rho (({\alpha }_2,{\beta }_2),({\tilde{\alpha }}_2,{\tilde{\beta }}_2))=\cdots =\rho (({\alpha }_n,{\beta }_n),({\tilde{\alpha }}_n,{\tilde{\beta }}_n)).$

In this paper we continue along this line and further consider explicit expression of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ instead of the existing upper bound in (9).

0.5. Organization

The rest of this paper is organized as follows. Section 1 is the preliminary. In Section 2, we present an explicit expression of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ by Riemannian optimization models and the corresponding numerical methods. Finally, in Section 3 concluding remarks are drawn.

1. Preliminaries

In this section we propose some useful lemmas in order to deduce the main results.

Lemma 1. 

[21]Let $A_1,\dots ,A_m\in {\mathbb{C}}^{n\times n}$ with ${\sigma }_1\left(A_j\right)\ge {\sigma }_2\left(A_j\right)\ge \cdots \ge {\sigma }_n\left(A_j\right)\ge 0$ the singular values arranged in decreasing order and $c\in \mathbb{R},j=1,\dots ,m$. We have the following extreme values:

$$\begin{array}{c}\hfill \underset{U_1,\dots ,U_m\in {\mathbb{U}}_n}{max}\left|\mathrm{tr}(c{I}_n\pm \prod _{j=1}^m{U}_j{A}_j)\right|=n\mid c\mid +\sum _{i=1}^n\prod _{j=1}^m{\sigma }_i\left(A_j\right).\end{array}$$

Let $(A,B)$ and $(\tilde{A},\tilde{B})$ be two $(m,p,n)$ GMPs and

$$\begin{array}{c}\hfill {\mathbf{S}}_1=\left\{\begin{array}{c}{\left(Z^{H}Z\right)}^{-1/2}{A}^H,m\le n,m\le p;\\ A{\left(Z^{H}Z\right)}^{-1/2},n\le m,m\le p;\\ {\left(Z^{H}Z\right)}^{-1/2}{B}^H,p\le n,p\le m;\\ B{\left(Z^{H}Z\right)}^{-1/2},n\le p,p\le m;\end{array}\right.\end{array}$$

(10)

$${\mathbf{S}}_2=\left\{\begin{array}{c}{\left({\tilde{Z}}^H\tilde{Z}\right)}^{-1/2}{\tilde{A}}^H,m\le n,m\le p;\\ \tilde{A}{\left({\tilde{Z}}^H\tilde{Z}\right)}^{-1/2},n\le m,m\le p;\\ {\left({\tilde{Z}}^H\tilde{Z}\right)}^{-1/2}{\tilde{B}}^H,p\le n,p\le m;\\ \tilde{B}{\left({\tilde{Z}}^H\tilde{Z}\right)}^{-1/2},n\le p,p\le m\end{array}\right.$$

and

$$\begin{array}{cc}\hfill & \mathbf{M}\left(\mathrm{\Phi }\right)={\mathrm{\Phi }}^H{\mathbf{S}}_2^H{\mathbf{S}}_2\mathrm{\Phi }{\mathbf{S}}_1^H{\mathbf{S}}_1,\forall \mathrm{\Phi }\in {\mathbb{U}}_{min\{m,p,n\}}.\hfill \end{array}$$

(11)

Let $(A,B)$ and $(\tilde{A},\tilde{B})$ be two $(m,p,n)$ real matrix pairs defined in (4) and

$$\begin{array}{c}\hfill {\mathcal{S}}_1=\left\{\begin{array}{c}{\left(Z^{T}Z\right)}^{-1/2}{A}^T,m\le n,m\le p;\\ A{\left(Z^{T}Z\right)}^{-1/2},n\le m,m\le p;\\ {\left(Z^{T}Z\right)}^{-1/2}{B}^T,p\le n,p\le m;\\ B{\left(Z^{T}Z\right)}^{-1/2},n\le p,p\le m;\end{array}\right.\end{array}$$

(12)

$${\mathcal{S}}_2=\left\{\begin{array}{c}{\left({\tilde{Z}}^T\tilde{Z}\right)}^{-1/2}{\tilde{A}}^T,m\le n,m\le p;\\ \tilde{A}{\left({\tilde{Z}}^T\tilde{Z}\right)}^{-1/2},n\le m,m\le p;\\ {\left({\tilde{Z}}^T\tilde{Z}\right)}^{-1/2}{\tilde{B}}^T,p\le n,p\le m;\\ \tilde{B}{\left({\tilde{Z}}^T\tilde{Z}\right)}^{-1/2},n\le p,p\le m\end{array}\right.$$

and

$$\begin{array}{cc}\hfill & \mathcal{M}\left(\mathrm{\Phi }\right)={\mathrm{\Phi }}^T{\mathcal{S}}_2^T{\mathcal{S}}_2\mathrm{\Phi }{\mathcal{S}}_1^T{\mathcal{S}}_1,\forall \mathrm{\Phi }\in {\mathbb{OR}}_{min\{m,p,n\}}.\hfill \end{array}$$

(13)

We set

$$\begin{array}{c}\hfill {\mathbf{Q}}_i=\left\{\begin{array}{c}\mathrm{diag}(I_i,O_{(min\{m,p,n\}-i)\times (min\{m,p,n\}-i)}),m\le n,m\le p;n\le m,m\le p;\\ \mathrm{diag}(O_{(min\{m,p,n\}-i)\times (min\{m,p,n\}-i)},I_i),p\le n,p\le m;n\le p,p\le m\end{array}\right.\end{array}$$

(14)

for $1\le i\le min\{m,p,n\}$.

Lemma 2. 

[24] Let $f\left(X\right):{\mathbb{C}}^{n\times n}\to \mathbb{R}$ be an analytical function of several complex variables on the domain $X{X}^H\le I_n$. Then $f\left(X\right)$ attains its maximum modulus on the characteristic manifold $\{X\in {\mathbb{C}}^{n\times n}:X{X}^H=I_n\}$.

Lemma 3. 

Let $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ be two $(m,p,n)$-GMPs and let $\sigma \{A,B\}=\{({\alpha }_i,{\beta }_i),i=1,2\dots ,n\}$ and $\sigma \{\tilde{A},\tilde{B}\}=\{({\tilde{\alpha }}_j,{\tilde{\beta }}_j),j=1,2\dots ,n\}.$ Then

$$\begin{array}{cc}\hfill \sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)& =2\underset{\mathrm{\Phi }\in {\mathbb{U}}_{min\{m,p,n\}}}{max}tr\left(\mathbf{M}\left(\mathrm{\Phi }\right)\right)-tr({\mathbf{S}}_1^H{\mathbf{S}}_1+{\mathbf{S}}_2^H{\mathbf{S}}_2)+n,\hfill \end{array}$$

(15)

where $\mathbf{M}\left(\mathrm{\Phi }\right)$ is given by (11).

Proof. 

Let $Z=\left(\begin{array}{c}A\\ B\end{array}\right),\tilde{Z}=\left(\begin{array}{c}\tilde{A}\\ \tilde{B}\end{array}\right),$ then $Z^{H}Z=R^{-H}{R}^{-1},{\tilde{Z}}^H\tilde{Z}={\tilde{R}}^{-H}{\tilde{R}}^{-1}$ are nonsingular. Let

$$\begin{array}{cc}& S_1={\left(Z^{H}Z\right)}^{-1/2}{A}^H={\left(R^{-H}{R}^{-1}\right)}^{-1/2}{R}^{-H}{\mathrm{\Sigma }}_A^H{U}^H,\end{array}$$

$$\begin{array}{cc}& S_2={\left({\tilde{Z}}^H\tilde{Z}\right)}^{-1/2}{\tilde{A}}^H={\left({\tilde{R}}^{-H}{\tilde{R}}^{-1}\right)}^{-1/2}{\tilde{R}}^{-H}{\mathrm{\Sigma }}_{\tilde{A}}^H{\tilde{U}}^H\\ & S_3={\left(Z^{H}Z\right)}^{-1/2}{B}^H={\left(R^{-H}{R}^{-1}\right)}^{-1/2}{R}^{-H}{\mathrm{\Sigma }}_B^H{V}^H,\end{array}$$

(16)

$$\begin{array}{cc}& S_4={\left({\tilde{Z}}^H\tilde{Z}\right)}^{-1/2}{\tilde{B}}^H={\left({\tilde{R}}^{-H}{\tilde{R}}^{-1}\right)}^{-1/2}{\tilde{R}}^{-H}{\mathrm{\Sigma }}_{\tilde{B}}^H{\tilde{V}}^H\end{array}$$

(17)

and

$$\begin{array}{cc}\hfill & M_1\left(\mathrm{\Phi }\right)=S_1{\mathrm{\Phi }}^H{S}_2^H{S}_2\mathrm{\Phi }{S}_1^H,\forall \mathrm{\Phi }\in {\mathbb{U}}_m;M_2\left(\mathrm{\Phi }\right)=S_1^H{\mathrm{\Phi }}^H{S}_2{S}_2^H\mathrm{\Phi }{S}_1,\forall \mathrm{\Phi }\in {\mathbb{U}}_n\hfill \\ \hfill & M_3\left(\mathrm{\Phi }\right)=S_3{\mathrm{\Phi }}^H{S}_4^H{S}_4\mathrm{\Phi }{S}_3^H,\forall \mathrm{\Phi }\in {\mathbb{U}}_p;M_4\left(\mathrm{\Phi }\right)=S_3^H{\mathrm{\Phi }}^H{S}_4{S}_4^H\mathrm{\Phi }{S}_3,\forall \mathrm{\Phi }\in {\mathbb{U}}_n.\hfill \end{array}$$

(18)

We first prove

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)-tr({\mathbf{S}}_1^H{\mathbf{S}}_1+{\mathbf{S}}_2^H{\mathbf{S}}_2)+n.$$

(19)

Using the GSVDs of $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ in Definition 3 we have

$$A=U{\mathrm{\Sigma }}_{A}R,B=V{\mathrm{\Sigma }}_{B}R,\tilde{A}=\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}\tilde{R},\tilde{B}=\tilde{V}{\mathrm{\Sigma }}_{\tilde{B}}\tilde{R},$$

where $U,\tilde{U},V,\tilde{V},{\mathrm{\Sigma }}_A,{\mathrm{\Sigma }}_B,{\mathrm{\Sigma }}_{\tilde{A}}$ and ${\mathrm{\Sigma }}_{\tilde{B}}$ are given by (2) and (3). Let $M_1\left(\mathrm{\Phi }\right)$ be given by (18). Then

$$\begin{array}{cc}\hfill \underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)& =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left({\mathrm{\Phi }}^H{S}_2^H{S}_2\mathrm{\Phi }{S}_1^H{S}_1\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(\mathrm{\Phi }\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^H{\tilde{U}}^H{\mathrm{\Phi }}^{H}U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H{U}^H\right).\hfill \end{array}$$

(20)

Next we consider two cases for different m and n. Case 1: if $n\le m$, then we write

$${\mathrm{\Sigma }}_A=\left(\begin{array}{c}{\widehat{\mathrm{\Sigma }}}_A\\ O_{(m-n)\times n}\end{array}\right),{\mathrm{\Sigma }}_{\tilde{A}}=\left(\begin{array}{c}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}\\ O_{(m-n)\times n}\end{array}\right),U^H\mathrm{\Phi }\tilde{U}=\left(\begin{array}{cc}{W}_{11}& W_{12}\\ W_{21}& W_{22}\end{array}\right),\forall \mathrm{\Phi }\in {\mathbb{U}}_m,$$

where ${\widehat{\mathrm{\Sigma }}}_A,{\widehat{\mathrm{\Sigma }}}_{\tilde{A}},W_{11}\in {\mathbb{C}}^{n\times n}$ and $W_{22}\in {\mathbb{C}}^{(m-n)\times (m-n)}$. Substituting into (20) we have

$$\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)=\underset{W_{11}{W}_{11}^H\le I_n}{max}\mathrm{tr}\left({\widehat{\mathrm{\Sigma }}}_A^H{W}_{11}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^H{W}_{11}^H{\widehat{\mathrm{\Sigma }}}_A\right),$$

where $W_{11}{W}_{11}^H\le I_n$ means that $I_n-W_{11}{W}_{11}^H$ is positive semi-definite. As we know, $\mathrm{tr}\left({\widehat{\mathrm{\Sigma }}}_A^H{W}_{11}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^H{W}_{11}^H{\widehat{\mathrm{\Sigma }}}_A\right)$ as an analytical function of several complex variables on the domain $W_{11}{W}_{11}^H\le I_n$ attains its maximum modulus on the characteristic manifold $\{W_{11}\in {\mathbb{C}}^{n\times n}:W_{11}{W}_{11}^H=I_n\}$ (see [21]), therefore by Lemma 2 we have

$$\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)=\underset{W_{11}{W}_{11}^H=I_n}{max}\mathrm{tr}\left({\widehat{\mathrm{\Sigma }}}_A^H{W}_{11}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^H{W}_{11}^H{\widehat{\mathrm{\Sigma }}}_A\right),$$

which together with Lemma 1 and (20) yields

$$\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)=\sum _{i=1}^n{\alpha }_i^2{\tilde{\alpha }}_i^2.$$

(21)

Let $U^H\mathrm{\Psi }U=\left(\begin{array}{cc}{U}_{11}& U_{12}\\ U_{21}& U_{22}\end{array}\right),{\tilde{U}}^H\tilde{\mathrm{\Psi }}\tilde{U}=\left(\begin{array}{cc}{\tilde{U}}_{11}& {\tilde{U}}_{12}\\ {\tilde{U}}_{21}& {\tilde{U}}_{22}\end{array}\right),\forall \mathrm{\Psi },\tilde{\mathrm{\Psi }}\in {\mathbb{U}}_m$ with $U_{11},{\tilde{U}}_{11}\in {\mathbb{C}}^{n\times n}$ and $U_{22},{\tilde{U}}_{22}\in {\mathbb{C}}^{(m-n)\times (m-n)}$, then by Lemma 1 and (33) we have

$$\begin{array}{c}\hfill \mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)\\ \hfill & =\mathrm{tr}({\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H+{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^H),\hfill \\ \hfill & =\sum _{i=1}^n({\alpha }_i^2+{\tilde{\alpha }}_i^2).\hfill \end{array}$$

(22)

Case 2: if $m\le n$, then we write

$${\mathrm{\Sigma }}_A=({\widehat{\mathrm{\Sigma }}}_A,O_{m\times (n-m)}),{\mathrm{\Sigma }}_{\tilde{A}}=({\widehat{\mathrm{\Sigma }}}_{\tilde{A}},O_{m\times (n-m)}).$$

Then by (20) and Lemma 1 we have

$$\begin{array}{cc}\hfill \underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)& =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(\mathrm{\Phi }\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^H{\tilde{U}}^H{\mathrm{\Phi }}^{H}U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H{U}^H\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(\begin{array}{cc}\mathrm{\Phi }\tilde{U}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^H{\tilde{U}}^H{\mathrm{\Phi }}^{H}U{\widehat{\mathrm{\Sigma }}}_A{\widehat{\mathrm{\Sigma }}}_A^H{U}^H& 0\\ 0& 0_{(n-m)\times (n-m)}\end{array}\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(\mathrm{\Phi }\tilde{U}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^H{\tilde{U}}^H{\mathrm{\Phi }}^{H}U{\widehat{\mathrm{\Sigma }}}_A{\widehat{\mathrm{\Sigma }}}_A^H{U}^H\right)\hfill \\ \hfill & =\sum _{i=1}^m{\alpha }_i^2{\tilde{\alpha }}_i^2=\sum _{i=1}^n{\alpha }_i^2{\tilde{\alpha }}_i^2\hfill \end{array}$$

(23)

since ${\alpha }_{m+1}=\cdots ={\alpha }_n={\tilde{\alpha }}_{m+1}=\cdots ={\tilde{\alpha }}_n=0$ by the definition of GSVD. It follows from Lemma 1 and (33) that

$$\begin{array}{cc}\hfill \mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)& =\mathrm{tr}(U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H{U}^H+\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^H{\tilde{U}}^H),\hfill \\ \hfill & =\sum _{i=1}^m({\alpha }_i^2+{\tilde{\alpha }}_i^2)=\sum _{i=1}^n({\alpha }_i^2+{\tilde{\alpha }}_i^2).\hfill \end{array}$$

(24)

By Definition 3 we have ${\beta }_i^2=1-{\alpha }_i^2,{\tilde{\beta }}_i^2=1-{\tilde{\alpha }}_i^2,$ and together with (20)-(24) we have

$$\begin{array}{ccc}\hfill \sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)& =& \sum _{i=1}^n\left(2{\alpha }_i^2{\tilde{\alpha }}_i^2-({\alpha }_i^2+{\tilde{\alpha }}_i^2)+1\right)\hfill \\ & =& 2\sum _{i=1}^n{\alpha }_i^2{\tilde{\alpha }}_i^2-\sum _{i=1}^n({\alpha }_i^2+{\tilde{\alpha }}_i^2)+n\hfill \\ & =& 2\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n.\hfill \end{array}$$

(25)

By Cases 1 and 2 we deduce (35) holds true. By the similar aforementioned methods we get

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{U}}_n}{max}\mathrm{tr}\left(M_2\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n.$$

(26)

Note that in (41)

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\sum _{i=1}^n{\beta }_i^2{\tilde{\beta }}_i^2-\sum _{i=1}^n({\beta }_i^2+{\tilde{\beta }}_i^2)+n,$$

and hence by the similar aforementioned methods we have

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{U}}_p}{max}\mathrm{tr}\left(M_3\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n,$$

(27)

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{U}}_n}{max}\mathrm{tr}\left(M_4\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n.$$

(28)

In order to select the smallest dimensional variable unitary matrix by (35), (42)-(44) we have (31) holds. This completes the proof.    □

Lemma 4. 

Let $\mathrm{\Lambda }=diag({\lambda }_1,\dots ,{\lambda }_n),\Delta =diag({\delta }_1,\dots ,{\delta }_n)\in {\mathbb{R}}^{n\times n}$ be diagonal matrices with ${\lambda }_1\ge \cdots \ge {\lambda }_n\ge 0$ and ${\delta }_1\ge \cdots \ge {\delta }_n\ge 0$. Then we have

$$\begin{array}{ccc}& & \underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(\mathrm{\Lambda }{\mathrm{\Phi }}^T\Delta \mathrm{\Phi }\right)=\sum _{i=1}^n{\lambda }_i{\delta }_i,\hfill \end{array}$$

Proof. 

It follows from (1) that

$$\begin{array}{cc}\hfill & \underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(\mathrm{\Lambda }{\mathrm{\Phi }}^T\Delta \mathrm{\Phi }\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(\mathrm{\Lambda }{\mathrm{\Phi }}^H\Delta \mathrm{\Phi }\right)\hfill \\ \hfill & \le \underset{\mathrm{\Phi }\in {\mathbb{U}}_n}{max}\mathrm{tr}\left(\mathrm{\Lambda }{\mathrm{\Phi }}^H\Delta \mathrm{\Phi }\right)\hfill \\ \hfill & =\sum _{i=1}^n{\lambda }_i{\delta }_i.\hfill \end{array}$$

(29)

Meanwhile,

$$\begin{array}{c}\hfill \sum _{i=1}^n{\lambda }_i{\delta }_i=\mathrm{tr}\left(\mathrm{\Lambda }\Delta \right)\le \underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(\mathrm{\Lambda }{\mathrm{\Phi }}^T\Delta \mathrm{\Phi }\right).\end{array}$$

(30)

Combing (29) and (30) yields

$$\underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(\mathrm{\Lambda }{\mathrm{\Phi }}^T\Delta \mathrm{\Phi }\right)=\sum _{i=1}^n{\lambda }_i{\delta }_i.$$

This completed the proof.    □

Lemma 5. 

Let $A\in {\mathbb{R}}^{m\times n},B\in {\mathbb{R}}^{p\times n}$ have rank $(A^T,B^T)=n$ and the GSV $({\alpha }_i,{\beta }_i)$ of $\{A,B\}$ be given in (4) and (5). Then

$$\begin{array}{cc}\hfill \sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)& =2\underset{\mathrm{\Phi }\in {\mathbb{OR}}_{min\{m,p,n\}}}{max}tr\left(\mathcal{M}\left(\mathrm{\Phi }\right)\right)-tr({\mathcal{S}}_1^T{\mathcal{S}}_1+{\mathcal{S}}_2^T{\mathcal{S}}_2)+n,\hfill \end{array}$$

(31)

where $\mathcal{M}\left(\mathrm{\Phi }\right)$ is given by (13).

Proof. 

Let $Z=\left(\begin{array}{c}A\\ B\end{array}\right),\tilde{Z}=\left(\begin{array}{c}\tilde{A}\\ \tilde{B}\end{array}\right),$ then $Z^{T}Z=R^{-T}{R}^{-1},{\tilde{Z}}^T\tilde{Z}={\tilde{R}}^{-T}{\tilde{R}}^{-1}$ are nonsingular. Let

$$\begin{array}{cc}& S_1={\left(Z^{T}Z\right)}^{-1/2}{A}^T={\left(R^{-T}{R}^{-1}\right)}^{-1/2}{R}^{-T}{\mathrm{\Sigma }}_A^T{U}^T,\end{array}$$

$$\begin{array}{cc}& S_2={\left({\tilde{Z}}^T\tilde{Z}\right)}^{-1/2}{\tilde{A}}^T={\left({\tilde{R}}^{-T}{\tilde{R}}^{-1}\right)}^{-1/2}{\tilde{R}}^{-T}{\mathrm{\Sigma }}_{\tilde{A}}^T{\tilde{U}}^T\\ & S_3={\left(Z^{T}Z\right)}^{-1/2}{B}^T={\left(R^{-T}{R}^{-1}\right)}^{-1/2}{R}^{-T}{\mathrm{\Sigma }}_B^T{V}^T,\end{array}$$

(32)

$$\begin{array}{cc}& S_4={\left({\tilde{Z}}^T\tilde{Z}\right)}^{-1/2}{\tilde{B}}^T={\left({\tilde{R}}^{-T}{\tilde{R}}^{-1}\right)}^{-1/2}{\tilde{R}}^{-T}{\mathrm{\Sigma }}_{\tilde{B}}^T{\tilde{V}}^T\end{array}$$

(33)

and

$$\begin{array}{cc}\hfill & M_1\left(\mathrm{\Phi }\right)=S_1{\mathrm{\Phi }}^T{S}_2^T{S}_2\mathrm{\Phi }{S}_1^T,\forall \mathrm{\Phi }\in {\mathbb{OR}}_m;M_2\left(\mathrm{\Phi }\right)=S_1^T{\mathrm{\Phi }}^T{S}_2{S}_2^T\mathrm{\Phi }{S}_1,\forall \mathrm{\Phi }\in {\mathbb{OR}}_n\hfill \\ \hfill & M_3\left(\mathrm{\Phi }\right)=S_3{\mathrm{\Phi }}^T{S}_4^T{S}_4\mathrm{\Phi }{S}_3^T,\forall \mathrm{\Phi }\in {\mathbb{OR}}_p;M_4\left(\mathrm{\Phi }\right)=S_3^T{\mathrm{\Phi }}^T{S}_4{S}_4^T\mathrm{\Phi }{S}_3,\forall \mathrm{\Phi }\in {\mathbb{OR}}_n.\hfill \end{array}$$

(34)

We first prove

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)-tr({\mathcal{S}}_1^T{\mathcal{S}}_1+{\mathcal{S}}_2^T{\mathcal{S}}_2)+n.$$

(35)

Then

$$\begin{array}{cc}\hfill \underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)& =\underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left({\mathrm{\Phi }}^T{S}_2^H{S}_2\mathrm{\Phi }{S}_1^T{S}_1\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(\mathrm{\Phi }\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^T{\tilde{U}}^T{\mathrm{\Phi }}^{H}U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H{U}^T\right),\hfill \end{array}$$

(36)

which together with Lemma 1 and (20) yields

$$\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)=\sum _{i=1}^n{\alpha }_i^2{\tilde{\alpha }}_i^2.$$

(37)

Let $U^T\mathrm{\Psi }U=\left(\begin{array}{cc}{U}_{11}& U_{12}\\ U_{21}& U_{22}\end{array}\right),{\tilde{U}}^T\tilde{\mathrm{\Psi }}\tilde{U}=\left(\begin{array}{cc}{\tilde{U}}_{11}& {\tilde{U}}_{12}\\ {\tilde{U}}_{21}& {\tilde{U}}_{22}\end{array}\right),\forall \mathrm{\Psi },\tilde{\mathrm{\Psi }}\in {\mathbb{OR}}_m$ with $U_{11},{\tilde{U}}_{11}\in {\mathbb{C}}^{n\times n}$ and $U_{22},{\tilde{U}}_{22}\in {\mathbb{C}}^{(m-n)\times (m-n)}$, then by Lemma 1 and (33) we have

$$\begin{array}{cc}\hfill \mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)& =\mathrm{tr}({\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H+{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^H),\hfill \\ \hfill & =\sum _{i=1}^n({\alpha }_i^2+{\tilde{\alpha }}_i^2).\hfill \end{array}$$

(38)

Case 2: if $m\le n$, then we write

$${\mathrm{\Sigma }}_A=({\widehat{\mathrm{\Sigma }}}_A,O_{m\times (n-m)}),{\mathrm{\Sigma }}_{\tilde{A}}=({\widehat{\mathrm{\Sigma }}}_{\tilde{A}},O_{m\times (n-m)}).$$

Then by (20) and Lemma 1 we have

$$\begin{array}{cc}\hfill \underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)& =\underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(\mathrm{\Phi }\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^T{\tilde{U}}^T{\mathrm{\Phi }}^{T}U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^T{U}^T\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(\begin{array}{cc}\mathrm{\Phi }\tilde{U}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^T{\tilde{U}}^T{\mathrm{\Phi }}^{T}U{\widehat{\mathrm{\Sigma }}}_A{\widehat{\mathrm{\Sigma }}}_A^T{U}^T& 0\\ 0& 0_{(n-m)\times (n-m)}\end{array}\right)\hfill \\ \hfill & =\underset{\mathrm{\Phi }\in {\mathbb{OR}}_m}{max}\mathrm{tr}\left(\mathrm{\Phi }\tilde{U}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}{\widehat{\mathrm{\Sigma }}}_{\tilde{A}}^T{\tilde{U}}^T{\mathrm{\Phi }}^{T}U{\widehat{\mathrm{\Sigma }}}_A{\widehat{\mathrm{\Sigma }}}_A^T{U}^T\right)\hfill \\ \hfill & =\sum _{i=1}^m{\alpha }_i^2{\tilde{\alpha }}_i^2=\sum _{i=1}^n{\alpha }_i^2{\tilde{\alpha }}_i^2\hfill \end{array}$$

(39)

since ${\alpha }_{m+1}=\cdots ={\alpha }_n={\tilde{\alpha }}_{m+1}=\cdots ={\tilde{\alpha }}_n=0$ by the definition of GSVD. It follows from Lemma 1 and (33) that

$$\begin{array}{cc}\hfill \mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)& =\mathrm{tr}(U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^T{U}^T+\tilde{U}{\mathrm{\Sigma }}_{\tilde{A}}{\mathrm{\Sigma }}_{\tilde{A}}^T{\tilde{U}}^T),\hfill \\ \hfill & =\sum _{i=1}^m({\alpha }_i^2+{\tilde{\alpha }}_i^2)=\sum _{i=1}^n({\alpha }_i^2+{\tilde{\alpha }}_i^2).\hfill \end{array}$$

(40)

By Definition 3 we have ${\beta }_i^2=1-{\alpha }_i^2,{\tilde{\beta }}_i^2=1-{\tilde{\alpha }}_i^2,$ and together with (20)-((24)) we have

$$\begin{array}{ccc}\hfill \sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)& =& \sum _{i=1}^n\left(2{\alpha }_i^2{\tilde{\alpha }}_i^2-({\alpha }_i^2+{\tilde{\alpha }}_i^2)+1\right)\hfill \\ & =& 2\sum _{i=1}^n{\alpha }_i^2{\tilde{\alpha }}_i^2-\sum _{i=1}^n({\alpha }_i^2+{\tilde{\alpha }}_i^2)+n\hfill \\ & =& 2\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n.\hfill \end{array}$$

(41)

By Cases 1 and 2 we deduce (19) holds true. By the similar aforementioned methods we get

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(M_2\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n.$$

(42)

Note that in (25)

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\sum _{i=1}^n{\beta }_i^2{\tilde{\beta }}_i^2-\sum _{i=1}^n({\beta }_i^2+{\tilde{\beta }}_i^2)+n,$$

and hence by the similar aforementioned methods we have

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{OR}}_p}{max}\mathrm{tr}\left(M_3\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n,$$

(43)

$$\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)=2\underset{\mathrm{\Phi }\in {\mathbb{OR}}_n}{max}\mathrm{tr}\left(M_4\left(\mathrm{\Phi }\right)\right)-\mathrm{tr}(S_1^H{S}_1+S_2^H{S}_2)+n.$$

(44)

In order to select the smallest dimensional variable unitary matrix by (35), (42)-(44) we have (31) holds. This completes the proof.    □

2. Explicit Formula of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ with Riemannian Optimization Models

We first present a new explicit formula of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ for complex grassman matrix pair. Then we use the corresponding numerical methods to solve solutions of the involved Riemannian optimization models.

2.1. For complex grassman matrix pair

We present a new explicit formula of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ for complex grassman matrix pair.

Lemma 6. 

Let $\mathbf{M}\left(\mathrm{\Phi }\right),{\mathbf{Q}}_i,{\mathbf{S}}_1,{\mathbf{S}}_2$ be given by (10), (11) and (14). Then

$$\underset{\mathrm{\Phi }\in {\mathbb{U}}_{min\{m,p,n\}}}{max}tr\left(\mathbf{M}\left(\mathrm{\Phi }\right)\right)=tr\left(\mathbf{M}\left({\mathbf{F}}_2{\mathbf{F}}_1^H\right)\right),$$

where ${\mathbf{F}}_1,{\mathbf{F}}_2\in {\mathbb{U}}_{min\{m,p,n\}}$ can be obtained from ${\mathrm{\Pi }}_1,{\mathrm{\Pi }}_2$ by solving the trace function optimization models

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_1^H{\mathbf{S}}_1^H{\mathbf{S}}_1{\mathrm{\Pi }}_1{\mathbf{Q}}_i\right)\right|,\end{array}$$

(45)

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_2\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_2^H{\mathbf{S}}_2^H{\mathbf{S}}_2{\mathrm{\Pi }}_2{\mathbf{Q}}_i\right)\right|\end{array}$$

(46)

for every $i=1,\dots ,n$, respectively.

Proof. 

We first prove the case for $m\le n,m\le p$ and for others cases when $n\le m,m\le p;$$p\le n,p\le m;$ and $n\le p,p\le m$ the corresponding results can be proved similarly. By (10), (11) and (18) ${\mathbf{S}}_1,{\mathbf{S}}_2,\mathbf{M},{\mathbf{F}}_1,{\mathbf{F}}_2$ are taken as $S_1,S_2,M_1,F_1,F_2$, where $F_1$ and $F_2$ can be obtained by solving

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_1^H{S}_1^H{S}_1{\mathrm{\Pi }}_1{\mathbf{Q}}_i\right)\right|,\end{array}$$

(47)

$$\underset{{\mathrm{\Pi }}_2\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_2^H{S}_2^H{S}_2{\mathrm{\Pi }}_2{\mathbf{Q}}_i\right)\right|$$

for every $i=1,\dots ,n$, respectively. Let ${\mathbf{f}}_i$ be the i-th column of unitary matrix $F_1.$ By Lemma 1 we have for $i=1$

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}|tr\left({\mathrm{\Pi }}_1^H{S}_1^H{S}_1{\mathrm{\Pi }}_1{\mathbf{Q}}_1\right)|=|tr\left(F_1^H{S}_1^H{S}_1{F}_1{\mathbf{Q}}_1\right)|={\mathbf{f}}_1^H{S}_1^H{S}_1{\mathbf{f}}_1={\lambda }_1.\end{array}$$

(48)

Here ${\mathbf{f}}_1$ has some solutions and is set to satisfy $S_1^H{S}_1{\mathbf{f}}_1={\lambda }_1{\mathbf{f}}_1.$ Similarly, for $i>1$

$$\begin{array}{cc}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_1^H{S}_1^H{S}_1{\mathrm{\Pi }}_1{\mathbf{Q}}_i\right)\right|& =\left|tr\right(F_1^H{S}_1^H{S}_1{F}_1{\mathbf{Q}}_i\left)\right|\\ & ={\mathbf{f}}_1^H{S}_1^H{S}_1{\mathbf{f}}_1+\cdots +{\mathbf{f}}_i^H{S}_1^H{S}_1{\mathbf{f}}_i\\ & ={\lambda }_1+\cdots +{\lambda }_i.\end{array}$$

(49)

Here ${\mathbf{f}}_i$ has some solutions and is set to satisfy $S_1^H{S}_1{\mathbf{f}}_i={\lambda }_i{\mathbf{f}}_i.$ Since $F_1$ is unitary, then ${\mathbf{f}}_k^H{\mathbf{f}}_j=0$ and ${\mathbf{f}}_k^H{\mathbf{f}}_k=1,k\ne j;k,j=1,\dots ,n$. It follows that ${\mathbf{f}}_k^H{S}_1^H{S}_1{\mathbf{f}}_j={\lambda }_i{\mathbf{f}}_k^H{\mathbf{f}}_j=0$ and ${\mathbf{f}}_k^H{S}_1^H{S}_1{\mathbf{f}}_k={\lambda }_k{\mathbf{f}}_k^H{\mathbf{f}}_k={\lambda }_k,k\ne j;k,j=1,\dots ,n$. Then

$$\begin{array}{c}\hfill F_1^H{S}_1^H{S}_1{F}_1={\mathrm{\Lambda }}_1,\end{array}$$

(50)

where ${\mathrm{\Lambda }}_1=diag\left({\lambda }_i\right)$. Similarly,

$$F_2^H{S}_2^H{S}_2{F}_2^H={\mathrm{\Lambda }}_2,$$

where ${\mathrm{\Lambda }}_2=diag\left({\chi }_i\right)$. By Lemma 1 and (47) we have for $i=1$

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}|tr\left({\mathrm{\Pi }}_1^H{S}_1^H{S}_1{\mathrm{\Pi }}_1{\mathbf{Q}}_1\right)|=|tr\left(F_1^H{S}_1^H{S}_1{F}_1{\mathbf{Q}}_1\right)|={\lambda }_1=\underset{1\le i\le n}{max}{\lambda }_i.\end{array}$$

(51)

For other $i=2,\dots ,n$, we have

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}|tr\left({\mathrm{\Pi }}_1^H{S}_1^H{S}_1{\mathrm{\Pi }}_1{\mathbf{Q}}_i\right)|=|tr\left(F_1^H{S}_1^H{S}_1{F}_1{\mathbf{Q}}_i\right)|=|tr\left({\mathrm{\Lambda }}_1{\mathbf{Q}}_i\right)|={\lambda }_1+\cdots +{\lambda }_i.\end{array}$$

(52)

Then ${\mathrm{\Lambda }}_1=diag\left({\lambda }_i\right),{\mathrm{\Lambda }}_2=diag\left({\chi }_i\right)$ have descending sorts with ${\lambda }_i\ge 0,{\chi }_i\ge 0$, otherwise, there will be conflicts with (51) and (52). Then by Lemma 1 we have

$$\begin{array}{cc}\hfill \underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)& =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left({\mathrm{\Phi }}^H{S}_2^H{S}_2\mathrm{\Phi }{S}_1^H{S}_1\right)\\ & =\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left({\mathrm{\Phi }}^H{F}_2{\mathrm{\Lambda }}_2{F}_2^H\mathrm{\Phi }{F}_1{\mathrm{\Lambda }}_1{F}_1^H\right)=\mathrm{tr}\left({\mathrm{\Lambda }}_2{\mathrm{\Lambda }}_1\right).\end{array}$$

(53)

It is easy to check that $\mathrm{tr}\left(M_1\left(F_2{F}_1^H\right)\right)=\mathrm{tr}\left({\mathrm{\Lambda }}_2{\mathrm{\Lambda }}_1\right),$ which together with (53) implies $\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}\mathrm{tr}\left(M_1\left(\mathrm{\Phi }\right)\right)=\mathrm{tr}\left(M_1\left(F_2{F}_1^H\right)\right).$ For other cases for $n\le m,m\le p;$$p\le n,p\le m;$ and $n\le p,p\le m;$ the desired corresponding results can be proved similarly. The proof is complete.    □

Theorem 3. 

Let $\mathbf{M}\left(\mathrm{\Phi }\right),{\mathbf{Q}}_i,{\mathbf{S}}_1,{\mathbf{S}}_2$ be given by (10), (11) and (14). Let $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ be two $(m,p,n)$-GMPs and $\sigma \{A,B\}=\{({\alpha }_i,{\beta }_i),i=1,2\dots ,n\}$ and $\sigma \{\tilde{A},\tilde{B}\}=\{({\tilde{\alpha }}_j,{\tilde{\beta }}_j),j=1,2\dots ,n\}.$ Then

$$\begin{array}{ccc}& & \sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))\hfill \\ & =& tr({\mathbf{S}}_1^H{\mathbf{S}}_1+{\mathbf{S}}_2^H{\mathbf{S}}_2)-2\underset{\mathrm{\Phi }\in {\mathbb{U}}_{min\{m,p,n\}}}{max}tr\left(\mathbf{M}\left(\mathrm{\Phi }\right)\right)-2tr\left({\mathbf{L}}_1^{{\displaystyle \frac{1}{2}}}{(I_{min\{m,p,n\}}-{\mathbf{L}}_1)}^{{\displaystyle \frac{1}{2}}}{\mathbf{L}}_2^{{\displaystyle \frac{1}{2}}}{(I_{min\{m,p,n\}}-{\mathbf{L}}_2)}^{{\displaystyle \frac{1}{2}}}\right)\hfill \\ & =& tr({\mathbf{S}}_1^H{\mathbf{S}}_1+{\mathbf{S}}_2^H{\mathbf{S}}_2)-2tr\left(\mathbf{M}\left({\mathbf{F}}_2{\mathbf{F}}_1^H\right)\right)-2tr\left({\mathbf{L}}_1^{{\displaystyle \frac{1}{2}}}{(I_{min\{m,p,n\}}-{\mathbf{L}}_1)}^{{\displaystyle \frac{1}{2}}}{\mathbf{L}}_2^{{\displaystyle \frac{1}{2}}}{(I_{min\{m,p,n\}}-{\mathbf{L}}_2)}^{{\displaystyle \frac{1}{2}}}\right),\hfill \end{array}$$

where

$${\mathbf{L}}_1={\mathbf{F}}_1^H{\mathbf{S}}_1^H{\mathbf{S}}_1{\mathbf{F}}_1,{\mathbf{L}}_2={\mathbf{F}}_2^H{\mathbf{S}}_2^H{\mathbf{S}}_2{\mathbf{F}}_2$$

and ${\mathbf{F}}_1,{\mathbf{F}}_2\in {\mathbb{U}}_{min\{m,p,n\}}$ can be obtained from ${\mathrm{\Pi }}_1,{\mathrm{\Pi }}_2$ by solving the trace function optimization models (45) and (46).

Proof. 

We first prove the case when $m\le n,m\le p$ and for others cases when $n\le m,m\le p;$$p\le n,p\le m;$ and $n\le p,p\le m$ the corresponding results can be proved similarly. By (10), (11) and (18) ${\mathbf{S}}_1,{\mathbf{S}}_2,\mathbf{M},{\mathbf{F}}_1,{\mathbf{F}}_2$ can be taken as $S_1,S_2,M_1,F_1,F_2$. Since $F_1$ and $F_2$ are solutions of (45) and (46) for every $i=1,\dots ,n$, respectively, then $S_1^H{S}_1=A{(A^{H}A+B^{H}B)}^{-1}{A}^H=U{\mathrm{\Sigma }}_A{\mathrm{\Sigma }}_A^H{U}^H$. For $i=1$ by (45) and Lemma 1 we have

$$\underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}|tr\left({\mathrm{\Pi }}_1^{H}A{(A^{H}A+B^{H}B)}^{-1}{A}^H{\mathrm{\Pi }}_1{Q}_1\right)|=|tr\left(F_1^{H}A{(A^{H}A+B^{H}B)}^{-1}{A}^H{F}_1{Q}_1\right)|={\alpha }_1,$$

For $i=2$ we have

$$\underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}|tr\left({\mathrm{\Pi }}_1^{H}A{(A^{H}A+B^{H}B)}^{-1}{A}^H{\mathrm{\Pi }}_1{Q}_i\right)|=|tr\left(F_1^{H}A{(A^{H}A+B^{H}B)}^{-1}{A}^H{F}_1{Q}_2\right)|={\alpha }_1+{\alpha }_2.$$

For $i>2$, we have

$$\underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_1^{H}A{(A^{H}A+B^{H}B)}^{-1}{A}^H{\mathrm{\Pi }}_1{Q}_i\right)\right|={\alpha }_1+\cdots +{\alpha }_i.$$

Hence, we have $L_1=F_1^H{S}_1^H{S}_1{F}_1=diag\left({\alpha }_i\right),L_2=F_2^H{S}_2^H{S}_2{F}_2=diag\left({\tilde{\alpha }}_i\right)$. It follows that

$$\sum _{i=1}^n{\alpha }_i{\beta }_i{\tilde{\alpha }}_i{\tilde{\beta }}_i=tr\left(L_1^{{\displaystyle \frac{1}{2}}}{(I_m-L_1)}^{{\displaystyle \frac{1}{2}}}{L}_2^{{\displaystyle \frac{1}{2}}}{(I_m-L_2)}^{{\displaystyle \frac{1}{2}}}\right),$$

which together with (25) and Lemma 6 yields

$$\begin{array}{ccc}& & \sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))=n-\sum _{i=1}^n({\alpha }_i^2{\tilde{\alpha }}_i^2+{\beta }_i^2{\tilde{\beta }}_i^2)-2\sum _{i=1}^n{\alpha }_i{\beta }_i{\tilde{\alpha }}_i{\tilde{\beta }}_i,\hfill \\ & =& tr(S_1^H{S}_1+S_2^H{S}_2)-2\underset{\mathrm{\Phi }\in {\mathbb{U}}_m}{max}tr\left(M_1\left(\mathrm{\Phi }\right)\right)-2tr\left(L_1^{{\displaystyle \frac{1}{2}}}{(I_m-L_1)}^{{\displaystyle \frac{1}{2}}}{L}_2^{{\displaystyle \frac{1}{2}}}{(I_m-L_2)}^{{\displaystyle \frac{1}{2}}}\right),\hfill \\ & =& tr(S_1^H{S}_1+S_2^H{S}_2)-2tr\left(M_1\left(F_2{F}_1^H\right)\right)-2tr\left(L_1^{{\displaystyle \frac{1}{2}}}{(I_m-L_1)}^{{\displaystyle \frac{1}{2}}}{L}_2^{{\displaystyle \frac{1}{2}}}{(I_m-L_2)}^{{\displaystyle \frac{1}{2}}}\right).\hfill \end{array}$$

For the cases when $n\le m,m\le p;$$p\le n,p\le m;$ and $n\le p,p\le m;$ by the similar aforementioned methods the desired results can be proved.    □

Remark 1. 

From Theorem 3 we see that the formula of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ in (3) only refers to two small size unitary matrices ${\mathbf{F}}_1\in {\mathbb{U}}_{min\{m,p,n\}},{\mathbf{F}}_2\in {\mathbb{U}}_{min\{m,p,n\}}$, which may require less computational cost for computing the explicit expression in Theorem 3.

When using Theorem 3 we first determine ${\mathbf{S}}_1,{\mathbf{S}}_2$ by comparing $m,p,n$. Then by using (45) and (46) ${\mathbf{F}}_1,{\mathbf{F}}_2$ can be computed.

2.2. Solving Riemannian Optimization Models in (45) and (46) for Complex Grassman Matrix Pair

We will use Newton methods on Riemannian manifolds to solve Riemannian optimization models in (45) and (46) of Theorem 3.

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_1{\mathbf{S}}_1^H{\mathbf{S}}_1{\mathrm{\Pi }}_1^H{\mathbf{Q}}_i\right)\right|,\end{array}$$

(54)

for $n\le m,n\le p$ and $m\le n,m\le p$ and

$$\begin{array}{c}\hfill \underset{{\mathrm{\Pi }}_2\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_2{\mathbf{S}}_1^H{\mathbf{S}}_1{\mathrm{\Pi }}_2^H{\mathbf{Q}}_i\right)\right|,\end{array}$$

(55)

for $p\le n,p\le m$ and $n\le p,p\le m.$ Solving $\underset{{\mathrm{\Pi }}_1\in {\mathbb{U}}_{min\{m,p,n\}}}{max}\left|tr\left({\mathrm{\Pi }}_1{\mathbf{S}}_2^H{\mathbf{S}}_2{\mathrm{\Pi }}_1^H{\mathbf{Q}}_i\right)\right|$ can be similarly discussed. For simplicity, we assume that $n\le m$ and $n\le p$ for (54) and $n\le p,p\le m$ for $\left(\right)$ and other cases for different relationships between $m,n,p$ can be similarly discussed. To compute (54) and (55), we consider the following matrix optimization problems:

$$\begin{array}{cc}max& {\displaystyle f_1\left({\mathrm{\Phi }}_1\right):=\frac{1}{2}\mathrm{tr}\left({\mathrm{\Phi }}_1^H{\mathbf{Q}}_i{\mathrm{\Phi }}_{1}C\right)}\\ \mathrm{s}.\mathrm{t}.& {\mathrm{\Phi }}_1\in {\mathbb{U}}_n\end{array}$$

(56)

and

$$\begin{array}{cc}max& {\displaystyle f_2\left({\mathrm{\Phi }}_2\right):=\frac{1}{2}\mathrm{tr}\left({\mathrm{\Phi }}_2^H{\mathbf{Q}}_i{\mathrm{\Phi }}_{2}D\right)}\\ \mathrm{s}.\mathrm{t}.& {\mathrm{\Phi }}_2\in {\mathbb{U}}_n\end{array}$$

(57)

for $1\le i\le n$. We only need to focus on the following matrix optimization problems:

$$\begin{array}{cc}max& {\displaystyle f_1\left({\mathrm{\Phi }}_{11}\right):=\frac{1}{2}\mathrm{tr}\left({\mathrm{\Phi }}_{11}^{H}C{\mathrm{\Phi }}_{11}\right)}\\ \mathrm{s}.\mathrm{t}.& {\mathrm{\Phi }}_{11}\in St(n,i):=\{{\mathrm{\Phi }}_{11}\in {\mathbb{C}}^{n\times i}|{\mathrm{\Phi }}_{11}^H{\mathrm{\Phi }}_{11}=I_i\},\end{array}$$

(58)

$$\begin{array}{cc}max& {\displaystyle f_2\left({\mathrm{\Phi }}_{22}\right):=\frac{1}{2}\mathrm{tr}\left({\mathrm{\Phi }}_{22}^{H}D{\mathrm{\Phi }}_{22}\right)}\\ \mathrm{s}.\mathrm{t}.& {\mathrm{\Phi }}_{22}\in St(n,n-i+1),\end{array}$$

(59)

where $C={(A^{H}A+B^{H}B)}^{-1/2}{A}^{H}A{(A^{H}A+B^{H}B)}^{-1/2}$, $D={(A^{H}A+B^{H}B)}^{-1/2}{B}^{H}B{(A^{H}A+B^{H}B)}^{-1/2}$, ${\mathrm{\Phi }}_{11}$ is the first i columns of ${\mathrm{\Phi }}_1^H$ and ${\mathrm{\Phi }}_{22}$ is the last $n-i+1$ columns of ${\mathrm{\Phi }}_2^H$. In the following, we focus on the solution of the matrix optimization problem (58) and the matrix optimization problem (59) can be solved in a similar way.

We note that $f_1\left({\mathrm{\Phi }}_{11}Q\right)=f_1\left({\mathrm{\Phi }}_{11}\right)$ for all ${\mathrm{\Phi }}_{11}\in St(n,i)$ and $Q\in {\mathbb{U}}_i$. Let the complex Grassmann manifold $\mathrm{Grass}(n,i)$ be the set of all i-dimensional complex subspace of ${\mathbb{C}}^n$. If $\left[X\right]$ means the subspace spanned by the columns of $X\in St(n,i)$, then we have $\left[X\right]\in \mathrm{Grass}(n,i)$. In particular, for any $X\in St(n,i)$, the natural projection $\left[X\right]\in \mathrm{Grass}(n,i)$ corresponds the equivalent class $\{XQ\in St(n,i)|Q\in {\mathbb{U}}_i\}$ of $St(n,i)$. Thus, instead of problem (58), we consider the following optimization problem:

$$\begin{array}{cc}max& {\displaystyle {\tilde{f}}_1\left(\left[{\mathrm{\Phi }}_{11}\right]\right):=f_1\left({\mathrm{\Phi }}_{11}\right)}\\ \mathrm{s}.\mathrm{t}.& \left[{\mathrm{\Phi }}_{11}\right]\in \mathrm{Grass}(n,i).\end{array}$$

(60)

Next, we present Newton’s method for solving the optimization problem (60). Let ${\mathrm{\Phi }}_{11}\in St(n,i)$. The tangent space of $St(n,i)$ at ${\mathrm{\Phi }}_{11}$ is given by [29]

$$T_{{\mathrm{\Phi }}_{11}}St(n,i)=\{Z\in {\mathbb{C}}^{n\times i}|Z={\mathrm{\Phi }}_{11}\Omega +{\left({\mathrm{\Phi }}_{11}\right)}_{\perp }K,{\Omega }^H=-\Omega ,\Omega \in {\mathbb{C}}^{i,i},K\in {\mathbb{C}}^{n-i,i}\},$$

which can be endowed with the inner product

$$\langle Z_1,Z_2\rangle =\mathfrak{R}\left[\mathrm{tr}\left(Z_2^H{(I-{\displaystyle \frac{1}{2}}{\mathrm{\Phi }}_{11}{\mathrm{\Phi }}_{11})}^H\right)Z_1\right)],Z_1,Z_2\in T_{{\mathrm{\Phi }}_{11}}St(n,i),{\mathrm{\Phi }}_{11}\in St(m,i).$$

Here, ${\left({\mathrm{\Phi }}_{11}\right)}_{\perp }$ means that span$\left({\left({\mathrm{\Phi }}_{11}\right)}_{\perp }\right)$ is the orthogonal complement of span$\left({\mathrm{\Phi }}_{11}\right)$, where span$\left({\mathrm{\Phi }}_{11}\right)$ denotes a linear space spanned by the column vectors of ${\mathrm{\Phi }}_{11}$. We note that the tangent space of $\mathrm{Grass}(n,i)$ at $\left[{\mathrm{\Phi }}_{11}\right]$ is given by [29]

$$T_{\left[{\mathrm{\Phi }}_{11}\right]}\mathrm{Grass}(n,i)=\{Z\in {\mathbb{C}}^{n\times i}|Z={\left({\mathrm{\Phi }}_{11}\right)}_{\perp }K,K\in {\mathbb{C}}^{n-i,i}\}\subset T_{{\mathrm{\Phi }}_{11}}St(n,i).$$

Hence, we can define a Riemannian metric on $\mathrm{Grass}(n,i)$ by

$$\langle Z_1,Z_2\rangle =\mathfrak{R}\left[\mathrm{tr}\left(Z_2^H{Z}_1\right)\right],Z_1,Z_2\in T_{\left[{\mathrm{\Phi }}_{11}\right]}\mathrm{Grass}(n,i),{\mathrm{\Phi }}_{11}\in St(n,i)$$

with the induced norm $\parallel ·\parallel$. Then the orthogonal projection onto $T_{\left[{\mathrm{\Phi }}_{11}\right]}\mathrm{Grass}(m,i)$ is given by

$$P_{{\mathrm{\Phi }}_{11}}Z=(I-{\mathrm{\Phi }}_{11}{\mathrm{\Phi }}_{11}^H)Z,\forall Z\in {\mathbb{C}}^{n\times i}.$$

We define the local cost function $g:T_{\left[{\mathrm{\Phi }}_{11}\right]}\mathrm{Grass}(n,i)\to \mathbb{R}$ by

$$g\left(Z\right)={\tilde{f}}_1\left([{\mathrm{\Phi }}_{11}+Z]\right).$$

It is easy to check that, for any $Z\in T_{\left[{\mathrm{\Phi }}_{11}\right]}\mathrm{Grass}(n,i)$,

$$\begin{array}{ccc}\hfill g\left(Z\right)& =& f_1\left({\mathrm{\Phi }}_{11}\right)+\mathfrak{R}\mathrm{tr}\left(Z^H{D}_{{\mathrm{\Phi }}_{11}}\right)\hfill \\ & & +{\displaystyle \frac{1}{2}}\mathrm{vec}{\left(Z\right)}^H\left(H_{{\mathrm{\Phi }}_{11}}-{\left({\mathrm{\Phi }}_{11}^H{D}_{{\mathrm{\Phi }}_{11}}\right)}^T\otimes I_n\right)\mathrm{vec}\left(Z\right),\hfill \end{array}$$

where $D_{{\mathrm{\Phi }}_{11}}=C{\mathrm{\Phi }}_{11}$ and $H_{{\mathrm{\Phi }}_{11}}=I_i\otimes C$ are the derivative and Hessian of $f_1$ at ${\mathrm{\Phi }}_{11}$, respectively.

Based on the above analysis, Newton’s method for solving the optimization problem (60) can be described as follows [29].

Algorithm 4. 

Step 0. Choose ${\mathrm{\Phi }}_{11}^0\in St(n,i)$, $\rho ,\eta \in (0,1)$, $\sigma \in (0,1/2]$ and let $k:=1$.

Step 1.

Apply the conjugate gradient (CG) method to solving

$$P_{{\mathrm{\Phi }}_{11}^k}(C{Z}^k-Z^k{\left({\mathrm{\Phi }}_{11}^k\right)}^H{D}_{{\mathrm{\Phi }}_{11}^k})=-P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k}$$

for$Z^k\in T_{\left[{\mathrm{\Phi }}_{11}^k\right]}\mathrm{Grass}(n,i)$such that

$$\parallel P_{{\mathrm{\Phi }}_{11}^k}(C{Z}^k-Z^k{\left({\mathrm{\Phi }}_{11}^k\right)}^H{D}_{{\mathrm{\Phi }}_{11}^k})+P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k}\parallel \le {\eta }_k\parallel P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k}\parallel$$

(61)

and

$$\langle P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k},Z^k\rangle \le -{\eta }_k\langle Z^k,Z^k\rangle ,$$

(62)

where ${\eta }_k=min\{\eta ,\parallel P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k}\parallel \}$. If (61) and (62) are not attainable, then let

$$Z^k=-P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k}.$$

Step 2.

Let $l_k>0$ be the smallest integer m such that

$$f_1\left(\pi ({\mathrm{\Phi }}_{11}^k+{\beta }^l{Z}^k)\right)\le f_1\left({\mathrm{\Phi }}_{11}^k\right)+\sigma {\rho }^l\langle P_{{\mathrm{\Phi }}_{11}^k}{D}_{{\mathrm{\Phi }}_{11}^k},Z^k\rangle$$

Set 

$$Z^{k+1}=\pi ({\mathrm{\Phi }}_{11}^k+{\beta }^l{Z}^k).$$

Step 3.

Replace k by $k+1$ and go to Step 1.

In Step 2 of Algorithm 4, $\pi :{\mathbb{C}}^{m\times i}\to \mathrm{Grass}(n,i)$ is the projection onto $\mathrm{Grass}(n,i)$, which is defined as follows: Let $X\in {\mathbb{C}}^{n\times i}$ be full column rank. Then

$$\pi \left(X\right)=\left[\underset{Y\in St(n,i)}{argmin}{\parallel X-Y\parallel }^2\right].$$

As noted in [29], if the SVD of X is given by $X=U\mathrm{\Sigma }{V}^H$, then $\pi \left(X\right)=U{I}_{n,i}{V}^H$. If the QR decomposition of X is $X=QR$, then $\pi \left(X\right)=Q_{n,i}$.

Remark 2.

(i) An explicit expression of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ is given in Theorem 3. As a comparison, Theorem 2 only provides an upper bound. Obviously, Theorem 3 improves the existing results in [22] in theory, see Example below. We adopt trace function optimization formula with orthogonal constraints in Theorem 3 to evaluate the explicit expression of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$, which costs less complexity. While Theorem 3.2 in [22] needs unitary matrices $U,V,\tilde{U},\tilde{V}$ of the GSVDs of two matrix pairs $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$. It is known that computing GSVDs may cost much higher amount of calculations than computing small-size unitary matrice. Hence, on algorithm aspect Theorem 3 also improves Theorem 3.2 in [22].

(ii) The equality in Theorem 3 refers to matrix optimization with orthogonal constraints. Based on matrix optimization formula with orthogonal constraints we use classical Golub-Kahan bidiagonalization method to solve it efficiently. The proposed algorithms by trace function optimization under two variables are efficient for computing for $A^{H}A+B^{H}B$ being not ill-posed. For $A^{H}A+B^{H}B=W$ being ill-posed, by matrix decompositions (eg., Schur decomposition or QR decomposition) we compute $W_1,W_2$ with $W=W_1{W}_2$, where $W_1,W_2$ have small condition numbers. We calculate $W_1^{-1},W_2^{-1}$ and then compute multiplications for $W^{-1}=W_2^{-1}{W}_1^{-1}$, see [27,28]. In general we first compute the inverse of the matrix with small condition numbers and then multiply these two inverse matrices, which can well get the inverse of the original matrix with large condition numbers. The proposed algorithms by trace function optimization under two variables are efficient for computing for condition numbers of $A^{H}A+B^{H}B$ being large, see Table 1. In practical applications, obviously, by the new explicit expression formula getting exact values of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ are more precise than its upper bound.

Example. To test the comparison of the explicit formula in Theorem 3, and the upper bound of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ in [22], we randomly generate grassman matrix pencils $\{A,B\}$ and $\{\tilde{A},\tilde{B}\}$ by MATLAB command randn(m,n)+i*randn(m,n) for $A,\tilde{A}$ and randn(p,n)+i*randn(p,n) for $B,\tilde{B}$ with different random $m,n,p$. We first assess condition numbers of $A^{H}A+B^{H}B$ by using two-norm and denoted by ${\kappa }_2(·)$. By simple computations we present the table 1.

Table 1. Comparisons of exact values, explicit formula by Alg. 4 and the upper bound with different $(m,p,n)$

$(\mathit{m},\mathit{p},\mathit{n})$	${\mathit{\kappa }}_2(·)$	Exact values	formula by Alg. 4	Upper Bound
(80,40,60)	30.2	27.24709031	$27.24698583$	41.3510
(200,500,450)	218.6	$189.26904380$	189.26910247	380.4629
(900,800,700)	359.4	$458.42617335$	458.42618210	573.0744
(60,120,140)	$600.3$	103.56173621	103.56172847	126.0846
(100,200,150)	$400.6$	79.35162440	79.35157629	118.5329
(250,500,450)	$O\left({10}^3\right)$	268.49009138	268.49008938	376.4290
(800,600,500)	$O\left({10}^3\right)$	409.05215925	409.05215570	454.7393
(2000,1900,1800)	$O\left({10}^4\right)$	1516.38636811	1516.38632996	1704.1106
(500,600,800)	$O\left({10}^5\right)$	379.23164800	379.23164655	634.2572

Table 1. Comparisons of exact values, explicit formula by Alg. 4 and the upper bound with different $(m,p,n)$

$(\mathit{m},\mathit{p},\mathit{n})$	${\mathit{\kappa }}_2(·)$	Exact values	formula by Alg. 4	Upper Bound
(80,40,60)	30.2	27.24709031	$27.24698583$	41.3510
(200,500,450)	218.6	$189.26904380$	189.26910247	380.4629
(900,800,700)	359.4	$458.42617335$	458.42618210	573.0744
(60,120,140)	$600.3$	103.56173621	103.56172847	126.0846
(100,200,150)	$400.6$	79.35162440	79.35157629	118.5329
(250,500,450)	$O\left({10}^3\right)$	268.49009138	268.49008938	376.4290
(800,600,500)	$O\left({10}^3\right)$	409.05215925	409.05215570	454.7393
(2000,1900,1800)	$O\left({10}^4\right)$	1516.38636811	1516.38632996	1704.1106
(500,600,800)	$O\left({10}^5\right)$	379.23164800	379.23164655	634.2572

Here `Explicit expression’ denotes the equality in Theorem 3, `Upper Bound’ denotes the upper bound in [22]. This table obviously shows the equality in Theorem 3 is almost equal to exact value of $\sum _{i=1}^n{\rho }^2(({\alpha }_i,{\beta }_i),({\tilde{\alpha }}_i,{\tilde{\beta }}_i))$ and may be much sharper than the upper bounds in [22]. This again verifies Remark (i).

3. Concluding Remarks

In this paper we give explicit expressions of sum of chordal metric between generalized singular values of Grassmann matrix pairs, which absolutely improves the existing upper bounds in [22]. For the new expression formula only two small-size singular value decompositions are required in the algorithm which dominates the computational cost. As a byproduct, we also present an interval estimate without using the singular values.