A New Proof Of Eckart-Young-Mirsky Theorem

1. Introduction

The Eckart–Young–Mirsky theorem is a fundamental result in matrix approximation, stating that for a given matrix A and rank k, the best rank-k approximation in the Frobenius norm (or any unitarily invariant norm) is obtained by truncating the Singular Value Decomposition (SVD) of A.[1,3] Formally, if

$$A=U\Sigma V^T$$

is the SVD of A and ${\Sigma }_k$ is obtained from $\Sigma$ by keeping only the k largest singular values (and setting the rest to zero), then

$$A_k=U{\Sigma }_k{V}^T$$

is the unique minimizer of $\parallel A-X\parallel$ over all rank-k matrices X [2,4].

This theorem underpins numerous applications, from image compression to principal component analysis, yet standard proofs often rely on variational arguments or operator norm inequalities that can obscure geometric intuition [5]. In this paper, we present a more elementary proof(only using basic linear algebraic).

2. Proof

In this section, we will give an elementary and short proof of the Eckart-Young-Mirsky Theorem.

Let A be a real matrix with $\mathrm{rank}\left(A\right)=r$ and ${\sigma }_1\ge {\sigma }_2\ge ...\ge {\sigma }_r>0$ in a descending order be all the non-zero singular values of A. The SVD factors A into

$$A=U\Sigma V^t=U\left(\begin{array}{cc}\Lambda & 0\\ 0& 0\end{array}\right)V^t$$

where U and V are orthogonal matrices and $\Lambda =\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_r)$ is a $r\times r$ diagonal matrix.

Let $0<k<r$ be an integer. Define

$$A_k=U{\Sigma }_k{V}^t=U\left(\begin{array}{cc}{\Lambda }_k& 0\\ 0& 0\end{array}\right)V^t$$

with ${\Lambda }_k=\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_k,0,\dots ,0)$ being a $r\times r$ diagonal matrix.

The Eckart-Young-Mirsky Theorem states that

$${∥A-A_k∥}_F=\sqrt{\sum _{i=k+1}^r{\sigma }_i^2}=\underset{\mathrm{rank}\left(X\right)=k}{min}{\parallel A-X\parallel }_F$$

(1)

where ${\parallel ·\parallel }_F$ is the Frobenius norm defined by

$${\parallel A\parallel }_F=\sqrt{\mathrm{trace}\left(A^{t}A\right)}=\sqrt{\sum _{i=1}^m\sum _{j=1}^n{\left|a_{ij}\right|}^2}.$$

(2)

for any real matrix $A={\left(a_{ij}\right)}_{m\times n}$.

In the following, we will relax the condition $\mathrm{rank}\left(X\right)=k$ to $\mathrm{rank}\left(X\right)\le k$ and prove that

$$\underset{\mathrm{rank}\left(X\right)\le k}{min}{\parallel A-X\parallel }_F=\sqrt{\sum _{i=k+1}^r{\sigma }_i^2}.$$

(3)

Let

$$X=UY{V}^t=U\left(\begin{array}{cc}M& *\\ & *\end{array}\right)V^t$$

with M being a $r\times r$ matrix of $\mathrm{rank}\left(M\right)\le k$. By (2),

$${\parallel A-X\parallel }_F=\parallel U(\Sigma -Y)V^t{\parallel }_F={\parallel \Sigma -Y\parallel }_F\ge {\parallel \Lambda -M\parallel }_F.$$

Therefore, to show (3), it suffices to prove

$${\parallel \Lambda -M\parallel }_F\ge \sqrt{\sum _{i=k+1}^r{\sigma }_i^2}.$$

(4)

as $X=A_k$ achieves the minimum.

Fix a k-dimensional subspace $W\subset {\mathbb{R}}^r$ such that the column vectors of M lie in W. Choose an orthonormal basis $v_1,v_2,...,v_p,v_{p+1},...,v_r$ of ${\mathbb{R}}^r$ such that $v_{p+1},...,v_r$ span W, where $p+k=r$. Let

$$\Lambda =({\sigma }_1{e}_1,{\sigma }_2{e}_2,\dots ,{\sigma }_r{e}_r)\mathrm{and}M=(w_1,w_2,\dots ,w_r)$$

where ${\sigma }_i{e}_i$-s and $w_i$-s are column vectors of $\Lambda$ and M, respectively. We have

$${∥\Lambda -M∥}_F^2=\sum _{i=1}^r{∥{\sigma }_i{e}_i-w_i∥}^2.$$

(5)

To minimize (5), $w_i$ should be the projection of ${\sigma }_i{e}_i$ onto W, i.e.,

$${\sigma }_i{e}_i-w_i=\sum _{j=1}^p〈{\sigma }_i{e}_i,v_j〉v_j$$

where $〈,〉$ is the standard inner product. Then for any M whose column vectors are in W,

$$min{∥\Lambda -M∥}_F^2=\sum _{i=1}^r\sum _{j=1}^p{〈e_i,v_j〉}^2{\sigma }_i^2.$$

(6)

The coefficients of ${\sigma }_i^2$-s of (6) satisfy:

$$\begin{array}{cc}\hfill 0\le & \sum _{j=1}^p{〈e_i,v_j〉}^2\le 〈e_i,e_i〉\le 1;\hfill \\ \hfill \sum _{i=1}^r& \sum _{j=1}^p{〈e_i,v_j〉}^2=\sum _{j=1}^p\sum _{i=1}^r{〈e_i,v_j〉}^2=p.\hfill \end{array}$$

Since ${\sigma }_1^2\ge {\sigma }_2^2\ge ...\ge {\sigma }_r^2>0$ are in descending order and their coefficients all belong to $[0,1]$ with the sum being p, to minimize the right hand side of (6), the coefficients should concentrate to the lowest p singular values. Therefore,

$$\sum _{i=1}^r\sum _{j=1}^p{〈e_i,v_j〉}^2{\sigma }_i^2\ge \sum _{i=k+1}^r{\sigma }_i^2.$$

(7)

3. Conclusion

This paper offers an elementary yet powerful proof of the Eckart-Young-Mirsky theorem, which is essential for many fields, such as machine learning, image processing, and data science. By demonstrating the best rank-k approximation through a clear application of basic linear algebra techniques, the paper contributes to a deeper understanding of low-rank matrix approximation. This work simplifies the theorem’s proof, making it more accessible for those familiar with basic matrix theory and reinforcing its crucial role in real-world applications like dimensionality reduction, data compression, and statistical analysis.

Combining (6) and (7), we get (4), which concludes the proof.