Continuous Donoho-Elad Spark Uncertainty Principle

1. Introduction

Let $\mathcal{H}$ be a finite dimensional Hilbert space over $\mathbb{K}$ ($\mathbb{C}$ or $\mathbb{R}$). Recall that [1] a collection of nonzero elements ${\left\{{\tau }_j\right\}}_{j=1}^n$ in $\mathcal{H}$ is said to be a frame (also known as dictionary) $\mathcal{H}$ if there are $r,s>0$ such that

$$\begin{array}{c}\hfill {r\parallel h\parallel }^2\le \sum _{j=1}^n|\langle h,{\tau }_j\rangle {|}^2\le s{\parallel h\parallel }^2,\forall h\in \mathcal{H}.\end{array}$$

It is well-known that a collection ${\left\{{\tau }_j\right\}}_{j=1}^n$ in $\mathcal{H}$ is a frame for $\mathcal{H}$ if and only if ${\left\{{\tau }_j\right\}}_{j=1}^n$ spans $\mathcal{H}$[2]. A frame ${\left\{{\tau }_j\right\}}_{j=1}^n$ for $\mathcal{H}$ is said to be normalized if $\parallel {\tau }_j\parallel =1$ for all $1\le j\le n$. Note that any frame can be normalized by dividing each element by its norm. Given a frame ${\left\{{\tau }_j\right\}}_{j=1}^n$ for $\mathcal{H}$, we define the analysis operator

$$\begin{array}{c}\hfill {\theta }_{\tau }:\mathcal{H}\ni h\mapsto {\theta }_{\tau }h{\left(\langle h,{\tau }_j\rangle \right)}_{j=1}^n\in {\mathbb{K}}^n.\end{array}$$

Adjoint of the analysis operator is known as the synthesis operator whose equation is

$$\begin{array}{c}\hfill {\theta }_{\tau }^{*}:{\mathbb{K}}^n\ni {\left(a_j\right)}_{j=1}^n\mapsto {\theta }_{\tau }^{*}{\left(a_j\right)}_{j=1}^n\sum _{j=1}^n{a}_j{\tau }_j\in \mathcal{H}.\end{array}$$

Given $d\in {\mathbb{K}}^n$, let ${\parallel d\parallel }_0$ be the number of nonzero entries in d. Central problem which occurs in many situations is the following ${\ell }_0$-minimization problem:

Problem 1.

Let ${\left\{{\tau }_j\right\}}_{j=1}^n$ be a normalized frame for $\mathcal{H}$. Given $h\in \mathcal{H}$, solve

$$\begin{array}{c}\hfill \underset{d\in {\mathbb{K}}^n}{minimize}{\parallel d\parallel }_0\mathrm{subject}\mathrm{to}{\theta }_{\tau }^{*}d=h.\end{array}$$

Recall that $c\in {\mathbb{K}}^n$ is said to be a unique solution to Problem 1 if it satisfies following two conditions.

(i)

${\theta }_{\tau }^{*}c=h$.

(ii)

If $d\in {\mathbb{K}}^n$ satisfies ${\theta }_{\tau }^{*}d=h$, then

$$\begin{array}{c}\hfill {\parallel d\parallel }_0>{\parallel c\parallel }_0.\end{array}$$

In 1995, Natarajan showed that Problem 1 is NP-Hard [3]. Therefore solution to Problem 1 has to be obtained using other methods. Work which is built around Problem 1 is known as sparseland (term due to Elad [4]) or compressive sensing or compressed sensing.

As the operator ${\theta }_{\tau }^{*}$ is surjective, for a given $h\in \mathcal{H}$, there is always a $d\in {\mathbb{K}}^n$ such that ${\theta }_{\tau }^{*}d=h.$ Thus the central problem is when solution to Problem 1 is unique. One of the greatest results of Donoho and Elad [5] with regard to this is using the notion of spark defined as follows. In the paper, given a subset $M\subseteq \mathbb{N}$, the cardinality of M is denoted by $o\left(M\right)$.

Definition 1.

[5] Given a normalized frame ${\left\{{\tau }_j\right\}}_{j=1}^n$ for $\mathcal{H}$, thesparkof ${\left\{{\tau }_j\right\}}_{j=1}^n$ is defined as

$$\begin{array}{cc}\hfill Spark\left({\left\{{\tau }_j\right\}}_{j=1}^n\right)& min\{o\left(M\right):M\subseteq \{1,\dots ,n\},{\left\{{\tau }_j\right\}}_{j\in M}\mathrm{is}\mathrm{linearly}\mathrm{dependent}\}\hfill \\ \hfill & =min\{\parallel d{\parallel }_0:d\in ker\left({\theta }_{\tau }^{*}\right),d\ne 0\}.\hfill \end{array}$$

In 2003, Donoho and Elad derived the following breakthrough spark uncertainty principle [5].

Theorem 2.

[5] (Donoho-Elad Spark Uncertainty Principle) Let ${\left\{{\tau }_j\right\}}_{j=1}^n$ be a normalized frame for $\mathcal{H}$. If $a,b\in {\mathbb{K}}^n$ are distinct and ${\theta }_{\tau }^{*}a={\theta }_{\tau }^{*}b$, then

$$\begin{array}{c}\hfill {\parallel a\parallel }_0+{\parallel b\parallel }_0\ge Spark\left({\left\{{\tau }_j\right\}}_{j=1}^n\right).\end{array}$$

In the same paper [5], Donoho and Elad also gave a characterization for the solution of Problem 1 using spark.

Theorem 3.

[5,6] (Donoho-Elad Spark Sparsity Theorem) Let ${\left\{{\tau }_j\right\}}_{j=1}^n$ be a normalized frame for $\mathcal{H}$.

(i)

For every $h\in \mathcal{H}$ and every $1\le k\le n$, there exists atmost one vector $c\in {\mathbb{K}}^n$ such that

$$\begin{array}{c}\hfill h={\theta }_{\tau }^{*}c\mathrm{satisfying}{\parallel c\parallel }_0\le k\end{array}$$

if and only if

$$\begin{array}{c}\hfill Spark\left({\left\{{\tau }_j\right\}}_{j=1}^n\right)>2k.\end{array}$$

(ii)

If $h\in \mathcal{H}$ can be written as $h={\theta }_{\tau }^{*}c$ for some $c\in {\mathbb{K}}^n$ satisfying

$$\begin{array}{c}\hfill {\parallel c\parallel }_0<{\displaystyle \frac{1}{2}}Spark\left({\left\{{\tau }_j\right\}}_{j=1}^n\right),\end{array}$$

then c is the unique solution to Problem 1.

In this note, we show that Definition 1 can be extended largely. Using this, we show that Theorems 2 and 3 have continuous extensions.

2. Continuous Spark

Let $(\mathrm{\Omega },\mu )$ be a measure space and let

$$\begin{array}{c}\hfill \mathcal{M}(\mathrm{\Omega },\mu )\{f:\mathrm{\Omega }\to \mathbb{K}\mathrm{is}\mathrm{measurable}\}.\end{array}$$

Let $\mathcal{V}$ be a vector space over $\mathbb{K}$ and let $\mathcal{W}$ be a subspace of $\mathcal{M}(\mathrm{\Omega },\mu )$. Given a linear map $A:\mathcal{W}\to \mathcal{V}$, we define the spark of A as

$$\begin{array}{c}\hfill Spark\left(A\right)inf\left\{\mu \right(supp\left(f\right)):f\in ker(A),f\ne 0\}.\end{array}$$

We now have continuous version of Theorem 2.

Theorem 4. (Continuous Donoho-Elad Spark Uncertainty Principle) Let $A:\mathcal{W}\to \mathcal{V}$ be a linear map. If $f,g\in \mathcal{W}$ are distinct and $Af=Ag$, then

$$\begin{array}{c}\hfill \mu (supp(f\left)\right)+\mu (supp(g\left)\right)\ge Spark\left(A\right).\end{array}$$

Proof. 

Since $f-g\ne 0$ and $f-g\in ker\left(A\right)$, we have

$$\begin{array}{c}\hfill Spark\left(A\right)\le \mu (supp(f-g\left)\right)\le \mu (supp(f)\cup supp(g\left)\right)\le \mu (supp(f\left)\right)+\mu (supp(g\left)\right).\end{array}$$

□

We set of most general version of Problem 1 as follows.

Problem 5.

Let $A:\mathcal{W}\to \mathcal{V}$ be a linear map. Given $v\in \mathcal{V}$, solve

$$\begin{array}{c}\hfill \underset{g\in \mathcal{W}}{minimize}\mu (supp\left(g\right))\mathrm{subject}\mathrm{to}Ag=v.\end{array}$$

Following is continuous version of Theorem 3.

Theorem 6.(Continuous Donoho-Elad Spark Sparsity Theorem) Let $A:\mathcal{W}\to \mathcal{V}$ be a linear map.

(i)

Let $r\in [0,\infty ).$ If

$$\begin{array}{c}\hfill Spark\left(A\right)>2r,\end{array}$$

then for every $v\in \mathcal{V}$, there exists atmost one vector $f\in \mathcal{W}$ such that

$$\begin{array}{c}\hfill v=Af\mathrm{satisfying}\mu (supp(f\left)\right)\le r.\end{array}$$

(ii)

If $v\in \mathcal{V}$ can be written as $v=Af$ for some $f\in \mathcal{W}$ satisfying

$$\begin{array}{c}\hfill \mu (supp\left(f\right))<{\displaystyle \frac{1}{2}}Spark\left(A\right),\end{array}$$

then f is the unique solution to Problem 5.

Proof.

(i)

Let $r\in [0,\infty )$ and $v\in \mathcal{V}$. Let $g,h\in \mathcal{W}$ satisfy $v=Ag=Ah$ and $\mu (supp(g\left)\right)\le r$, $\mu (supp(h\left)\right)\le r$. We claim that $g=h$. If this is not true, then $g-h\ne 0$. Then $g-h\in ker\left(A\right)$ with $g-h\ne 0$. But then

$$\begin{array}{cc}\hfill 2r& <Spark\left(A\right)\le \mu (supp(g-h\left)\right)\le \mu (supp(g)\cup supp(h\left)\right)\hfill \\ \hfill & \le \mu (supp(g\left)\right)+\mu (supp(h\left)\right)\le r+r=2r\hfill \end{array}$$

which is impossible. Hence claim holds.

(ii)

Let $v\in \mathcal{V}$ and $f\in \mathcal{W}$ satisfies

$$\begin{array}{c}\hfill \mu (supp\left(f\right))<{\displaystyle \frac{1}{2}}Spark\left(A\right).\end{array}$$

Let $g\in \mathcal{W}$ be such that $v=Ag$ and $g\ne f$. Then we have

$$\begin{array}{c}\hfill A(f-g)=v-v=0.\end{array}$$

Hence $f-g\in ker\left(A\right)$ and $f-g\ne 0$. Definition of spark then gives

$$\begin{array}{cc}\hfill Spark\left(A\right)& \le \mu (supp(f-g\left)\right)\le \mu (supp(f\left)\right)+\mu (supp(g\left)\right)\hfill \\ \hfill & <{\displaystyle \frac{1}{2}}Spark\left(A\right)+\mu (supp\left(g\right)).\hfill \end{array}$$

Therefore

$$\begin{array}{c}\hfill \mu (supp\left(f\right))<{\displaystyle \frac{1}{2}}Spark\left(A\right)<\mu (supp\left(g\right)).\end{array}$$

Hence f is unique solution to Problem 5.

□

In view of Theorem 3, we have following problem: For which measure spaces, the converse of (i) in Theorem 6 holds? Note that the proof of converse of (i) in Theorem 3 is based on the technique of writing a $2k$-sparse vector as a difference of two k-sparse vectors [6] which we are unable do in continuous setting.