Gelfand Widths in Compressive Sensing Optimal Number of Measurements

Björn Bringmann Technische Universität München

15.01.2015

Björn Bringmann

Gelfand Widths in Compressive Sensing

1 / 26

Outline 1

Introduction to m-Widths Kolmogorov and Gelfand Widths Compressive m-widths

2

Gelfand Widths of l1 -balls Upper bound Lower bound

3

Applications Optimal Number of Measurements Kashin’s Decomposition Theorem

Björn Bringmann

Gelfand Widths in Compressive Sensing

2 / 26

Introduction to m-Widths

Outline 1

Introduction to m-Widths Kolmogorov and Gelfand Widths Compressive m-widths

2

Gelfand Widths of l1 -balls Upper bound Lower bound

3

Applications Optimal Number of Measurements Kashin’s Decomposition Theorem

Björn Bringmann

Gelfand Widths in Compressive Sensing

3 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Motivation Let X = {f ∈ C (R) : f is 2π periodic} and define

n

Sn (f )(x ) :=

∑

bf (k )eikx .

k =−n

How fast does kf − Sn (f )k2 converge to 0? Approximate for f ∈ C ([a, b]) Z a

n

b

f (x )dx ≈

∑ w k f ( xk ) .

k =0

What is a reasonable error estimate? Conclusion: Approximation by linear subspaces is a good idea! Björn Bringmann

Gelfand Widths in Compressive Sensing

4 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Motivation Let X = {f ∈ C (R) : f is 2π periodic} and define

n

Sn (f )(x ) :=

∑

bf (k )eikx .

k =−n

How fast does kf − Sn (f )k2 converge to 0? Approximate for f ∈ C ([a, b]) Z a

n

b

f (x )dx ≈

∑ w k f ( xk ) .

k =0

What is a reasonable error estimate? Conclusion: Approximation by linear subspaces is a good idea! Björn Bringmann

Gelfand Widths in Compressive Sensing

4 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Motivation Let X = {f ∈ C (R) : f is 2π periodic} and define

n

Sn (f )(x ) :=

∑

bf (k )eikx .

k =−n

How fast does kf − Sn (f )k2 converge to 0? Approximate for f ∈ C ([a, b]) Z a

n

b

f (x )dx ≈

∑ w k f ( xk ) .

k =0

What is a reasonable error estimate? Conclusion: Approximation by linear subspaces is a good idea! Björn Bringmann

Gelfand Widths in Compressive Sensing

4 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Motivation Let X = {f ∈ C (R) : f is 2π periodic} and define

n

Sn (f )(x ) :=

∑

bf (k )eikx .

k =−n

How fast does kf − Sn (f )k2 converge to 0? Approximate for f ∈ C ([a, b]) Z a

n

b

f (x )dx ≈

∑ w k f ( xk ) .

k =0

What is a reasonable error estimate? Conclusion: Approximation by linear subspaces is a good idea! Björn Bringmann

Gelfand Widths in Compressive Sensing

4 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Kolmogorov m-Width Definition Let X be a real (or complex) Banach space. For any subset K ⊂ X and any m-dimensional linear subspace Xm ⊂ X define d (K , Xm ; X ) := sup inf kx − z kX . x ∈K z ∈Xm

Now define the Kolmogorov m-width as dm (K ; X ) := inf{d (K , Xm ; X ) : Xm ⊂ X m-dimensional linear subspace} . y

X1

K x

Björn Bringmann

Gelfand Widths in Compressive Sensing

5 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Kolmogorov m-Width Definition Let X be a real (or complex) Banach space. For any subset K ⊂ X and any m-dimensional linear subspace Xm ⊂ X define d (K , Xm ; X ) := sup inf kx − z kX . x ∈K z ∈Xm

Now define the Kolmogorov m-width as dm (K ; X ) := inf{d (K , Xm ; X ) : Xm ⊂ X m-dimensional linear subspace} . y

X1

K x

Björn Bringmann

Gelfand Widths in Compressive Sensing

5 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Properties of the Kolmogorov m-width

Lemma Let X be a Banach space and K ⊂ X ,m ∈ N. ¯ ; X ), where K¯ is the closure of K . 1 dm (K ; X ) = dm (K 2

For every scalar α there holds dm (α K ; X ) = |α|dm (K ; X ).

3

dm (co(K ); X ) = dm (K ; X ), where co(K ) is the convex hull of K .

4

dm (K ; X ) ≥ dm+1 (K ; X ).

Björn Bringmann

Gelfand Widths in Compressive Sensing

6 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Gelfand m-Width Definition Let X be a real or complex Banach space and K a subset of X. The Gelfand m-width is defined as d m (K ; X ) = inf{ sup kx kX : Lm ⊂ X linear subspace with codim(Lm ) ≤ m} x ∈K ∩Lm

We say that a closed linear subspace Lm ⊂ X has codim(Lm ) = m if dim(X /Lm ) = m . Lemma A linear subspace Lm ⊂ X has codim(Lm ) ≤ m if and only if there exist f1 , . . . , fm ∈ X ∗ such that Lm = {x ∈ X : fi (x ) = 0 ∀i = 1, . . . , m} . Björn Bringmann

Gelfand Widths in Compressive Sensing

7 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Duality of Kolmogorov and Gelfand m-widths Theorem For 1 ≤ p, q ≤ ∞ let p∗ , q ∗ be such that

1 p

+ p1∗ = 1 and

1 q

+ q1∗ = 1. Then

m N N dm (BpN ; `N q ) = d (Bq ∗ ; `p∗ ) .

Lemma Let Y be a finite-dimensional subspace of a Banach space X. Given x ∈ X \Y and y ∗ ∈ Y , the following properties are equivalent: ∗ 1 y is a best approximation to x from Y . ∗ 2 For some λ ∈ X with kλ kX ∗ ≤ 1 and λ |Y ≡ 0, there holds

kx − y ∗ k = λ (x ) .

Björn Bringmann

Gelfand Widths in Compressive Sensing

8 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Connection with linear operators Note that

( m

d (K ; X ) = inf

) sup

m

kx k : A : X → K linear, continuous

.

x ∈K ∩Ker(A)

If Lm is a subspace with codim(Lm ) ≤ m, then choose f1 , . . . , fm ∈ X ∗ as in the previous Lemma and define

A : X → Km , x 7→ [f1 (x ), . . . , fm (x )]t .

If A : X → Km is given, define the corresponding linear subspace Lm := Ker(A). Björn Bringmann

Gelfand Widths in Compressive Sensing

9 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Connection with linear operators Note that

( m

d (K ; X ) = inf

) sup

m

kx k : A : X → K linear, continuous

.

x ∈K ∩Ker(A)

If Lm is a subspace with codim(Lm ) ≤ m, then choose f1 , . . . , fm ∈ X ∗ as in the previous Lemma and define

A : X → Km , x 7→ [f1 (x ), . . . , fm (x )]t .

If A : X → Km is given, define the corresponding linear subspace Lm := Ker(A). Björn Bringmann

Gelfand Widths in Compressive Sensing

9 / 26

Introduction to m-Widths

Kolmogorov and Gelfand Widths

Connection with linear operators Note that

( m

d (K ; X ) = inf

) sup

m

kx k : A : X → K linear, continuous

.

x ∈K ∩Ker(A)

If Lm is a subspace with codim(Lm ) ≤ m, then choose f1 , . . . , fm ∈ X ∗ as in the previous Lemma and define

A : X → Km , x 7→ [f1 (x ), . . . , fm (x )]t .

If A : X → Km is given, define the corresponding linear subspace Lm := Ker(A). Björn Bringmann

Gelfand Widths in Compressive Sensing

9 / 26

Introduction to m-Widths

Compressive m-widths

Compressive m-width Definition The compressive m-width of a subset K of a (real) Banach space X is defined as



E (K ; X ) := inf sup kx − ∆(A x )kX : A ∈ L (X , R ), ∆ : R → X m

m

m

 .

x ∈K

A is the measurement map and ∆ : Rm → X is the arbitrary reconstruction map.

∆

A Björn Bringmann

Gelfand Widths in Compressive Sensing

10 / 26

Introduction to m-Widths

Compressive m-widths

Adaptive compressive m-width Definition The adaptive map F : X → Rm is defined by

   

F (x ) := 

λ1 (x ) λ2;λ1 (x ) (x )



  . ..  . λm;λ1 (x ),...,λm−1 (x ) (x )

for λ1 (·), λ2;λ1 (x ) (·), . . . , λm;λ1 (x ),...,λm−1 (x ) (·) ∈ X ∗ . The adaptive compressive m-width of a subset K of a Banach space X is defined as m Eada (K ; X ) := inf



Björn Bringmann

m

m

sup kx − ∆(F (x ))k : F : X → R adaptive, ∆ : R → X x ∈K

Gelfand Widths in Compressive Sensing

11 / 26



Introduction to m-Widths

Compressive m-widths

Connection with the Gelfand m-width Theorem If K is a subset of a Banach space X, then m Eada (K ; X ) ≤ E m (K ; X ) .

If the subset K satisfies −K = K , then m d m (K ; X ) ≤ Eada (K , X ) .

If the set K further satisfies K + K ⊂ a K for some positive constant a, then E m (K ; X ) ≤ a d m (K ; X ) . Therefore under these assumptions 1 m m (K ; X ) ≤ E m (K ; X ) . E (K ; X ) ≤ d m (K ; X ) ≤ Eada a Björn Bringmann

Gelfand Widths in Compressive Sensing

12 / 26

Gelfand Widths of l1 -balls

Outline 1

Introduction to m-Widths Kolmogorov and Gelfand Widths Compressive m-widths

2

Gelfand Widths of l1 -balls Upper bound Lower bound

3

Applications Optimal Number of Measurements Kashin’s Decomposition Theorem

Björn Bringmann

Gelfand Widths in Compressive Sensing

13 / 26

Gelfand Widths of l1 -balls

Main result

Theorem For 1 < p ≤ 2 and m < N, there exist constants c1 , c2 > 0 depending only on p such that

 c1 min 1,

ln(eN /m) m

Björn Bringmann

1− p1

 1− p1 ln(eN /m) ≤ d m (B1N , `N ) ≤ c min 1 , . 2 p m

Gelfand Widths in Compressive Sensing

14 / 26

Gelfand Widths of l1 -balls

Upper bound

Upper bound

Theorem There is a constant C > 0 such that, for 1 < p ≤ 2 and m < N



d m (B1N , `N p ) ≤ C min 1,

Björn Bringmann

ln(eN /m) m

1− p1 .

Gelfand Widths in Compressive Sensing

15 / 26

Gelfand Widths of l1 -balls

Lower bound

Lower bound

Theorem There is a constant c > 0 such that, for 1 < p ≤ ∞ and m < N



d m (B1N , `N p ) ≥ c min 1,

Björn Bringmann

ln(eN /m) m

1− p1 .

Gelfand Widths in Compressive Sensing

16 / 26

Gelfand Widths of l1 -balls

Lower bound

Recovery of 2s-sparse vectors

Theorem Given a matrix A ∈ Rm×N , if every 2s-sparse vector x ∈ RN is a minimizer of kz k1 subject to A z = A x, then

 m ≥ c1 s ln where c1 =

1 ln(9)

N c2 s



and c2 = 4.

Björn Bringmann

Gelfand Widths in Compressive Sensing

17 / 26

Gelfand Widths of l1 -balls

Lower bound

Preparation

Lemma Given integers s < N, there exist

 n≥

N 4s

s/2

subsets S1 , . . . , Sn of [N ] such that each Sj has cardinality s and

card(Si ∩ Sj )
0 by the maximum number P of points xk ∈ T , k ∈ [P], which are t-separated, i.e. kxk − xl k > t for all k , l ∈ [P], k 6= l.

•

• •

• •

•

• •

•

Lemma Then for any norm k · k on Rm there holds

 P (B1 (0), k · k, t ) ≤ Björn Bringmann

2 1+ t

•

m . Gelfand Widths in Compressive Sensing

19 / 26

Gelfand Widths of l1 -balls

Lower bound

Preparation Definition Let X be a Banach space. For a subset T ⊂ X define the packing number P (T , k · kX , t ) for t > 0 by the maximum number P of points xk ∈ T , k ∈ [P], which are t-separated, i.e. kxk − xl k > t for all k , l ∈ [P], k 6= l.

•

• •

• •

•

• •

•

Lemma Then for any norm k · k on Rm there holds

 P (B1 (0), k · k, t ) ≤ Björn Bringmann

2 1+ t

•

m . Gelfand Widths in Compressive Sensing

19 / 26

Gelfand Widths of l1 -balls

Lower bound

Kolmogorov widths Revisited

Corollary For 2 ≤ p < ∞ and m < N, there exist constants c1 , c2 > 0 depending only on p such that the Kolmogorov widths satisfy

( c1 min 1,

eN m

ln

m

Björn Bringmann


)1−1/p

( ≤

dm (BpN , `N ∞)

≤ c2 min 1,

eN m

ln


)1−1/p

m

Gelfand Widths in Compressive Sensing

20 / 26

Applications

Outline 1

Introduction to m-Widths Kolmogorov and Gelfand Widths Compressive m-widths

2

Gelfand Widths of l1 -balls Upper bound Lower bound

3

Applications Optimal Number of Measurements Kashin’s Decomposition Theorem

Björn Bringmann

Gelfand Widths in Compressive Sensing

21 / 26

Applications

Optimal Number of Measurements

Estimates for the Compressive m-widths

Corollary For 1 < p ≤ 2 and m < N, the adaptive and nonadaptive compressive m-widths satisfy m Eada (B1N , `N p)

Björn Bringmann

E

m

(B1N , `N p)



ln(eN /m)  min 1, m

1− p1

Gelfand Widths in Compressive Sensing

.

22 / 26

Applications

Optimal Number of Measurements

Optimal Number of Measurements Theorem Let 1 < p ≤ 2. Suppose that the matrix A ∈ Rm×N and the map ∆ : Rm → RN satisfy

kx − ∆(A x )kp ≤

C σs (x )1 s1−1/p

∀x ∈ RN .

Then for some constant c > 0 depending only on C there holds

 m ≥ c s ln

eN s

 .

(1)

In particular, if A ∈ Rm×N satisfies δ2s (A) < 0.6246, then necessarily (1) holds with c = c (δ2s ).

Björn Bringmann

Gelfand Widths in Compressive Sensing

23 / 26

Applications

Optimal Number of Measurements

Donoho-Tanner Phase Transition

Figure: Success of L1 −Minimization from Random Partial Fourier Measurements. x-Axis: δ = m/N undersampling fraction y-Axis: ρ = s/m sparsity fraction Björn Bringmann

Gelfand Widths in Compressive Sensing

24 / 26

Applications

Kashin’s Decomposition Theorem

Kashin’s Decomposition Theorem

Theorem There exist universal constants α, β > 0 such that, for any m ≥ 1 the space R2m can be split into an orthogonal sum of two m-dimensional subspaces E and E ⊥ such that

√ √ α mkx k2 ≤ kx k1 ≤ β mkx k2 for all x ∈ E and for all x ∈ E ⊥ .

Björn Bringmann

Gelfand Widths in Compressive Sensing

25 / 26

References

References Holger Rauhut and Simon Foucart A Mathematical Introduction to Compressive Sensing Birkhäuser, 2013 Allan Pinkus n-Widths in Approximation Theory Springer, 1985 David L. Donoho and Jared Tanner Observed Universality of Phase Transitions in High Dimensional Geometry, with Implications for Modern Data Analysis and Signal Processing Philosophical Transactions of the Royal Society, 2009 Björn Bringmann

Gelfand Widths in Compressive Sensing

26 / 26