CMA 2007

Applications of free probability and random matrix theory Øyvind Ryan

December 2007

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Some important concepts from classical probability Random variables are functions (i.e. they commute w.r.t. multiplication) with a given p.d.f. (denoted f ) Expectation (denoted E ) is integration Independence Additive convolution (∗) and the logarithm of the Fourier transform Multiplicative convolution Central limit law, with special role of the Gaussian law

∗n Poisson distribution Pc : The limit of 1 − nc δ(0) + cn δ(1) as n → ∞. Divisibility: For a given a, find i.i.d. b1 , ..., bn such that fa = fb1 +···+bn . Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Can we find a more general theory, where the random variables are matrices (or more generally, operators), with their eigenvalue distribution (or spectrum) taking the role as the p.d.f.? What are the analogues to the above mentioned concepts for this theory? What are the applications of such a theory?

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Free probability Free probability was developed as a probability theory for random variables which do not commute, like matrices The random variables are elements in a unital ∗-algebra (denoted A), typically B(H), or Mn (C). Expectation (denoted φ) is a normalized linear functional on A. The pair (A, φ) is called a noncommutative probability space. For matrices, φ will be the normalized trace trn , defined by n

trn (a) =

1X aii . n i =1

For random matrices, we set φ(a) = τn (a) = E (trn (a)) is defined by. Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

What is the "central limit" for large matrices? We will attempt to make a connection with classical probability through large random matrices. We would like to define random matrices as "independent" if all entries in one are independent from all entries in the other. Assume that X1 , ..., Xm are n × n i.i.d. complex matrices, and τn (Xi ) = 0, τn (Xi2 ) = 1. What is the limit when m → ∞ in X1 + · · · + Xm √ ? m

If Xi = √1n Yi where Yi has i.i.d. complex standard Gaussian entries, then X1 + · · · + Xm √ ∼ X, m where X = √1n Y and Y has i.i.d. complex standard Gaussian entries. Therefore, matrices with complex standard Gaussian entries are central limit candidates. Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

The full circle law What happens when n is large? The eigenvalues converge to what is called the full circle law. Here for n = 500. 1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 −1

−0.5

0

0.5

1

plot(eig( (1/sqrt(1000)) * (randn(500,500) + j*randn(500,500)) ),’kx’) Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

The semicircle law 35

30

25

20

15

10

5

0 −3

−2

−1

0

1

2

3

A = (1/sqrt(2000)) * (randn(1000,1000) + j*randn(1000,1000)); A = (sqrt(2)/2)*(A+A’); hist(eig(A),40) Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

The Mar˘chenko Pastur law What happens with the eigenvalues of N1 XX H when X is an n × N random matrix with standard complex Gaussian entries? The eigenvalue distribution converges to the Mar˘chenko Pastur law with parameter Nn , denoted µ Nn . Let f µc be the p.d.f. of µc . Then p

(x − a)+ (b − x)+ , (1) 2πcx √ √ where (z)+ = max(0, z), a = (1 − c)+ and a = (1 + c)+ . f

µc

1 (x) = (1 − )+ δ(x) + c

The matrices N1 XX H occur most frequently as sample covariance matrices: N is the number of observations, and n is the number of parameters in the system.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Four different Mar˘chenko Pastur laws µ Nn are drawn. 1.6 c=0.5 c=0.2 c=0.1 c=0.05

1.4

1.2

Density

1

0.8

0.6

0.4

0.2

0

0.5

1

Øyvind Ryan

1.5

2

2.5

3

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Derivation of the limiting distribution for

√1 XXH N

When x is standard complex Gaussian, we have that
E |x|2p = p!.

A more general statement concerns a random matrix √1N XXH , where X is an n × N random matrix with independent standard complex Gaussian entries. It is known [HT] that p   1 X k(ˆπ ) l(ˆπ ) 1 H √ XX N n , = p τn N n N π∈Sp

where π ˆ is a permutation in S2p constructed in a certain way from π, and k(ˆ π ), l (ˆ π ) are functions taking values in {0, 1, 2, ...}. Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

One can show that this equals p   X X ak 1 H √ XX . = 1+ τn N 2k N π ˆ ∈NC k 2p

The convergence is "almost sure", which means that we have very accurate eigenvalue prediction when the matrices are large.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Motivation for free probability One can show that for the Gaussian random matrices we considered, the limits

φ Ai1 B j1 · · · Ail B jl = lim trn Ain1 Bnj1 · · · Ainl Bnjl n→∞

exist. If we linearly extend the linear functional φ to all polynomials in A and B, the following can be shown:

Theorem If Pi , Qi are polynomials in A and B respectively, with 1 ≤ i ≤ l , and φ(Pi (A)) = 0, φ(Qi (B)) = 0 for all i, then φ (P1 (A)Q1 (B) · · · Pl (A)Ql (B)) = 0. This motivates the definition of freeness, which is the analogy to independence. Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Definition of freeness

Definition A family of unital ∗-subalgebras (Ai )i ∈I is called a free family if   aj ∈ Aij   ⇒ φ(a1 · · · an ) = 0. (2) i1 6= i2 , i2 6= i3 , · · · , in−1 6= in   φ(a1 ) = φ(a2 ) = · · · = φ(an ) = 0

A family of random variables ai is called a free family if the algebras they generate form a free family.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

The free central limit theorem

Theorem If a1 , ..., an are free and self-adjoint, φ(ai ) = 0, φ(ai2 = 1, supi |φ(aik )| < ∞ for all k,

√ then the sequence (a1 + · · · + an )/ n converges in distribution to the semicircle law. In free probability, the semicircle law thus √ has the role of the 1 Gaussian law. it’s density is density 2π 4 − x 2

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Similarities between classical and free probability 1

Additive convolution ⊞: The p.d.f. of the sum of free random variables. The role of the logarithm of the Fourier transform is now taken by the R-transform, which satisfies Rµa ⊞µb (z) = Rµa (z) + Rµb (z).

2

The S-transform: Transform on probability distributions which satisfies Sµa ⊠µb (z) = Sµa (z)Sµb (z)

3

Poisson distributions have their analogy in the free Poisson distributions: These are given by the Mar˘cenko Pastur laws µc with parameter c, which also can be written as the limit of

⊞n as n → ∞ 1 − cn δ(0) + nc δ(1)

4

Infinite divisibility: There exists an analogy to the Lévy-Hinčin formula for infinite divisibility in classical probability.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Main usage of free probability in my papers Let A and B be random matrices. How can we make a good prediction of the eigenvalue distribution of A when one has the eigenvalue distribution of A + B and B? Simplest case is when one assumes that B is Gaussian (Noise). What about the eigenvectors? Assume that we have the eigenvalue distribution of 1 H N (R + X )(R + X ) , where R and X are n × N random matrices, with X Gaussian. If the columns of R are realizations of some random vector r , what is the covariance matrix E (ri rj∗ )? Have use for multiplicative free convolution with the Mar˘chenko Pastur law. This has an efficient implementation.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Channel capacity estimation

The following is a much used observation model in MIMO systems: ˆ i = √1 (H + σXi ) H n

(3)

where n is the number of receiving and transmitting antennas, ˆ i is the n × n measured MIMO matrix, H H is the n × n MIMO channel and

Xi is the n × n noise matrix with i.i.d zero mean unit variance Gaussian entries.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Channel capacity estimation With free probability we can estimate the eigenvalues of n1 HHH ˆ i . This helps us estimate the channel based on few observations H capacity: The capacity of a channel with channel matrix H and signal to noise ratio ρ = σ12 is given by C

= =

  1 1 H log det I + 2 HH n nσ n 1X 1 log(1 + 2 λl ) n σ

(4) (5)

l=1

where λl are the eigenvalues of n1 HHH .

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Observation model

Form the compound observation matrix h i ˆ 1, H ˆ 2 , ..., H ˆL , ˆ 1...L = √1 H H L from the observations ˆ i = √1 (H + σXi ) , H n

(6)

Using free probability, one can with high accuracy estimate the ˆ 1...LH ˆH . eigenvalues of n1 HHH from the eigenvalues of H 1...L

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Free capacity estimation for channel matrices of various rank 4.5

4

Capacity

3.5 True capacity, rank 3 Cf, rank 3 True capacity, rank 5 Cf, rank 5

3

True capacity, rank 6 Cf, rank 6 2.5

2

1.5 0

10

20

30

40 50 60 Number of observations

70

80

90

100

Figure: The free probability based estimator for various number of observations. σ 2 = 0.1 and n = 10. The rank of H was 3, 5 and 6. Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

Application areas

digital communications, nuclear physics, mathematical finance Situations in these fields, can often be modelled with random matrices. When the matrices get large, free probability theory is an invaluable tool for describing the asymptotic behaviour of many systems. Other types of matrices which are of interest are random unitary matrices and random Vandermonde matrices.

Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

List of papers Free Deconvolution for Signal Processing Applications. Submitted to IEEE Trans. Inform. Theory. arxiv.org/cs.IT/0701025. Multiplicative free Convolution and Information-Plus-Noise Type Matrices. Submitted to Ann. Appl. Probab. arxiv.org/math.PR/0702342. Channel Capacity Estimation using Free Probability Theory. Submitted to IEEE. Trans. Signal Process. arxiv.org/abs/0707.3095. Random Vandermonde Matrices-Part I: Fundamental results. Work in progress. Random Vandermonde Matrices-Part II: Applications to wireless applications. Work in progress. Applications of free probability in finance. Estimation of the covariance matrix itself (not only it’s eigenvalue distribution). 2008. Øyvind Ryan

Applications of free probability and random matrix theory

CMA 2007

Applications of free probability and random matrix theory

References

[HT]: "Random Matrices and K-theory for Exact C ∗ -algebras". U. Haagerup and S. Thorbjørnsen. citeseer.ist.psu.edu/114210.html. 1998. This talk is available at http://heim.ifi.uio.no/∼oyvindry/talks.shtml. My publications are listed at http://heim.ifi.uio.no/∼oyvindry/publications.shtml

Øyvind Ryan

Applications of free probability and random matrix theory