Machine Learning and Compressive Sensing for Electron Microscopy Andrew Stevens1,2 , Xin Yuan2 , Lawrence Carin2 , Nigel Browning1
[email protected] 1 Pacific
Northwest National Laboratory 2 Duke
University ECE
Related Results Models Video CS
Outline 1
Related Results STEM Inpainting STEM/TEM Super-resolution
2
Models Mixture models Factor analysis Mixture of factor analyzers
3
Video CS Data Camera system Demonstration
2
Related Results Models Video CS
Goals
Reduce dose (and data volume) through spatial compression. Increase speed and decrease data volume through temporal compression. Learn a representation for sample structures (bulk, defects, grain boundaries, etc.).
3
20% SrTiO3 STEM Inpainting [Stevens et al., 2013]
20% SrTiO3 STEM Inpainting [Stevens et al., 2013]
20% zeolite STEM inpainting [Stevens et al., 2013]
20% zeolite STEM inpainting [Stevens et al., 2013]
Related Results Models Video CS
STEM Inpainting STEM/TEM Super-resolution
Super-resolution images are not distributable.
8
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Compressive Sensing (CS) [Stevens et al., 2013, Zhou et al., 2012, Chen et al., 2010]
Given a sensing matrix Φ ∈ RQ×P , Q P, usually Gaussian or Bernoulli, and compressed measurements y i , y i = Φi (x i + i ). We want to recover x i .
Inpainting is the case when Φ is a subset of columns from the identity matrix.
9
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Sparse CS
y = Φ(x + ),
y ∈ RQ , x ∈ RP , Q P
The true signal x is assumed to be sparse in a some (overcomplete) basis D ∈ RP×K , P < K . y = Φ(Dw + ),
nnz(w ) K
10
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Manifold CS [Chen et al., 2010]
11
Gaussian mixture model [Rasmussen, 1999]
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Gaussian mixture model [Rasmussen, 1999]
p(xi |·) =
T X
λt N (µt , τt−1 )
t=1 −1 xi ∼ N (µt(i) , τt(i) )
µt ∼ N (a, b−1 ) τt ∼ G(c, d) λ1 , . . . , λT ∼ Dirichlet(α/T , . . . , α/T ) t(i) ∼ Multinomial(1; λ1 , . . . , λT )
p(t(i) = j|t(−i), α) = 13
n−ij + α/T n−1+α
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Chinese Restaurant Process
μ2,τ2
μ1,τ1
p(t = 1) =
1 α , p(t = 2) = 1+α 1+α 14
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Chinese Restaurant Process
μ2,τ2
μ3,τ3
μ1,τ1
p(t = 1) =
3 1 α , p(t = 2) = , p(t = 3) = 4+α 4+α 4+α 15
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Chinese Restaurant Process
μ2,τ2
μ4,τ4
μ3,τ3
μ1,τ1
{4, 3, 1, α}/(9 + α) 16
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Chinese Restaurant Process
μ2,τ2
μ4,τ4
μ6,τ6
μ1,τ1
μ3,τ3 μ5,τ5
{8, 5, 4, 2, 1, α}/(20 + α) 17
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Chinese Restaurant Process
μ2,τ2
μ4,τ4
μ6,τ6
μ1,τ1
μ3,τ3 μ5,τ5
p(table t) =
nt α , p(new) = n−1+α n−1+α 18
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
CRP Stick Breaking λt = vt
t−1 Y
(1 − vj )
j=1
vt ∼ Beta(1, α)
…
λ7
λ6
λ5
λ4
λ3
λ2
19
λ1
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Factor analysis
Given n samples x i ∈ RN x i = Dw i + µ + i i ∼ N (0, γ−1 I N ) w i ∼ N (0, I K ) where D ∈ RN×K and µ ∈ RN . Equivalently, x i ∼ N (Dw i + µ, γ−1 I N )
20
Image Patches
Image Patches …
8-‐56
64
56 48 40 32 24 16 8
…
8-‐56
64 patches/pixel
8 8 8 8 7 6 5 4 3 2 1
… …
…
8x8 patch
8 8 8 8 7 6 5 4 3 2 1
Dictionaries
Dictionaries
Haar Wavelet basis
Discrete cosine basis
Dictionaries
* -‐1.5 * -‐1.5 +
* 1.5 (-‐1.5,1.5,-‐1.3)
(-‐1.5,1.5,-‐1.3,-‐1.1)
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Sparsity via Beta-Bernoulli Process
For each x i ∈ RP we have a latent binary vector z i ∈ RK that encodes which dictionary elements are used by x i . a K −1 zki ∼ Bern(πk ), πk ∼ Beta ,b K K
26
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Draw From Indian Buffet Process [Griffiths and Ghahramani, 2011] First customer samples Poisson(α) dishes. The ith customer samples each old dish with probability #(previous samples)/i and samples Poisson(α/i) new dishes.
27
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Beta Process Factor Analysis (BPFA) [Zhou et al., 2012]
x i = Dw i + i d k ∼ N (0, P −1 I P ) k ∼ N (0, γ−1 I P ),
γ ∼ Gamma(c, d)
w i = si ~ z i si ∼ N (0, γs−1 I K ), zi ∼
K Y k =1
Bern(πk ),
γs ∼ Gamma(e, f ) K Y a K −1 π∼ ,b Beta K K k =1
28
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Connection to Optimization
− log p(D, S, Z , π|X , a, b, c, d, e, f ) N γ X = kx i − D(si ~ z i )k22 2
+
P 2
i=1 K X
k =1
kd k k22 +
N γs X ksi k22 2 i=1
− log fBeta-Bern (Z ; a, b) − log Gamma(γ |c, d) − log Gamma(γs |e, f ) + Const 29
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Mixture of factor analyzers [Chen et al., 2010] −1 x i ∼ N (D t(i) w i + µt(i) , γ,t(i) IP)
si ∼ Nt(i) (0, γs−1 I K )
w i = si ~ z t(i) , t(i) ∼ Mult(1; λ1 , . . . , λT ),
λt = vt
t−1 Y
(1 − vj )
j=1
zt ∼
K Y
π∼
Bernoulli(πk ),
k =1
K Y
Beta
k =1
a K −1 ,b K K
µt ∼ N (µ, τ0−1 I P )
˜ t(i) ∆t(i) , D t(i) = D ˜ (t) ∼ N (0, P −1 I P ), d k
(t)
∆kk ∼ N (0, τtk−1 )
30
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Mixture of factor analyzers [Chen et al., 2010]
−1 x i ∼ N (D t(i) w i + µt(i) , γ,t(i) IP)
si ∼ Nt(i) (0, γs−1 I K )
w i = si ~ z t(i) , t(i) ∼ CRP(α) z t ∼ IBP(a, b)
µt ∼ N (µ, τ0−1 I P )
˜ t(i) ∆t(i) , D t(i) = D ˜ (t) ∼ N (0, P −1 I P ), d k
(t)
∆kk ∼ N (0, τtk−1 )
31
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
Block/group sparsity
MFA is similar to Block sparse models. w1 x = [µ1 , D 1 | . . . |µT , D T ] ... wT
In the presented MFA only one of the w t vectors is non-zero. Each dictionary is usually low-rank (undercomplete), K < P.
32
Related Results Models Video CS
Mixture models Factor analysis Mixture of factor analyzers
CS-MFA
y = Φ(x + ) p(x) ≈
T X
λt N (x; χt , Ωt )
t=1
p(y|x) = N (y; Φx, R −1 ) p(x|y) = R =
p(x)p(y|x) p(x)p(y|x)dx
T X
˜ t N (x; χ ˜ t) λ ˜t , Ω
t=1
33
Related Results Models Video CS
Data Camera system Demonstration
Pixel-wise flutter-shutter [Llull et al., 2013]
Y ij = [Aij1 , Aij2 , . . . , Aij` ]
X ij1 X ij2 .. .
X ij` = Φij x ij Φ = diag(Φ1,1 , Φ1,2 , . . . , ΦNx ,Ny )
34
Optical camera setup [Llull et al., 2013]
Related Results Models Video CS
Data Camera system Demonstration
Video CS demonstration
36
Related Results Models Video CS
Data Camera system Demonstration
Thanks! 1
Related Results STEM Inpainting STEM/TEM Super-resolution
2
Models Mixture models Factor analysis Mixture of factor analyzers
3
Video CS Data Camera system Demonstration
37
Related Results Models Video CS
Questions?
Data Camera system Demonstration
[email protected]
1
Related Results STEM Inpainting STEM/TEM Super-resolution
2
Models Mixture models Factor analysis Mixture of factor analyzers
3
Video CS Data Camera system Demonstration
38
Related Results Models Video CS
Data Camera system Demonstration
References I M. Chen, J. Silva, J. Paisley, C. Wang, D. Dunson, and L. Carin. Compressive sensing on manifolds using a nonparametric mixture of factor analyzers: Algorithm and performance bounds. Signal Processing, IEEE Transactions on, 58(12): 6140–6155, Dec 2010. T. Griffiths and Z. Ghahramani. The indian buffet process: An introduction and review. The Journal of Machine Learning Research, 12:1185–1224, 2011. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. Brady. Coded aperture compressive temporal imaging. Optics express, 21(9):10526–10545, 2013. C. Rasmussen. The infinite gaussian mixture model. In NIPS, volume 12, pages 554–560, 1999. 39
Related Results Models Video CS
Data Camera system Demonstration
References II
A. Stevens, H. Yang, L. Carin, I. Arslan, and N. Browning. The potential for bayesian compressive sensing to significantly reduce electron dose in high-resolution stem images. Microscopy, 63(1):41–51, 2013. M. Zhou, H. Chen, J. Paisley, L. Ren, L. Li, Z. Xing, D. Dunson, G. Sapiro, and L. Carin. Nonparametric bayesian dictionary learning for analysis of noisy and incomplete images. Image Processing, IEEE Transactions on, 21(1):130–144, 2012.
40
10%
5%
Related Results Models Video CS
Data Camera system Demonstration
SrTiO3 Error Metrics
SrTiO3 SrTiO3 SrTiO3 SrTiO3
5% 10% 20% 100%
Estimated Noise Variance
Sample PSNR (dB)
Inpainted PSNR (dB)
33.75 32.00 31.83 28.36
9.04 9.28 9.79 -
15.91 17.73 18.78 20.50
43
Inpainted PSNR vs. Denoised (dB) 19.00 22.73 26.14 -
Related Results Models Video CS
Data Camera system Demonstration
SrTiO3 Structure Identification Average of 9 images reconstructed from 5% samples with overlaid grain boundary structure.
44
Related Results Models Video CS
Data Camera system Demonstration
SrTiO3 Quality Comparison
45
10%
5%
Related Results Models Video CS
Data Camera system Demonstration
Zeolite Error Metrics
zeolite 5% zeolite 10% zeolite 20% zeolite 100%
Estimated Noise Variance
Sample PSNR (dB)
Inpainted PSNR (dB)
10.69 11.26 11.83 11.51
7.72 7.96 8.47 -
23.61 26.34 26.58 27.27
48
Inpainted PSNR vs. Denoised (dB) 26.42 35.67 38.98 -
Related Results Models Video CS
Data Camera system Demonstration
Zeolite Quality Comparison
49
Related Results Models Video CS
Data Camera system Demonstration
SrTiO3 Dictionaries
5%
10%
20%
50
100%
Related Results Models Video CS
Data Camera system Demonstration
Zeolite Dictionaries
5%
10%
20%
100%
51
