Image Segmentation Image Segmentation
Objective: extract attributes (objects) of interest from an image
Image Processing with Biomedical Applications ELEG-475/675 Prof. Barner
Points, lines, regions, etc.
Common properties considered in segmentation: Discontinuities and similarities
Approaches considered:
Point and line detection
Edge linking
Thresholding methods Histogram, adaptive, etc.
Region growing and splitting
Image Processing Image Segmentation
Mask filtering approach: R = ω z + ω z + + ω z = ∑ω z Isolated point detection: |R|≥T Example: 9
1 1
2
Line Detection
Detection of Discontinuities
Prof. Barner, ECE Department, University of Delaware
2 2
9 9
i =1
i i
X-ray image T=90% of max value Input, gradient, threshold output
Line detection masks
Image Processing Image Segmentation
Detects lines one pixel wide Line orientation specific
Prof. Barner, ECE Department, University of Delaware
3
Set orientations specific thresholds
Second derivative based
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
4
1
Line Detection Example
Edge Detection
Wire-bond mask for electronic circuit Application of 45° edge mask Result of thresholding
Concepts:
Ideal edge:
Prof. Barner, ECE Department, University of Delaware
5
Image Processing Image Segmentation
Pulse Thick edges
Second derivative:
Spikes at onset and termination Zero crossing marks edge center
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
Prof. Barner, ECE Department, University of Delaware
6
Gaussian noise corrupted edge Derivatives amplify noise
Ramp edge The first derivative:
Ramp Ideal edges are smoothed by optics, sampling, illumination conditions Inch thickness determined by transition region
Edge Example – Noisy Case
Edge Example – Noiseless Case
Step
Practical edge:
Image Processing Image Segmentation
Edge – local Boundary – global
7
Even modest levels of noise severely degraded gradient-based edge detection Possible solution: noise smoothing prior to edge detection
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
8
2
Gradient Operators
Gradient Operators and Example
Two-dimensional gradient: ⎡ ∂f ⎤ ⎡ Gx ⎤ ⎢ ∂x ⎥ ∇f = ⎢ ⎥ = ⎢ ⎥ ⎣G y ⎦ ⎢ ∂f ⎥ ⎣⎢ ∂y ⎦⎥
Magnitude:
Direction (angle)
1/ 2
∇f = mag ( ∇f ) = ⎣⎡Gx2 + Gy2 ⎦⎤ ⎛ Gy ⎞ ⎟ ⎝ Gx ⎠
α ( x, y ) = tan −1 ⎜
Perpendicular to edge
Approximation: ∇f ≈ Gx + G y
Shown: mask realizations
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
9
Gradient Example (I)
Application: Horizontal, vertical and (additive) gradient
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
10
Gradient Example (II)
Preprocess image
Smooth detail textures Thicken edges Filter: 5x5 averaging filter
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
11
Extension to 45° gradients and their application
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
12
3
Laplacian of a Gaussian (LoG)
LoG Example
Angiogram example
Recall Laplacian: ∇2 f =
∂2 f ∂2 f + ∂x 2 ∂y 2
h( r ) = −e
−
r2
Combined (linear) smoothing and derivative operations ⎡r2 −σ 2 ⎤ 2
Image Processing Image Segmentation
⎥e ⎦
−
r2 2σ 2
Prof. Barner, ECE Department, University of Delaware
13
Edge Linking
Example criteria:
Strength of gradients
Direction of gradients
Applied over predefined search neighborhood
Image Processing Image Segmentation
14
Prof. Barner, ECE Department, University of Delaware
15
Gradient ≥ 25 Angle differences ≤ 15°
Final result:
α ( x, y ) − α ( x0 , y0 ) < A
Prof. Barner, ECE Department, University of Delaware
Goal: license plate localization Shown: horizontal and vertical gradient images Linking criteria:
∇f ( x, y ) − ∇f ( x0 , y0 ) ≤ E
LoG models certain aspects of the human visual system
Image Processing Image Segmentation
Noise, illumination irregularities, etc.
Link neighboring segments based on predefined criteria
Numerous false (spaghetti) edges First derivative more widely used
Edge Linking Example
Procedures often yield broken edges
Threshold LoG Mark zero crossings
2σ 2
∇ h(r ) = − ⎢ 4 ⎣ σ
To obtain edges:
Edge detection limitations: Produces double edges, insensitive to edge direction, sensitive to noise Pre-smooth with Gaussian filter
Sobel output shown for reference
Linked edges Search for license plate based on rectangle side ratios
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
16
4
Hough Transform (I)
General approach:
Project feature into a parameter space Examples: lines, circles, etc.
Line case:
Defining parameters: slope and intercept Map lines into the single (slope, intercept) 2-tuple
Hough Transform (II)
Advantage: an infinite number of points get mapped to a single 2-tuple
Reverse operation for isolated (binary) points
Line case: a point is located on an infinite number of lines
Map to all (slope, intercept) 2-tuples corresponding to the infinite number of lines passing through the point Result: a curve in the (slope, intercept)
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
17
Problem: infinite slopes Solution: normal representation of a line
19
Five points Five curves
Parameters space curve intersections:
Peak values in ρ,θ plane correspond to lines in the image planes Prof. Barner, ECE Department, University of Delaware
18
Parameter space:
Increment cells in subdivided ρ,θ plane crossed by curve
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
Image space:
Subdivide ρ,θ plane Map each point in the image plane to a curve in the ρ,θ plane
Image Processing Image Segmentation
Procedure for binary images:
Crossing point: common (slope, intercept)
Hough Transform Example (I)
x cos θ + y sin θ = ρ
Fix xi and yi Line in parameter space: b=-xia+yi All lines (in parameter space) for points on a line in image space cross at a single point
Hough Transform (II)
Line equation: yi=axi+b Parameter space:
Lines connecting points in image space
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
20
5
Collimation of X-Ray Images (I)
Hough Transform Example (II)
Edge localization example Input: infrared image Process:
Edge detection Hough transform Peak detection Map (lines) back to image space
Prof. Barner, ECE Department, University of Delaware
21
Collimation of X-Ray Images (II)
Enhance edges Detect edges Radon transform detected edges
Generalizations to other shapes
Image Processing Image Segmentation
Problem: identify region of exposure Problem: x-ray scattering smoothes edges Solution:
Hough transform generalization
Identify lines Map border (lines) back to image space
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
22
Collimation of X-Ray Images (III)
Radon transform:
R ( ρ ,θ ) = ∫∫ f ( x, y )δ ( x cos θ + y sin θ −ρ )dxdy xy
Examples:
Sample border
Noise free and noisy cases
Image and transform domain representations
Peaks in transform domain correspond to lines in image domain
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
Test case categories and the percentage of images in each category 23
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
24
6
Collimation of X-Ray Images (IV)
Collimation of X-Ray Images (V)
Radon transform of edge detection results
Sobel edge detection results Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
25
Collimation of X-Ray Images (VI)
Image Processing Image Segmentation
Thresholding is appropriate when:
Collimation results based on lines detected in the radon transform domain
27
Objects and background have different intensities Multimodal distribution
Threshold can be set:
Prof. Barner, ECE Department, University of Delaware
26
Thresholding Approaches
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
Globally Locally Adaptively
Utilized multiple thresholds
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
28
7
Illumination Effects
K-Means Algorithm
Recall image model:
z ( x, y ) = ln f ( x, y )
Procedure:
= ln i ( x, y ) + ln r ( x, y )
If components independent:
Uneven illumination yields convolved, distorted distributions
4.
No longer separable
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
9 8 7 6 5
10
10
9
9
8
8
7
7
6
6
5
5
4
4 3 2 1 0 0
1
2
3
4
5
6
7
8
K=2 Arbitrarily choose K object as initial cluster center
9
10
Assign each objects to most similar center
3 2 1 0 0
1
2
3
4
5
6
7
8
9
10
4 3 2 1 0 1
2
3
4
5
6
7
8
9
10
reassign
10
10
9
9
8
8
7
7
30
Input image: grayscale Histogram shows two modes Set threshold with Kmeans algorithm
K=2
6
6 5 4
2 1
Image Processing Image Segmentation
0
Prof. Barner, ECE Department, University of Delaware
Fingerprint Example
reassign
3
Java Demo
Update the cluster means
For Euclidian distance, calculate the mean of all points in a specific cluster
Repeat Steps 2 and 3 until centroid movements areto below a fixed threshold
Image Processing Image Segmentation
29
K-Means Clustering Example 10
Utilized appropriate distance metric (Euclidian, city block, etc.)
Recalculate the positions (values) of the K centroids
3.
Density is a delta
Initial cluster centroids Set randomly or with a priori knowledge
Assign all points (pixel values) to the cluster defined by the closest centroid
2.
Convolve distributions If i(x,y) is constant
Place K points in the space
1.
= i '( x, y ) + r '( x, y )
Apply to spatial or multidimensional samples Apply to intensities (one-dimensional) to cluster histogram
Utilizing the log:
Clustering algorithm
f ( x, y ) − i ( x, y ) r ( x, y )
0 0
1
2
3
4
5
6
7
8
9
10
Prof. Barner, ECE Department, University of Delaware
Update the cluster means
5 4 3 2 1 0 0
1
2
3
4
5
6
7
8
9
31
10
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
32
8
Mixture Model and the EM Algorithm (I)
Statistical modeling
Mixture Model and the EM Algorithm (II)
Relaxation of K-means
Assume samples are from a mixture of two Gaussians:
Y1 ~ N(μ1,σ21) Y2 ~ N(μ2,σ22) Y = (1-Δ)Y1 + ΔY2 where Δ∈{0,1} with Pr(Δ=1)=π
Objective: given N observed samples, determined all parameters θ = (π,θ1,θ2) = (π,μ1,σ21,μ2,σ22) Estimation technique: Maximum Likelihood
Log-likelihood function: N
L (θ ; Z) = ∑ log[(1 − π i ) pθ ( yi ) + π i pθ 2 ( yi )]
The PDF of the samples is thus pY(y) = (1-π)pθ (y) + πpθ (y) 1 2
Direct maximization is difficult Solution: suppose we know the values of the Δi’s
Log-likelihood function reduces to: N
where pθ (y) is a Gaussian PDF with parameters θ=(μ,σ2)
L (θ ; Z, Δ) = ∑ log[(1 − Δ i ) pθ ( yi ) + Δ i pθ 2 ( yi )]
1
Prof. Barner, ECE Department, University of Delaware
33
Image Processing Image Segmentation
N
Given L (θ ; Z, Δ) = ∑ log[(1 − Δ i ) pθ ( yi ) + Δi pθ 2 ( yi )]
1.
2.
ML estimates of μ1 and σ21 are the sample mean and variance for all samples with Δi = 0
3.
ML estimates of μ2 and σ22 are the sample mean and variance for all samples with Δi = 1
Prof. Barner, ECE Department, University of Delaware
∑ (1 − γˆ ) y ∑ (1 − γˆ ) ∑ γˆ y = ∑ γˆ N
i =1 N
i
i =1
i
σˆ 12 =
μˆ 2
i =1 i N i =1
∑
N i =1
i
i
σˆ 22
∑ =
(1 − γˆi )( yi − μˆ1 ) 2
∑
i
N
N i =1
(1 − γˆi )
γˆ ( yi − μˆ 2 ) 2 i =1 i N
∑
N i =1
γˆi
and the mixing probability πˆ = 1 N ∑ i =1 γˆi Integrate steps 2 and 3 until convergence N
Determined using soft assignments of all samples and the probability of a sample being from distribution #2
Image Processing Image Segmentation
2
Maximization Step: compute the weighted means and variances μˆ1 =
Solution: proceed in an iterative fashion utilizing a the expected values of Δi’s λ (θ ) = E (Δ i | θ , Z) = Pr(Δ i = 1| θ , Z) These are hidden terms, referred to as responsibilities
2 2 Make an initial guesses for the parameters μˆ1 , σˆ 1 , μˆ 2 , σˆ 2 , πˆ Expectation Step: compute the responsibilities πˆ pθˆ ( yi ) γˆi = , i = 1, 2,… , N (1 − πˆ ) pθˆ ( yi ) + πˆ pθˆ ( yi ) 1
But Δi’s are unknown
34
2
Case 2:
1
i =1
Prof. Barner, ECE Department, University of Delaware
EM Algorithm for Two-Component Gaussian Mixture
Mixture Model and the EM Algorithm (III)
Case 1:
1
i =1
Image Processing Image Segmentation
1
i =1
35
4.
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
36
9
Biomedical Example – Clustering of Assay Data
Three Mixture Example
Model samples as three Gaussian mixtures Initialize with guess Colors indicate probability of belonging to each parent distribution Shown:
Samples with probabilities Initial guess distributions Distribution variance contour Iterations 1-6 and final result (iteration 20)
Shown:
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
37
In the EM (Baum-Welch) algorithm
μˆ1 =
∑ ∑
i =1 N
(1 − γˆi ) yi
i =1
(1 − γˆi )
∑ ∑
N
μˆ 2 =
γˆ yi
i =1 i N
γˆ
Trivially generalized to a larger number of partitions/mixtures
Mixture model gives soft (probability) cluster assignments
Samples assigned to more than one cluster – membership function Assignments are functions of distance
More robust
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
39
Example: K-means
Shown:
K-medoids – use cluster median as central representative point
Example: test variance
Segment nonhomogeneous partitions
Fuzzy C-Means
38
Partition image Check homogeneity in each partition
i =1 i
Generalizations
Prof. Barner, ECE Department, University of Delaware
Simple approach:
Equivalent to K-means (K=2)
Utilized binary decisions: γˆi = 1 if | yi − μˆ 2 | n (1) < n (0)
Watershed Algorithm (III)
Union of flooded catchment basins: C [ n] = ∪ C (M ) Union of all catchment basins: n
T [ n ] = {( s, t ) | g ( s, t ) < n}
Flood the image, and mark all pixels < the flood plane g(x,y)=n Set containing (coordinates) of points in catchment basin C (Mi) that are < n Cn ( M i ) = C ( M i ) ∩ T [ n ]
Condition 1 violations: some dilations outside q (not shown) Point satisfying both conditions constitute dams Marked with x’s
Image Processing Image Segmentation
The watershed algorithm is typically applied to a gradient image Regional minima (coordinates) in image g(x,y): M1, M2,…, MR Set containing the coordinates of the samples in the catchment basin associated with Mi: C (Mi) Set of image points less than threshold n:
Image Processing Image Segmentation
A ridge separating two (or more) catchment basins is encountered A damn is built to prevent flooding across basins Dilate q∩C[n-1] with a 3x3 structuring element, restricting the dilation to q See previous description Prof. Barner, ECE Department, University of Delaware
68
17
Watershed Segmentation Example
Over Segmentation
Diffuse object example Images shown:
Prof. Barner, ECE Department, University of Delaware
69
Image Processing Image Segmentation
Results of gradient definition
70
Region that is surrounded by greater magnitude points Points in region form a connected component Points in the connected component have the same gray level value
Marked points shown on a smoothed image
Light gray denotes markers
Apply watershed
Markers are the only allowable minima Each region contains a single marker and background Partition each region into foreground and background
Prof. Barner, ECE Department, University of Delaware
Prof. Barner, ECE Department, University of Delaware
Use markers to define “super minima”
Median removes isolated minima Thresholding removes inconsequential background minima
Image Processing Image Segmentation
Preprocessing filtering Restrict minima by using markers Merged generated regions
Over Segmentation – Markers
Postprocess to remove background lines
Excessive number of minima
Over segmentation solutions:
Original image gradient contains excessive minima Prefiltering:
Over segmentation
Over Segmentation – Prefiltering
Closed boundaries Good edge localization
Watershed disadvantages:
Observation Gradient Watershed result Watershed result superimposed on observation
Image Processing Image Segmentation
Watershed advantages:
71
Each region is considered in independent “image” Final results consist of boundaries around the foreground in each marker defined region
Image Processing Image Segmentation
Prof. Barner, ECE Department, University of Delaware
72
18
