Digital Image Processing, 2nd ed.

Digital Image Processing Chapter 11

Image Representation and Description Dr. Kai Shuang Department of Electronic Engineering China University of Petroleum [email protected] Some slides and illustrations from Dr. Jimin Liang and Dr. Nawapak Eua-Anant © 2002 R. C. Gonzalez & R. E. Woods

Image Representation and Description

Image Representation and Description? Objective: To represent and describe information embedded in an image in other forms that are more suitable than the image itself.

Benefits: – Easier to understand – Require fewer memory, faster to be processed – More “ready to be used”

What kind of information we can use? - Boundary, shape - Region - Texture - Relation between regions

Image Representation and Description

Shape Representation by Using Chain Codes

Why we focus on a boundary? The boundary is a good representation of an object shape and also requires a few memory. Chain codes: represent an object boundary by a connected sequence of straight line segments of specified length and direction.

4-directional chain code

8-directional chain code

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Examples of Chain Codes

Object boundary (resampling)

0033…01 (4-directional)

Boundary vertices

0766…12 (8-directional)

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

The First Difference of a Chain Codes Problem of a chain code: a chain code sequence depends on a starting point. Solution: treat a chain code as a circular sequence and redefine the starting point so that the resulting sequence of numbers forms an integer of minimum magnitude. The first difference of a chain code: counting the number of direction change (in counterclockwise) between 2 adjacent elements of the code.

Example: 1 2 3

Chain code : The first Example: difference - a chain code: 10103322 0Æ1 1 - The first difference = 3133030 2 - Treating a chain code as a 0 0Æ2 0Æ3 3 circular sequence, we get 2Æ3 1 the first difference = 33133030 2Æ0 2 The first difference is rotational 2Æ1 3 invariant.

Image Representation and Description

Polygon Approximation Represent an object boundary by a polygon

Minimum perimeter polygon Minimum perimeter polygon consists of line segments that minimize distances between boundary pixels. Object boundary

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Polygon Approximation:Splitting Techniques

0. Object boundary

1. Find the line joining two extreme points

3. Draw a polygon

2. Find the farthest points from the line

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Distance vs. Angle Signatures Represent an 2-D object boundary in term of a 1-D function of radial distance with respect to θ.

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Boundary Segments Concept: Partitioning an object boundary by using vertices of a convex hull.

Partitioned boundary

Convex hull (gray color) Object boundary (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Convex Hull Algorithm Input : A set of points on a cornea boundary Output: A set of points on a boundary of a convex hull of a cornea 1. Sort the points by x-coordinate to get a sequence p1, p2, … ,pn For the upper side of a convex hull 2. Put the points p1 and p2 in a list Lupper with p1 as the first point 3. For i = 3 to n 4. Do append pi to Lupper 5. While Lupper contains more than 2 points and the last 3 points in Lupper do not make a right turn 6. Do delete the middle point of the last 3 points from Lupper

Turn Right OK!

Turn Right OK!

Turn Left NOK!

Image Representation and Description

Convex Hull Algorithm (cont’d) For the lower side of a convex hull 7. Put the points pn and pn-1 in a list Llower with pn as the first point 8. For i = n-2 down to 1 9. Do append pi to Llower While Llower contains more than 2 points and the last 3 points in Llower do not make a right turn Do delete the middle point of the last 3 points from Llower 12. Remove the first and the last points from Llower 13. Append Llower to Lupper resulting in the list L 14. Return L Turn Right Turn OK! Right Turn OK! Left NOK!

Image Representation and Description

Skeletons Obtained from thinning or skeletonizing processes

Medial axes (dash lines) (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Thinning Algorithm Concept:

1. Do not remove end points 2. Do not break connectivity 3. Do not cause excessive erosion Apply only to contour pixels: pixels “1” having at least one of its 8 neighbor pixels valued “0” Notation: Neighborhood p9 p2 p3 arrangement Let p8 p1 p4 = for the thinning Example algorithm p7 p6 p5 0 0 1 Let N ( p1 ) = p 2 + p3 + K + p8 + p9 1 p1 0 T(p1) = the number of transition 0-1 in 1 0 1 the ordered sequence p2, p3, … N(p1) = 4 , p8, p9, p2. T(p1) = 3

Image Representation and Description

Thinning Algorithm (cont’d) Step 1. Mark pixels for deletion if the following conditions are true. a) 2 ≤ N ( p1 ) ≤ 6 p9 p2 p3 (Apply to all border pixels) b) T(p1) =1 p8 p1 p4 c) p2 ⋅ p4 ⋅ p6 = 0 p7 p6 p5 p ⋅ p ⋅ p = 0 d) 4 6 8 Step 2. Delete marked pixels and go to Step 3. Step 3. Mark pixels for deletion if the following conditions are true. a) 2 ≤ N ( p1 ) ≤ 6 (Apply to all border pixels) b) T(p1) =1 c) p2 ⋅ p4 ⋅ p8 = 0 d) p2 ⋅ p6 ⋅ p8 = 0 Step 4. Delete marked pixels and repeat Step 1 until no change occurs.

Image Representation and Description

Example: Skeletons Obtained from Thinning

Skeleton

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Boundary Descriptors 1. Simple boundary descriptors: we can use – Length of the boundary – The size of smallest circle or box that can totally enclosing the object

2. Shape number 3. Fourier descriptor 4. Statistical moments

Image Representation and Description

Shape Number Shape number of the boundary definition: the first difference of smallest magnitude The order n of the shape number: the number of digits in the sequence

1 2

0 3

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Shape Number (cont’d) Shape numbers of order 4, 6 and 8

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Example: Shape Number

2. Find the smallest rectangle that fits the shape

Chain code: 000030032232221211 First difference: 300031033013003130

3. Create grid

4. Find the nearest Grid.

Shape No. 000310330130031303 (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Fourier Descriptor Fourier descriptor: view a coordinate (x,y) as a complex number (x = real part and y = imaginary part) then apply the Fourier transform to a sequence of boundary points.

Let s(k) be a coordinate s( k ) = x (k ) + jy ( k ) of a boundary point k : 1 K −1 Fourier descriptor : a (u ) = ∑ s( k )e −2πuk / K K k =0 Reconstruction formula 1 K −1 s( k ) = ∑ a (u )e 2πuk / K K k =0 Boundary points

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Example: Fourier Descriptor Examples of reconstruction from Fourier descriptors

1 sˆ( k ) = K

P −1

∑ a (u )e π

2 uk / K

k =0

P is the number of Fourier coefficients used to reconstruct the boundary

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Fourier Descriptor Properties Some properties of Fourier descriptors

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Statistical Moments Definition: the nth moment K −1

μ n (r ) = ∑ (ri − m) g (ri ) n

i =0

K −1

where

m = ∑ ri g ( ri )

Example of moment: The first moment = mean The second moment = variance

i =0

Boundary segment

1D graph

1. Convert a boundary segment into 1D graph 2. View a 1D graph as a PDF function 3. Compute the nth order moment of the graph

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Regional Descriptors Purpose: to describe regions or “areas” 1. Some simple regional descriptors - area of the region - length of the boundary (perimeter) of the region - Compactness A( R ) C= 2 P ( R)

where A(R) and P(R) = area and perimeter of region R Example: a circle is the most compact shape with C = 1/4π 2. Topological Descriptors 3. Texture 4. Moments of 2D Functions

Image Representation and Description

Example: Regional Descriptors White pixels represent “light of the cities”

Infrared image of America at night

Region no.

% of white pixels compared to the total white pixels

1 2 3 4

20.4% 64.0% 4.9% 10.7%

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Topological Descriptors Use to describe holes and connected components of the region Euler number (E): E =C−H

C = the number of connected components H = the number of holes

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Topological Descriptors (cont’d) E = -1

Euler Formula E=0

V −Q + F = C − H = E

V = the number of vertices Q = the number of edges F = the number of faces E = -2

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Example: Topological Descriptors

Original image: Infrared image Of Washington D.C. area

The largest connected area (8479 Pixels) (Hudson river)

After intensity Thresholding (1591 connected components with 39 holes) Euler no. = 1552

After thinning

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Texture Descriptors Purpose: to describe “texture” of the region. Examples: optical microscope images:

B

C

A

Cholesterol Microprocessor Superconductor (smooth texture) (coarse texture) (regular texture) (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Statistical Approaches for Texture Descriptors We can use statistical moments computed from an image histogram: K −1 z = intensity μn ( z ) = ∑ ( zi − m) n p( zi ) p(z) = PDF or histogram of z i =0 where

K −1

m = ∑ zi p( zi ) i =0

Example: The 2nd moment = variance Æ measure “smoothness” The 3rd moment Æ measure “skewness” The 4th moment Æ measure “uniformity” (flatness) A B C (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Fourier Approach for Texture Descriptor Concept: convert 2D spectrum into 1D graphs

Fourier Original FFT2D coefficient Divide into areas image +FFTSHIFT by angles image Divide into areas by radius

Sum all pixels in each area R0

S (θ ) = ∑ S r (θ ) r =1

Sum all pixels in each area

π

S ( r ) = ∑ Sθ ( r ) θ =0

Image Representation and Description

Fourier Approach for Texture Descriptor Original image

S(r)

Another image

2D Spectrum (Fourier Tr.)

S(θ)

Another S(θ) (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Moments of 2-D Functions The moment of order p + q m10 m pq = ∑∑ x p y q f ( x, y ) x= m00 x y The central moments of order p + q

m01 y= m00

μ pq = ∑∑ ( x − x ) p ( y − y ) q f ( x, y ) x

μ00 = m00

y

μ01 = μ10 = 0

μ11 = m11 − x m01 = m11 − ym10 μ20 = m20 − x m10 μ02 = m02 − ym01 μ21 = m21 − 2 x m11 − ym20 + 2 x 2m01

μ30 = m30 − 3x m20 + 2 x 2m10

μ12 = m12 − 2 ym11 − x m02 + 2 y 2m10

μ03 = m03 − 3 ym02 + 2 y 2m01

Image Representation and Description

Invariant Moments of 2-D Functions The normalized central moments of order p + q

μ pq η pq = γ μ00

γ =

where

p+q +1 2

Invariant moments: independent of rotation, translation, scaling, and reflection 2 φ1 = η20 + η02 ( ) φ2 = η20 − η02 + 4η112

φ3 = (η30 − 3η12 )2 + (3η21 − η03 )2 φ4 = (η30 + η12 )2 + (η21 + η03 )2

[ )[3(η

] )]

φ5 = (η30 − 3η12 )(η30 + η12 ) (η30 + η12 )2 − 3(η21 + η03 )2 +

(3η21 − η03 )(η21 + η03

[

+ η12 ) − (η21 + η03 2

30

]

φ6 = (η20 − η02 ) (η30 + η12 )2 − (η21 + η03 )2 + 4η11 (η30 + η12 )(η21 + η03 )

2

Image Representation and Description

Example: Invariant Moments of 2-D Functions 1. Original image

2. Half size

4. Rotated 2 degree

3. Mirrored

5. Rotated 45 degree

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Example: Invariant Moments of 2-D Functions Invariant moments of images in the previous slide

Invariant moments are independent of rotation, translation, scaling, and reflection (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Principal Components for Description Purpose: to reduce dimensionality of a vector image while maintaining information as much as possible.

Let

x = [ x1

Mean:

1 m x = E{x} = K

x2 ... xn ]T K

∑x

k

k =1

1 Covariance matrix C x = E{(x − m x )( x − m x )T } = K

K

T T x x − m m ∑ k k x x k =1

Image Representation and Description

Hotelling transformation Let

y = A(x − m x )

Then

m y = E{y} = 0

and

C y = AC x A T

⎡λ1 0 ⎢0 λ 1 If we can find A such that C y = ⎢ ⎢ ... ... ⎢ ⎣ 0 ...

0⎤ ... 0 ⎥ ⎥ ... ... ⎥ ⎥ ... λ1 ⎦ ...

Then elements of y = A ( x − m x ) are uncorrelated. The component of y with the largest λ is called the principal component.

Image Representation and Description

Example • Considering 4 vectors x1=(0,0,0)T, x2=(1,0,0)T, x3=(1,1,0)T and x4=(1,0,1)T ⎡0 1 1 1 ⎤ Χ = ⎢⎢0 0 1 0⎥⎥ ⎢⎣0 0 0 1⎥⎦

• Mean vector and covariance matrix are ⎡3⎤ 1⎢ ⎥ mx = ⎢1⎥ 4 ⎢⎣1⎥⎦

⎡3 1 1 ⎤ 1 C x = ⎢⎢1 3 − 1⎥⎥ 16 ⎢⎣1 − 1 3 ⎥⎦

• How to interpret entries of the covariance matrix?

Image Representation and Description What to do with Cx?

• We want to transform the vectors such that elements of the new vectors are uncorrelated. – Making the off-diagonal elements of covariance matrix 0.

• Cx is real and symmetric, so there exists a set of n orthonormal eigenvectors, ei with corresponding eigenvalues λi in descending order.

Image Representation and Description

Reconstruction of x from y • A is orthonormal, i.e. A-1 = AT • x can be recovered from corresponding y

x = AT y + mx • Approximation can be made by using the first k eigenvectors of Cx to for Ak

ˆx = AkT y + mx • The mean square error between x and xˆ n

k

e ms = ∑ λ j − ∑ λ j = j =1

j =1

n

∑λ

j = k +1

j

Image Representation and Description

Principal Components for Description

e1 and e2 are the eigenvectors of the object

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Principal Components for Description(cont’d) • are the points on the boundary • x=(a,b)T, where a and b are the coordinate values , x1and x2-axes. • The result of Hotelling (principal component) transformation is a new a new coordinate system: – Origin at the centroid of the boundary points – Axes are the direction of the eigenvectors of Cx

Image Representation and Description

Principal Components for Description(cont’d)

• Aligning the data with the eigenvectors, i.e. y = A( x − mx ) , we get

• Aside: λi is the variance of component yi along eigenvector ei

Image Representation and Description

Principle axes • Eigenvectors of the segment’s covariance matrix (scatter matrix) S = ∑ ( x − μ )( x − μ )T x∈s

•

ei , λi are the direction and length of the axes

Image Representation and Description

Example: Principal Components 6 spectral images from an airborne Scanner.

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Example: Principal Components (cont’d)

Component λ 1 3210 2 931.4 3 118.5 4 83.88 5 64.00 6 13.40 (Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Example: Principal Components (cont’d)

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Original image

After Hotelling transform

Image Representation and Description

Relational Descriptors

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Relational Descriptors

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Relational Descriptors

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Relational Descriptors

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Relational Descriptors

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.

Image Representation and Description

Structural Approach for Texture Descriptor

(Images from Rafael C. Gonzalez and Richard E. Wood, Digital Image Processing, 2nd Edition.