1

2

Multimedia Video Coding & Architectures (5LSE0), Module 06

5LSE0 - Mod 06 Part 1

Transform Coding Decorrelation and Transforms

Basic principles of transformation, Fundamental example and theory

Peter H.N. de With ([email protected] ) slides version 1.0

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Principle of Transform Coding

PdW / 2016 Fac. EE SPS-VCA

3

∗ Predictive coding: decorrelate video signal as a signal stream

Transform Coding diagram / key steps – Transformation (case of decorrelation) – Quantization of transform coefficients – Coding of coefficients (transformed samples)

∗ Transform coding: Alternative approach

video in

1x2 block line n+0 line n+1 line n+2

– Rewrite sample combination (x0, x1) in vector form as: (x0, x1) = x0.(1, 0) + x1.(0, 1) – Sample sequence (x0, x1) can be approximated by (x0, x1) „= +/-“( (x0+x1)/2 , (x0+x1)/2 ) – The sample sequence (x0, x1) is exactly described by taking the approximation and adding a detailed modification (x0, x1) = (x0+x1)/2 (1, 1) + (-x0+x1)/2 (-1, 1) PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

quantiquantization zation

coding coding

bits

video out decodecoding ding

dequanti dequanti -zation -zation

s(i) ENCODER

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

∗ Consider the simplest transform of 1x2 samples

compr. out

transfortransformation mation

– Break signal up into vectors – Decorrelate samples within this vector – Quantize transformed samples (transform coefficients) independently and with simple quantizers (PCM/DPCM)

Transform / Basic principles – (1)

4

∗ Transform coding

– Sample by sample processing in time domain – mutually decorrelated sample differences – One difference signal variance (exploited by bit allocation)

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

DECODER

inv. transinv. transformation formation sr(i)

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

5

6

Transform / Basic principles – (2) Example 1x2 blocks: statistics of all (x0, x1) combinations „Car1.y“ sequence, f = 13.5 MHz Note that ∗ „mountain“ landscape ∗ all grey values occur ∗ extends over both (2) dimensions PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

1

Transform / Basic principles – (3)

7

Linear algebra  x0 1  0  x1  = x0.0 + x1.1       

 x0 x0 + x1 1 −x0 + x1 −1 x0 + x1 −x0 + x1   = .  + .  = b0 + .b1 2. 2 1  2 2  x1  2. 2 1

Note that ∗ same landscape occurs ∗ extends now over sample sum only! ∗ Concentration to one dimension

∗ The sample sequence (x0, x1) is a linear combination with – (x0+x1)/ √2 as weight or low-frequency transform coefficient – (-x0+x1)/ √2 as weight or high-frequency transform coefficient – the vectors b0 and b1 are the basis vectors PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Transform / Basic principles – (5)

9

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Transform / Basic principles – (6)

10

Results of geometric interpretation

Geometric interpretation

∗ Transformation is rotation

x1 = x1.e1

∗ (e0, e1) and (b0, b1) are orthogonal vectors ∗ These vectors have normalized length (norm) ∗ (b0, b1) are along axis (projected) x0=x1 and axis x0= -x1 ∗ Thus: a rotated coordinate system! PdW / 2016 Fac. EE SPS-VCA

8

Example 1x2 blocks: „Car1.y“ sequence, f = 13.5 MHz, based on sample sum and difference

∗ Rewrite the sample combination (x0, x1) in vector form

∗ Or, alternatively

Transform / Basic principles – (4)

– Rotation to a different coodinate system for the input samples. The amount of samples gives the amount of dimensions.

∗ Transformation concentrates energy e1 b1

– Coordinates are chosen such that global approximations are found in the total signal and additions that make the description exact.

b0 e0

∗ Transformation to perform ... decorrelation

x0 = x0.e0

– There is statistical dependence between adjacent samples. This is exploited by the global approximation. New coordinates are decorrelated ones.

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

11

Basic principles of transformation Fundamentals in matrix notation

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Transform / Basic principles – (7)

12

Matrix Notation ∗ Rewrite in vector / matrix form (forward)

 y0 x0 + x1 − x0 + x1 1 +1+1 1  x0 y =   = .b0 + .b1= . .   = A.x y 1 2 2 2 −1+1 2  x1    ∗ Hence, y = A x , and similarly, for the backward transform

 xr0 1 +1−1 1  y0 −1 xr =   = . .   = A .y ∗ Some properties  xr1  2 +1+1 2  y1  – Matrices A and A-1 are the transform and inverse transform matrix. Mostly, they are not equal and they are orthogonal. – Basis vectors b0 and b1 are the rows in the transform matrix – Reconstructed samples xr0 and xr1 are linear combination of basis vectors – The weight of each vector is called the transform coefficient PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

2

Transform / Basic principles – (8)

13

14

Matrix notation for generalized 2-D transforms

5LSE0 - Mod 06 Part 2

∗ In matrix notation N−1 N−1

N−1 N−1

i=0

i=0

y(u, v) = ∑

∑x(i, j).A(u,v,i, j) = ∑

∑x(i, j).Av(u,i).Ah(v, j)

j=0

j=0

∗ Perform 2-D transform as two times a 1-D transform, thus N−1 N−1

N−1

N−1

i=0

i=0

j=0

Transformation and Decorrelation

y(u, v) = ∑{∑x(i, j).Av(u, i).Ah(v, j)}= ∑Av(u, i).{∑x(i, j).Ah(v, j)} j =0

∗ Geometric meaning is: the reconstructed sample data is a linear combination of basis images Buv, being an outer product N−1 N−1 of basis vectors T xr(i, j) = ∑ ∑y(u, v).buvKwithKbuv = bu ⊗bv u=0 v=0

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Transform / Removing Correlation – (1) N

15

Transform / Removing Correlation – (2)

16

100

N

N x(n-N+1)

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

50

x(n-1) x(n)

0

n

50

100

∗ Collect (block of) data into vector: ∗ x = (x0,x1,…xN-1) = (x(n), x(n-1),…, x(n-N+1))

0

200

400

600

x1

 x0    x   1

– The signal samples x0 to xN-1 are correlated – Linear transform:  y0   a0,0 L a0, N −1  x0       y = M = M O M  M  = A x  y  a    N −1   N −1,0 L a N −1, N −1  x N −1 

y1

1000

y0 20

σ02 σ12

40

800

40

20

0

20

40

x0

20

to be designed

40

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Transform / Removing Correlation – (3)

PdW / 2016 Fac. EE SPS-VCA

17

Transform Coding System / Co-dec x(n)

50

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

x

y

A

yˆ

A-1

xˆ

18

xˆr (n)

0

50

0

100

200

300

400

500

600

700

800

Vector

900

40

x21

 y0    y   1

Decorrel. Transform

Q-1

Q

Correlating Transform

Vector Inverse

y0 Correlation structure 20

40

20

∗ Transform: 0

20

40

xx01 20

y1

– Removes correlation (when well chosen) – Packs energy (variance) in as few transform coefficients as possible

xr = A−1 yˆ = A−1Q[ y] = A−1Q[ Ax] ≠ x quantization errors

40

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

3

19

Design of Transform Coding System

Transform choice / Objectives

20

General requirements Decorrelating Transform

 y0   a0,0 L a0, N −1  x0       y= M = M O M  M  = A x       y N −1   a N −1,0 L a N −1, N −1  x N −1 

Optimal KLT Static transforms Two-dimensional signals

Transform Coder

∗ Objective is to choose A such that y is uncorrelated:

Correlation considerations Transform Coefficient Quantization

JPEG standard (still) MPEG standard (video)

PCM versus DPCM PDF of transform coefficients Bit allocation Perceptual Aspects

0 0  var(y0 )  0  var( y ) 0 M 1  CX ⇒ CY = ACX AT =   M  O M   (auto-)correlation matrix L L var(yN −1 )  0

DV standard (video) PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Transform choice / Energy Conservation

21

Transform choice / Orthogonal

∗ Transform A is always orthogonal (unitary) −1

∗ Structure of orthogonal transform matrix A:

A =A

transform domain

( )

T

[

 a00 a10 L aN −10    a11 M   a01 A =  O M   M a   0 N −1 a1N −1 L aN −1N −1  T

]

= E[( y − yˆ )T A−1 A−1 ( y − yˆ )] = E ( y − yˆ ) 2 = 1424 3 Identity

= σ q2, y ∗ Quantization error is domain-independent

Basis vector a

0 Basis vector a1 | ak |= 1 • Columns are mutually orthonormal • Columns form the N basis vectors of alternate coordinate system

– optimality in transform domain guarantees optimality in time domain PdW / 2016 Fac. EE SPS-VCA

22

x = AT y

T

∗ Consequence (for one component): σ q2 = E[( x − xˆ ) 2 ] = E[( A−1 y − A−1 yˆ ) 2 ] = time domain

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

23

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Example: Autoregressive Signal – (1)

24

∗ For the autoregressive signal, we find for N=2, the following autocorrelation matrix CX

5LSE0 – Mod 06 Part 3

Choosing a decorrelating transform Example of AR (auto-regressive) source

1 C X = σ x2  ρ

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

 1

∗ The matrix AT in this case is: AT =

1 1 1    2 1 − 1

How to obtain this?

∗ So that CY =

PdW / 2016 Fac. EE SPS-VCA

ρ

σ

2 1 x   2  1

1  1  − 1   ρ

PdW / 2016 Fac. EE SPS-VCA

ρ  1

 1   1

1  1 + ρ  = σ x2  − 1   0

0   1 − ρ 

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

4

25

Example Autoregressive Signal – (2) 1 1 1    A = 2 1 − 1

 y  1 1 1  2   52     =  −1  y =  0  = 2 1 − 1 3   2   y1  1 1  5  2  2  ∗ Inverse transform x =  12 1  −1   2 − 2  2 

T

40

x1

Basis vector a1

20

a0 20

x0

0

20

Basis vector a0

 − 1  12  +  1  2  − 2   5 −1 = a0 + a1 = y0 a 0 + y1 a1 2 2

40

=

a1 20

weighted sum of (two) basis vectors

40

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Decomposition onto Basis Vectors – (1)

27

N −1

xn = ∑ yk an,k

∗ The forward transform A can be seen as a way of finding the weights yk for a given signal x ∗ Only the transform coefficients/weights are quantized a0 0.707

n = 0,1,K, N − 1

k =0

∗ Signal x seen as weighted combination of basis vectors

a1 a2

N −1

x = ∑ yk a k = y0 a 0 + y1 a1 + L + y N −1 a N −1

0.25

a00

x( m, n)

y00 PdW / 2016 Fac. EE SPS-VCA

1

-1

1

-1

-1

a01

y01

1

1

-1

-1

2

a10

y10

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

1

-1

-1

1

a11

y11

-0.702

1

0.638

1

3.377

1

1

1.020

∗ Decorrelating transforms on images are applied to (non-overlapping) subblocks of size NxN (Usually N=8) ∗ Easiest interpretation: subblock is decomposed into basis images ∗ Example:

2.266

29

Decomposition 2-D Image Transforms (1)

4.516

a8

1.718

-0.2

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

1

0.0 0.063

(Example)

3.165

∗ In transform coding the transform coefficients are often referred to as weights of the basis vectors

x

-0.125 0.0

k =0

4

28

k =0

k = 0,1,K, N − 1

n =0

=

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

N −1

yk = ∑ ak ,n xn

4 6

1 2 1 2

x = ∑ yk a k = y0 a 0 + y1 a1 + L + y N −1 a N −1

N −1

PdW / 2016 Fac. EE SPS-VCA

5  2 

Decomposition onto Basis Vectors – (2)

∗ N-dimensional vectors:

6 0

26

∗ Transform example: x=(2, 3)T

∗ Basis vectors

40

Example Autoregressive Signal – (3)

y

-0.025Multimedia Video Coding & A / 5LSE0 / PdW / 2016 Module 06 Transformation Theory Fac. EE SPS-VCA

2-D Image Transforms / Reconstruct.– (2)

30

∗ Subblocks as sum of weighted basis images: N −1 N −1

xm,n = ∑∑ yk ,l a k ,l k =0 l =0

NxN subblock

NxN basis images

∗ Basis images are built from basis vectors of 1-D transforms (assuming separable transform)

a k ,l = a k ⊗ a l PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

5

31

32

2-D Transf. /Example 2-D Basis Images – (3) N=2

 1 1 1 1  a 00 =   ⊗   =   1 1 1 1  1  1  1 − 1  a 01 =   ⊗   =   1  − 1 1 − 1  1  1 1 1  a10 =   ⊗   =   − 1 1  − 1 − 1 1 1  1 − 1  a11 =   ⊗   =   − 1  − 1  −1 1 

Intentional blank slide

+1 times first vector -1 times first vector PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

33

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

34

Walsh-Hadamard Transform – (1) ∗ Basis functions are built up with +1 and -1 values ∗ N=2

5LSE0 – Mod 06 Part 4

1 

1 1 1 1 − 1 − 1 1 1 1 − 1 − 1 1 1 A= 1 1 −1 8 1 1 1 − 1 1 − 1 − 1  1 1 − 1 − 1 − 1 1 − 1 − 1 1 − 1 

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Walsh-Hadamard Transform – (2)

35

x

-0.039 -0.129 Hadamard Transform with ordered rows (sequency ordering)

0 0.326 -0.232

PdW / 2016 Fac. EE SPS-VCA

1

1 −1 1 −1 −1 1 −1 1

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

(Walsh-)Hadamard transform – (3) Hadamard transform

0.324

0.162

1

PdW / 2016 Fac. EE SPS-VCA

0.707

∗ N=8

1

y

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

1 1 1 1   1 1 − 1 1 − 1 A=  4 1 1 − 1 − 1   1 1  1 − 1 − 1 1   1 −1  −1 −1   −1 1  Build from − 1 − 1 recurrence rule,  −1 1 but unordered  1 1  rows! 1 − 1 

N=4

∗ N=8 1 − 1 1 − 1

Choosing a decorrelating transform Practical transforms Hadamard, DCT

PdW / 2016 Fac. EE SPS-VCA

1 1 1    2 1 − 1

A=

1-D Hadamard, N=8

∗ Square waves as basis vectors ∗ Only +1 and -1 as terms in transform matrix ∗ Implementation based on additions and subtractions ∗ Sequency ordering property (increasing # zero transitions) PdW / 2016 Fac. EE SPS-VCA

36

1 1 1 1 1 1 1 1

1 1 1 1 -1 -1 -1 -1

1 1 -1 -1 -1 -1 1 1

1 1 -1 -1 1 1 -1 -1

1 -1 -1 1 1 -1 -1 1

1 -1 -1 1 -1 1 1 -1

1 -1 1 -1 -1 1 -1 1

1 -1 1 -1 1 -1 1 -1

normalize with 1/sqrt(8)

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

6

Transform / Hadamard transform – (4)

37

38

The best fixed transform: Discrete Cosine Transform (DCT)

2-D Hadamard transform ∗ Square wave basis vectors can be used to form 2-D basis images ∗ 8x8 Transform has 64 basis images of 8x8 samples PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

1-D Discrete Cosine Transform – (1)

PdW / 2016 Fac. EE SPS-VCA

39

∗ WHT uses too “coarse” basis functions ∗ For most “natural” signals, cosine basis functions – seem to correspond with structures reasonably well – attractive because of relation to discrete Fourier transform ∗ DCT basis functions:  {au } =  

2 N

1 C (u) =  2 1

(2n + 1)uπ  C (u) cos( ) 2N n=0,1,..., N −1

u = 0,1,..., N − 1

u=0

a0

40

N=8

a1 0.354 0.49

a2

0.462 0.416 0.354 0.278 0.191 0.098

0.354 0.416 0.191 0.098 0.354 0.49 0.354 0.278 0.191 0.49

a3

AT =

0.462 0.278

0.354 0.098 0.462 0.416

0.354 0.098 0.462 0.278 0.354 0.416 0.191 0.49 0.354 0.098 0.462 0.278 0.354 0.416 0.191 0.49

a4

0.354 0.278 0.191 0.49

0.354 0.098 0.462 0.416

0.354 0.416 0.191 0.098 0.354 0.49 0.354 0.49

0.462 0.278

0.462 0.416 0.354 0.278 0.191 0.098

a6 a7 Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Discrete Cosine Transform – (3)

PdW / 2016 Fac. EE SPS-VCA

41

∗ Another way to write down the DCT:

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Discrete Cosine Transform DCT – (4)

42

2-D Discrete Cosine Transform (DCT)

yu =

N −1 2 (2i + 1)uπ C (u )∑ xi cos( ) N 2N i =0

u = 0,1,...N − 1

xi =

2 N −1 (2i + 1)uπ ∑ C (u ) yu cos( 2 N ) N u =0

i = 0,1,...N − 1

∗ Practical systems make use of efficient implementations of the above equations – Butterfly structure: similar structure as Discrete Fourier Transform – Finite word length and accuracy considerations

∗ DCT in many image/video compression standards! PdW / 2016 Fac. EE SPS-VCA

1-D Discrete Cosine Transform – (2)

a5

u≠0

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

∗ Orthogonal transform (in practice, note the constants)

y(u, v) =

N−1 2 C(u)C(v)∑ N i=0

N−1

(2i +1)uπ (2 j +1)vπ ).cos( ) 2N 2N

∑x(i, j).cos( j=0

∗ Separable as two times a 1-D transform, thus

y(u, v) =

N−1 N−1 2 (2i +1)uπ 2 (2 j +1)vπ C(u)∑cos( ){ C(v)∑x(i, j).cos( )} N 2 N N 2N i=0 j =0

∗ Implementation as: transform rows => transpose => transform rows PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

7

Discrete Cosine transform – (5) Discrete Cosine Transform ∗ Originally designed for image restoration ∗ Real terms => real transform, has a relation with DFT with phase encrypted in amplitude information ∗ implementation more expensive, due to cosine terms (reals)

PdW / 2016 Fac. EE SPS-VCA

43

A A A A A C D E -E -D G -G -F -F -G E B -D D B -A -A A A -A -B E C -C -E -F F -G -G F -D C -B B -C

A A -C -B G F E -C -A A B -D -F G D -E

∗ Cosine wave basis vectors can be used to form 2-D basis images ∗ 8x8 Transform has 64 basis images of 8x8 samples ∗ High-frequency coefficients have „modulated“ nature

A=1/sqrt(2), B=cos(1π/16), C=cos(3π/16), D=cos(5π/16), E=cos(7π/16), F=cos(2π/16), G=cos(6π/16), normalize matrix with 1/2

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Transforms / Compare DCT+Hadamard

PdW / 2016 Fac. EE SPS-VCA

45

2-D DCT transform vs. Hadamard transform

Transform / Fast DCT algorithms – (1)

46

∗ Reconsider transform matrix ∗ Different cosine terms map on same value (symmetry) ∗ Exploit distributive and commutative properties ∗ Leads to fast computation flowgraph with 22 mult‘s and 28 add‘s

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Transform / Fast DCT algorithms – (2)

PdW / 2016 Fac. EE SPS-VCA

47

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Transform / Fast DCT algorithms – (3)

48

8-Point 1-D fast DCT implementation (Lee) ∗ Repeated butterfly occurrence leads to special architecture

8-point 1-D fast DCT (Lee) ∗ Discrete Fourier Transform has radix-2 development for fast computation ∗ Similar basic butterfly for DCT exists and found by Lee ∗ Leads to fast computation flowgraph with 16 mult‘s only! ∗ But, …. Consecutive multiplications! PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

1-D DCT transform matrix

∗ Compare distribution of high-freq. (7,8) and lowfreq. (3,2) coefficient ∗ High-frequency coefficient of DCT has clearly peaked probability distribution ∗ Energy is transferred from HF to LF coefficients PdW / 2016 Fac. EE SPS-VCA

44

2-D DCT transform

1-D DCT, N=8 A B F C A D G E

Discrete Cosine Transform – (6)

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

C1 C1

D1 D1

E1 E1

+

A1 A1

F1 F1

F2 F2

PdW / 2016 Fac. EE SPS-VCA

O1 O1 +

X2 X2 E2 E2

Y1 Y1 Y2 Y2

X1 X1

D2 D2

O2 O2

cos a cos a -

A2 A2

X

B2 B2

Y3 Y3 O7 O7 O6 O6 O4 O4 O0 O0

O2 O2 O3 O3

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

8

Transform / Fast DCT algorithms – (4)

49

Alternative normalization of 2-D DCT

∗ A: Efficient 2-D (I)DCT, note the constants, scale down at encoder N−1 N−1 4 (2i +1)uπ (2 j +1)vπ y(u, v) = 2 C(u)C(v)∑ ∑x(i, j).cos( ).cos( ) N 2N 2N i=0 j=0

Comparison of calculations (e.g. compute F(2) ) ∗ In both transforms, the preliminary additions are fA = [ f(1) + f(6) ] - [ f(2) + f(5) ] fB = [ f(0) + f(7) ] - [ f(3) + f(4) ]

(2i +1)uπ (2 j +1)vπ ).cos( ) 2N 2N

N−1 N−1

xr (i, j) = ∑

∑C(u)C(v)y(u,v).cos(

u=0 v=0

∗ The proposed algorithm has for F(2) F(2) = fA C616 + fB C216

∗ B: Alternative High Quality/precision: downscale at decoder side

(2i +1)uπ (2 j +1)vπ ).cos( ) 2N 2N

N−1 N−1

y(u, v) = C(u)C(v)∑

∗ Lee‘s algorithm F(2) = fA (1/ 2C38 ) + fB (1/ 2C18 ) + [fB (1/ 2C18 ) - fA (1/ 2C38 ) ] (1/ 2C14) PdW / 2016 Fac. EE SPS-VCA

50

i=0

xr (i, j) =

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

4 N−1 ∑ N2 u=0

∑x(i, j).cos( j=0

(2i +1)uπ (2 j +1)vπ ).cos( ) 2N 2N

N−1

∑C(u)C(v) y(u, v).cos( v=0

PdW / 2016 Fac. EE SPS-VCA

51

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

52

Karhunen-Loeve Transform – (1)

Theoretically Optimal Decorrelating Transform

∗ Optimally decorrelating transform:  C00 C01 var(y0 ) C0N−1  0 0      C C C M 0 var( y ) 0 M  10 11 12 1   = C ⇒C =  X Y  M   O M M O M      C L L C 0 L L var( yN−1)  N−1N −1  N−10 

Karhunen-Loeve Transform Example pictures

∗ Find matrix A such that CY = A C x A is diagonal ∗ Solution: Eigenvector transform, Hotelling transform, Karhunen-Loeve transform (KLT) T

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Karhunen-Loeve Transform – (2)

PdW / 2016 Fac. EE SPS-VCA

53

∗ Assume that x(n) is autoregressive (AR-1) signal with correlation factor ρ =0.95 ∗ Example for N=8

 x0    T CX = E[ x x ] = E[( x0 ,L, xN −1 ) M ] x   N −1 

PdW / 2016 Fac. EE SPS-VCA

0.95

0.902 0.857 0.815 0.774 0.735 0.698

0.338 0.481 0.467 0.423 0.36

0.95

1

0.95

0.902 0.857 0.815 0.774 0.735

0.351 0.42

1

0.95

0.902 0.857 0.815 0.774

1

0.95

0.902 0.857 0.815

1

0.95

0.902 0.857

1

0.95 1

0.857 0.902 0.95

– Optimally decorrelating transform, best energy packing – Data dependent, need to notify decoder what matrix A is – Requires estimation of autocorrelation function Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Eigenvectors AT

Autocorrelation Matrix 1

0.902 0.95

∗ Properties:

54

Karhunen-Loeve Transform – (3)

∗ Calculate autocorrelation matrix (assume E[x(n)]=0)

∗ Find transform A that diagonalizes CX – Find eigenvectors of CX ak

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

0.815 0.857 0.902 0.95

0.774 0.815 0.857 0.902 0.95

0.735 0.774 0.815 0.857 0.902 0.95

0.698 0.735 0.774 0.815 0.857 0.902 0.95

0.36

0.283 0.195 0.1

0.207 0.085 0.347 0.488 0.462 0.279

0.286 0.179 0.486 0.356 0.094 0.46

0.416

0.364 0.101 0.456 0.278 0.351 0.415 0.19

0.49

0.364 0.101 0.456 0.278 0.351 0.415 0.19

0.49

0.902

0.36

0.416

0.95

0.351 0.42

1

0.338 0.481 0.467 0.423 0.36

=

0.286 0.179 0.486 0.356 0.094 0.46

0.207 0.085 0.347 0.488 0.462 0.279 0.283 0.195 0.1

a1 PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

9

55

Karhunen-Loeve Transform – (4) 0.711

a1

-0.248

a2

0.105

a3

-0.001

a4

-0.004

a5

0.439

a6

0.199

a7

-0.024 PdW / 2016 Fac. EE SPS-VCA

∗ Example for N=256 samples

x log(variance qk)

a0

56

KLT / Optimal Energy Packing

Karhunen Loeve transform

Walsh-Hadamard Transform DCT KLT

y

1

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

57

Examples pictures DCT – (1)

k

256

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

58

Example pictures DCT – (2)

8 rows 64 pixels 8 pixels

89

8

8

1

0

-1

1

0

2

3

0

0

0

0

0

0

0

1

0

1

0

0

0

0

0

0

0

1

0

0

0

0

2

0

0

1

0

1

0

0

0

0

-1

1

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

Cosine patterns/DCT basis functions PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

59

Example pictures DCT – (3)

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Example: 2-D DCT of Image – (1)

60

!

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

10

Example: 2-D DCT of Image – (2) y00

2-D DCT

y70

61

Remember DCT Coefficients Meaning

62

y07

y77 A DCT coefficients is a weight of particular DCT basis function

Spatial domain PdW / 2016 Fac. EE SPS-VCA

DCT domain Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Example: 2-D DCT of Image – (3) DCT blocks at their spatial locations

PdW / 2016 Fac. EE SPS-VCA

63

Collect all DCT coefficients belonging to same basis image into “band”

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

64

Overlapped Transforms – (1) ∗ Block- or segment-based transforms lead to discontinuities at block boundaries – In combination with quantization of transform coefficients, this gives rise to blocking artifacts

∗ Lapped transforms take data from overlapping blocks/segments N

N

N x(n)

x(n-N+1)

n

2N PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Overlapped Transforms – (2) ∗ Extension N-points DCT to 2N points Lapped Orthogonal Transform (LOT) a0

a4

a1

a5

PdW / 2016 Fac. EE SPS-VCA

65

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

Sampler

PCM Encoded Image

Fine Quantizer

Compressed bitstream Quantizer

VLC coding

COMPRESSION/SOURCE CODING

a3

(166 Mb/s)

A/D CONVERTER

Transform

a2

66

General Compression System

(10 Mb/s)

a6 8 basis functions with 16 entries a7

Channel bitstream Error prot.

Encipher

(15 Mb/s)

CHANNEL CODING PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

11

67

PdW / 2016 Fac. EE SPS-VCA

Multimedia Video Coding & A / 5LSE0 / Module 06 Transformation Theory

12