PROGRESSIVE SCAN CONVERSION BASED ON EDGE-DEPENDENT INTERPOLATION USING FUZZY LOGIC

P. Brox, I. Baturone, S. Sánchez-Solano

Instituto de Microelectrónica de Sevilla - Centro Nacional de Microelectrónica Avda. Reina Mercedes s/n, (Edif. CICA), E-41012, Sevilla, Spain.

Proc. 4th Conference of the European Society for Fuzzy Logic and Technology (EUSFLAT 2005), Barcelona, September 7-9, 2005.

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Progressive scan conversion based on edge-dependent interpolation using fuzzy logic P. Brox I. Baturone S. Sánchez-Solano [email protected] [email protected] [email protected] Instituto de Microelectrónica de Sevilla, Centro Nacional de Microelectrónica Avda. Reina Mercedes S/N, Edificio CICA, 41012 Seville (Spain)

Abstract De-interlacing algorithms realize the interlaced to progressive conversion required in many applications. The most cost efficient are intra-field techniques, which interpolate pixels of the same field. Some of these methods use the upper and lower pixels lines. Among them, the ELA algorithm is widely employed since it reconstructs the edges of the de-interlaced image with more accuracy eliminating nondesired problems such as blurring and staircase effects. However, the ELA algorithm does not perform well when there are non clear edges or in presence of noise. In order to reduce these drawbacks, a new algorithm is presented in this paper. It is based on a simple fuzzy system which models heuristic rules to improve the ELA algorithm. Two enhancements of this new algorithm are also presented in this paper. Simulation results of video sequences prove the advantageous of the new algorithms. Keywords: Fuzzy logic, de-interlacing, edgedirection detection, ELA.

1

Introduction

The standard television video systems (NTSC, SECAM, PAL) are based on an interlaced video signal which halves the signal bandwidth. However, the advent of high quality monitors, displays and HDTV systems have increased the development of effective techniques which provide an interlaced to progressive conversion [1-9]. They can be classified into three categories: spatial (or intra-field) techniques, temporal (or inter-field) techniques, and hybrid methods. The most cost-efficient are intrafield techniques, which interpolate between pixels from the same field. They are widely used because they require less computational power and only one

delay line buffer. Among them, the simplest one is line doubling. However, jagged and flicker effects appear in the oblique lines and edges. The line average method which interpolates with the upper and lower pixels is also very usual. It reduces considerably flicker and alias problem but the staircase effect remains in the edges. This problem is avoided with the direction-dependent interpolation algorithms. The simplest one is edge-based line average (ELA) algorithm which uses the directional correlations between 3+3 pixels of the adjacent scan lines to interpolate the missing line linearly [1]. This algorithm works well when the edge directions are estimated correctly but, otherwise, it introduces errors and degrades the image quality. These problems appear when directional correlations are similar (in horizontal edges, for instance) and/or the image is corrupted with noise. To increase the robustness and reduce the sensitivity to noise, a three-point median filter that uses information from the previous field is proposed in [2], but this requires the use of an expensive field memory. To improve estimation of edge directions, the ELA algorithm is combined with the line doubling method in [3] and an adaptive ELA technique is proposed in [4], which are more costly computationally. Other authors resort to the use of a larger neighbourhood for the detection of edge directions (5+5 taps in [5], 6+6 taps in [6], 7+7 taps in [7], up to 11+11 taps in [8], and 34+34 taps in [9]), with the resulting increase of hardware resources. Besides, the use of threshold values is proposed in [3] and [8] to ensure that estimated edges are dominant. A new directional edge interpolation technique is presented in this paper which improves the robustness of the original ELA algorithm. A set of simple fuzzy rules indicate the edge directions and a linear interpolation is realized between 3+3 taps of the current field. The improvements of this new method are obtained where there are non clear edges and when noisy images are processed. Besides, this is achieved with a low increase in computational cost. Two modifications of this new algorithm

working with a large neighbourhood of pixels of the same field have been also analyzed. The first one considers 5+5 taps while the second one uses samples from the previous de-interlaced line. The paper is organized as follows. Section 2 describes the proposed fuzzy ELA algorithm and its modification. Several simulation and comparison results are shown in Section 3. Finally, some conclusions are given in Section 4.

Table 1: Fuzzy Rule Set if

antecedents

1)

(a is SMALL) and (b is LARGE)

2.1

Figure 1 shows the pixels used by the ELA algorithm to interpolate the pixel value X=F(x,y). The pseudo code of this algorithm is as follows: b=⏐B-E⏐

c=⏐C-D⏐

(1)

if min{a,b,c}=a, X=(A+F)/2 elseif min{ a,b,c}=c, X=(C+D)/2 else X=(B+E)/2

The direction a corresponds to an edge under 135º and b to an edge under 45º. The zero or minimum correlation does not always indicate the direction of an edge. For example in presence of noise or when there is an edge not clear. If we apply heuristics, we will say that an edge is clear in direction a not only if a is small but also b and c are large, and something similar happens to an edge in direction c. If there is a strongly small correlation in directions a and c and a large correlation in direction b, there is not an edge but vertical linear interpolation does not perform well. The best option is a linear interpolation between the neighbours with small correlations: A, C, D, F. In other case the best thing is to calculate a vertical linear interpolation. This heuristic knowledge is fuzzy since the concepts of small and large are not understood as threshold values but as fuzzy ones. Hence, our proposal is to y-1

x-1

x

x+1

A

B

C

2)

y+1

D

(a is LARGE) and (b is LARGE)

(C+D)/2

and (c is SMALL) 3)

(a is strongly SMALL) and (b is LARGE)

otherwise

(B+E)/2

The rule base of our fuzzy system is described in Table 1. Using fuzzy logic, the concepts of SMALL and LARGE are represented by membership functions that change continuously instead of abruptly between 0 and 1 membership values, as shown in Figure 2 (a), (b). The linguistic hedge ‘strongly’ acting upon the concept of SMALL modifies its membership function as illustrated in Figure 2 (c) [10]. This fuzziness makes all the rules may be activated simultaneously, contrary to what happens in the ELA algorithm. The minimum operator is used as connective and of antecedents and the activation degrees of the rules, αi, are calculated as follows:

α1 = min{µ SMALL a (h ), µ LARGE b (h ), µ LARGE c (h )}

α 2 = min{µ LARGE a (h ), µ LARGE b (h ), µ SMALL c (h )}

α3 = min{µ stronglySMALL a (h ), µ LARGE b (h ), µ stronglySMALL c (h )}

F

ORIGINAL LINE

Figure 1: 3x3 window for the ELA algorithm

(2)

α 4 = 1 − α1 − α 2 − α3

Since the consequents, ci, of the rules are not fuzzy, the global conclusion provided by the system is calculated by applying the Fuzzy Mean defuzzification method, as follows: 4

F(x, y ) =

∑α i =1 4

i

∑α i =1

ci = i

α 1 c1 + α 2 c 2 + α 3 c 3 + α 4 c 4 α1 + α 2 + α 3 + α 4

(3)

Substituting the consequents, ci, by their values in µ

µ

µ strongly

SMALL

SMALL

INTERPOLATED LINE

E

(A+F+C+D)/4

and (c is strongly SMALL)

LARGE

x

y

ORIGINAL LINE

(A+F)/2

model this knowledge by a fuzzy system.

Fuzzy_ELA algorithm (3+3 taps)

a=⏐A-F⏐

consequent

and (c is LARGE)

4)

2 Fuzzy Edge-based Line Average Algorithms

then

H

(a)

h

H

(b)

h

H/2

(c)

h

Figure 2: Membership functions SMALL and LARGE

y-1

x-2

x-1

x

x+1

x+2

A’

A

B

C

C’

y

Interpolated line

X

y+1

D’

D

E

F

directions a’ and c’(corresponding to 153,4º and 26,6º). The concepts of SMALL and LARGE are represented by the membership functions previously defined. The product operator is now used to represent the connective ‘and’ so that the activation degrees of the rules, αi,, could be calculated as follows: α1 = prod {µ SMALL a' (h ), µ LARGE a (h ), µ LARGE b (h ), µ LARGE c (h ), µ LARGE c' (h )}

F’

α 2 = prod {µ LARGE a' (h ), µ LARGE a (h ), µ LARGE b (h ), µ LARGE c (h ), µ SMALL c' (h )}

Figure 3: 5x5 window for the Fuzzy_ELA5+5

Table 1, and applying that α1+α2+α3+α4 is equal to 1, the above expression can be given as: ⎛ A + F⎞ ⎛C+D⎞ ⎛A+F+C+ D⎞ F(x, y ) = α1 ⎜ ⎟ + α2 ⎜ ⎟ + α3 ⎜ ⎟+ (4) 4 ⎝ 2 ⎠ ⎝ 2 ⎠ ⎝ ⎠ + B E ⎛ ⎞ + α4 ⎜ ⎟ ⎝ 2 ⎠

α 3 = prod {µ SMALL a (h ), µ LARGE b (h ), µ LARGE c (h )}

α 4 = prod {µ LARGE a (h ), µ LARGE b (h ), µ SMALL c (h )}

α 5 = prod {µ stronglySMALL a (h ), µ LARGE b (h ), µ stronglySMALL c (h )} α 6 = 1 − α1 − α 2 − α 3 − α 4 − α 5

The conclusion is also obtained using the Fuzzy Mean Defuzzication method as follows: ⎛A + F⎞ ⎛ C'+ D' ⎞ ⎛ A'+ F' ⎞ F(x, y ) = α1 ⎜ ⎟+ ⎟ + α3 ⎜ ⎟ + α2 ⎜ ⎝ 2 ⎠ ⎝ 2 ⎠ ⎝ 2 ⎠ ⎛B+E⎞ ⎛A+F+C+D⎞ ⎛C+D⎞ + α4 ⎜ ⎟ ⎟ + α6 ⎜ ⎟ + α5 ⎜ 4 ⎝ 2 ⎠ ⎠ ⎝ ⎝ 2 ⎠

Hence, the fuzzy ELA algorithm applies a linear interpolation in the directions a or b if there is a clear edge in that direction (one αi takes the value 1 and the others are 0). Otherwise several rules are active and the interpolation is non linear. 2.2

2.3

Fuzzy_ELA algorithm (5+5 taps)

(6)

(7)

Recursive Fuzzy_ELA algorithm

(5)

The original Fuzzy ELA algorithm (3+3 taps) is modified in order to increase the edge-detection consistency. The dominant direction is achieved using further information from the neighbourhood. Figure 4 shows the pixels evaluated. Ai, Ci are samples from the previous de-interlaced line (the first line de-interlaced is calculated with the Fuzzy ELA 3+3 method described previously). The new correlational directions considered are given by the expressions in (8).

The added rules (shown in Table 2) consider the new

ai =⏐Ai -A⏐ aii =⏐Ai -F⏐ ci =⏐Ci -C⏐ cii =⏐Ci -D⏐ (8)

The algorithm proposed in the previous section searches for edges in only two directions a and c. A larger neighbourhood has been considered in order to increase the edge directions. Figure 3 shows the pixels used in this new version of fuzzy ELA algorithm. The number of taps has been increased from 3+3 to 5+5 and two new edge directions are considered a’ y c’. a’=⏐A’-F’⏐

c’=⏐C’-D’⏐

Table 2: Fuzzy Rule Set if

antecedents

then

1) (a’ is SMALL) and (a is LARGE) and (b is LARGE)

consequent (A’+F’)/2

and (c is LARGE) and (c’ is LARGE)) 2) (a’ is LARGE) and (a is SMALL) and (b is LARGE)

The new fuzzy rule set is described in Table 3. The first and second rules are modified to ensure the presence of an edge in one of the two orientations y-2

(A+F)/2

and (c is LARGE) and (c’ is LARGE)) 3) (a’ is LARGE) and (a is LARGE) and (b is LARGE)

(C+D)/2

y-1

x-2

x+2

Ai

Ci

x-1

x

x+1

A

B

C

and (c is SMALL) and (c’ is LARGE)) 4) (a’ is LARGE) and (a is LARGE) and (b is LARGE)

(C’+D’)/2

y

Previous Deinterlacing line

De-interlacing line

X

and (c is LARGE) and (c’ is SMALL)) 5) (a is strongly SMALL) and (b is LARGE)

(A+F+C+D)/4

and (c is strongly SMALL) 6)

otherwise

(B+E)/2

y+1

D

E

F

Figure 4: Window for the Fuzzy-Recursive ELA algorithm

Table 3: Fuzzy Rule Set if

antecedents

then

consequent

1)((a is SMALL) or (ai is SMALL) or (aii is SMALL)) (2Ai+A+F )/4 and (b is LARGE) and (c is LARGE) and (cii is LARGE)) 2) (a is LARGE) and (aii is LARGE) and (b is LARGE) (2Ci+C+D )/4 and( (c is SMALL) or (ci is SMALL) or (cii is SMALL)) 3) (a is strongly SMALL) and (b is LARGE) (A+F+C+D)/4 and (c is strongly SMALL) 4) otherwise

(B+E)/2

(angle 45º or 135º). The activation degree of the rules, showed in (10), are calculated using the minimum as connective ‘and’ and the maximum as connective ‘or’. ⎧⎪max(µ SMALL a (h ), µ SMALL ai (h ), µ SMALL aii (h )),⎫⎪ α1 = min ⎨ ⎬ ⎪⎭ ⎪⎩µ LARGE b (h ), µ LARGE c (h ), µ LARGE cii (h ) ⎧⎪µ LARGE a (h ), µ LARGE aii (h ), µ LARGE b (h ), ⎫⎪ α 2 = min ⎨ ⎬ ⎪⎩max(µ SMALL a (h ), µ SMALL ci (h ), µ SMALL c ii (h ))⎪⎭ α3 = min {µ stronglySMALL a (h ), µ LARGE b (h ), µ stronglySMALL c (h )}

(9)

α 4 = 1 − α1 − α 2 − α3

The final conclusion is obtained by applying the same defuzzification method, as follows: ⎛ A + Ai + A + F ⎞ ⎛ C + Ci + C + D ⎞ F(x, y ) = α 1 ⎜ i ⎟ + α2⎜ i ⎟+ 4 4 ⎝ ⎠ ⎝ ⎠ ⎛B+E⎞ ⎛A+F+C+D⎞ + α3⎜ ⎟ ⎟ + α4⎜ 4 ⎝ 2 ⎠ ⎠ ⎝

3

(10)

Simulation results

Extensive simulations conduced with video sequences and static images have been performed to analyze the proposed methods over other intra-field methods. The even/odd lines of original progressive frames have been eliminated from the frame of the sequence with an even/odd number. These missing lines have been calculated by applying a deinterlacing technique. The line doubling, line average, ELA, the Fuzzy_ELA 3+3 algorithm with crisp instead of fuzzy descriptions of labels SMALL and LARGE (crisp method), the method developed in [5], and our fuzzy ELA algorithms have been programmed in Matlab. Table 4 shows the average of PSNR values from 50th to 70th de-interlaced frames of the Claire sequence. Table 5 shows the average of PSNR values from 16th to 40th frame of Susie

Figure 5: a) One progressive frame of the Claire sequence

sequence. The robustness of the algorithms against noise has been analyzed by adding artificial noise to original frames. Two types of noise (Gaussian and impulsive) have been analyzed with different degree of degradation. The results show that our proposed methods perform considerably better than the original ELA algorithm. Comparing with the line average method the results are slightly better. However, fuzzy ELA algorithms reconstruct better the edges of the de-interlaced frames. This is corroborated with the zooms (Figure 6) of the de-interlaced frame obtained from one of the progressive frames of the sequence (Figure 5). Fuzzy ELA algorithms perform better when artificial noise is added. The first column of tables show the PSNR value obtained from the comparison between the progressive original frame and the noisy one. The second column shows the best expected result which has been obtained deinterlacing fields with the original lines without noise. The rest of the columns show the obtained results applying de-

(a)

(b)

(c)

(d)

Figure 6: Zoom of the de-interlaced frame with line doubling (a), line average (b), ELA (c), and the Fuzzy_ELA3+3 algorithms (d)

(a)

(b)

(c)

(d)

Figure 7: Original progressive frame corrupted by Gaussian noise (a) and impulsive noise (b). The corresponding de-interlaced images with Fuzzy_ELA 3+3 method (c) and (d)

interlacing methods (Figure 7). Analyzing the results, the Fuzzy_ELA 5+5 method slightly works better than the Fuzzy_ELA 3+3 in cases that there is no noise or with images corrupted by impulsive noise, whereas the recursive fuzzy_ELA algorithm improves the results gradually when the degree of degradation of the noisy image increases. The fuzzy ELA algorithms presented in this paper have been also described using a development environment for fuzzy inference⎯based systems called Xfuzzy3.0. It integrates a tool, named xfsl, which allows the user to apply supervised learning algorithms to tune fuzzy systems within the design flow of Xfuzzy3.0 [11]. The threshold value H used in the descriptions of the fuzzy labels (Figure 2) has been tuned using xfsl. The fuzzy systems have been trained with pattern files composed by three frames of each video sequence. The tuned H values for all video sequences are very similar.

4

Conclusions

The Fuzzy_ELA 3+3 algorithm improves considerably the quality of de-interlaced images in comparison with ELA results. It works as well as

ELA reconstructing clear edges and enhances the results in the rest of the frame. The improvements increase gradually with the presence of noise in the frames of the sequences. Besides, it requires a low computational cost so that its hardware implementation is very simple. The two enhancements of the algorithm improve slightly the results and require a rise of resources for their implementation. These results have been obtained from the analysis of the extensive simulations of video sequences.

References [1] T. Doyle and M. Looymans, “Progressive scan conversion using edge information”, Proc. 3rd Int. Workshop on HDTV, 1989. [2] T. Doyle, “Interlaced to sequential conversion for EDTV applications”, Proc. 2nd Int. Workshop Signal Processing of HDTV, pp. 412430, 1988. [3] M. H. Lee, J.H. Kim, J.S. Lee, K.K.Ryu and D. Song, “ A new algorithm for interlaced to progressive scan conversion based on directional correlations and its IC design”, IEEE Trans. on Consumer Electronics, vol. 40, n. 2, pp. 119129, 1994.

Table 4: PSNR (in dBs) values for Claire sequence (50th- 70th frame) Input noisy image

Best expected image

No noise

Impulse Noise (Density Noise)

Gaussian Noise (Variance)

Double

Average

ELA

Crisp 3+3

Modified ELA [5]

Fuzzy ELA 3+3

Fuzzy ELA 5+5

Recursive FuzzyELA

35,75

40,95

40,55

40,94

40,96

41,12

41,25

41,11

5,5%

17,55

20,57

17,51

18,77

20,24

19,59

19,86

20,32

20,35

20,34

6,5%

16,79

19,87

16,81

18,06

19,47

18,89

19,14

19,58

19,59

19,61

7,5%

16,23

19,26

16,19

17,45

18,82

18,29

18,51

18,95

18,96

18,99

12%

14,23

17,19

14,15

15,38

16,54

16,22

16,33

16,76

16,80

16,82

15%

13,21

16,23

13,20

14,41

15,42

15,24

15,33

15,68

15,75

15,77

0,003

25,65

28,66

25,24

26,69

26,71

26,69

26,71

27,22

27,01

27,23

0,009

21,01

23,96

20,85

22,15

22,19

22,15

22,23

22,76

22,59

22,79

0,015

18,84

21,81

18,75

20,01

20,07

20,01

20,18

20,63

20,53

20,69

0,021

17,42

20,49

17,36

18,59

18,66

18,59

18,83

19,21

19,15

19,31

0,027

16,39

19,37

16,32

17,56

17,63

17,57

17,84

18,15

18,14

18,27

[4] C. J. Kuo, C. L. and C. C. Lin, “Adaptive interpolation technique for scanning rate conversion”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 6, n. 3, 1996. [5] H. Y. Lee, J. W. Park, T. M. Bae, S. U. Choi and Y. H. Ha, “Adaptive scan rate up-conversion system based on human visual characteristics”, IEEE Trans. on Consumer Electronics, vol. 46, n. 4, 2000. [6] J. Salonen and S. Kalli, “Edge adaptive interpolation for scanning rate conversion”, in Signal Processing of HDTV IV, pp. 757-764, 1993. [7] G. De Haan, R. Lodder, “De-interlacing of video data using motion vector and edge information”, Proc. IEEE Int. Conf. on Consumer Electronics, 2002.

[8] Y. L. Chang, S. F. Lin and L. G. Chen, “Extended intelligent edge-based line average with its implementation and test method”, Proc. ISCAS’2004, 2004. [9] H. Yoo and J. Jeong, “Direction-oriented interpolation and its application to deinterlacing”, IEEE Trans. on Consumer Electronics, vol. 48, n. 4, pp. 954-962, 2002. [10] M. Sugeno and T. Yasukawa, “A fuzzy-logicbased approach to qualitative modeling”, IEEE Trans. Fuzzy Systems, vol. 1, n. 1, pp. 7-31, 1993. [11] F. J. Moreno-Velo, I. Baturone, R. Senhadji and S. Sánchez-Solano, “Tuning complex fuzzy systems by supervised learning algorithms”, Proc. IEEE Int. Conf. on Fuzzy Systems, 2003.

Table 5: PSNR (in dBs) values for Susie sequence (16th- 40th frame) Input noisy image

Best expected image

No noise

Impulse Noise (Density Noise)

Gaussian Noise (Variance)

Double

Average

ELA

Crisp 3+3

Modified ELA [5]

Fuzzy ELA 3+3

Fuzzy ELA 5+5

Recursive FuzzyELA

32,71

36,16

35,66

36,31

36,53

36,39

36,73

36,38

5,5%

18,28

21,31

17,76

19,45

20,82

20,12

20,55

20,97

21,03

20,99

6,5%

17,55

20,59

17,45

18,74

20,06

19,44

19,79

20,23

20,29

20,26

7,5%

16,93

19,96

16,86

18,11

19,38

18,81

19,17

19,58

19,65

19,62

12%

14,91

17,91

14,84

16,09

17,12

16,78

17,07

17,42

17,51

17,48

15%

13,93

16,94

13,88

15,13

15,98

15,79

16,06

16,34

16,46

16,43

0,003

26,99

29,99

25,95

27,59

27,43

27,61

27,65

28,08

27,91

28,09

0,009

20,48

23,49

20,23

21,58

21,51

21,57

21,69

22,19

22,05

22,28

0,015

18,29

21,31

18,23

19,45

19,43

19,45

19,63

20,04

19,98

20,18

0,021

16,87

19,89

16,76

18,06

17,99

18,06

18,28

18,63

18,61

18,79

0,027

15,85

18,85

15,75

17,03

16,96

17,03

17,31

17,58

17,58

17,76