Analysis of Motion Estimation Algorithm in HEVC

Spring 2014 Multimedia Processing - EE5359

Advisor: Dr. K. R. Rao Department of Electrical Engineering University of Texas, Arlington

Update: 3/01/2014

Tuan Ho 1001006444 [email protected]

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

Acronym • • • • • • • • • • • • • • • • •

2DLOG: Two dimensional logarithmic CTU: Coding Tree Unit CU: Coding Unit FSS: Four steps search HEVC: High Efficiency Video Coding HD: High Definition JCT-VC: Joint Collaborative Team on Video Coding PU: Prediction Unit ME: Motion Estimation MPEG: Moving Picture Experts Group NTSS: New three steps search OSA: Orthogonal search algorithm OTA: One-at-a-time search SAO: Sample Adaptive Offset T.B.D: To Be Determined TSS: Three steps search VCEG: Video Coding Experts Group

-2-

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

Overview about HEVC [1]: High Efficiency Video Coding (HEVC) [1] [5] is the current joint video coding standardization project of the ITU-T Video Coding Experts Group (ITU-T Q.6/SG 16) and ISO/IEC Moving Picture Experts Group (ISO/IEC JTC 1/SC 29/WG 11). The Joint Collaborative Team on Video Coding (JCT-VC) has been established to work on this project. The Joint Collaborative Team on 3D Video Coding Extension Development (JCT3V) has been established to work on 3D video coding extensions of HEVC and other video coding standards [1]. Figure 1 shows the block diagram of an HEVC encoder (with decoder components modeling in light gray) [1].

Figure 1: HEVC Encoder Block Diagram (with decoder components modeling in light gray) [1]

-3-

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

HEVC as an Hybrid-video coding scheme with quadtreebased techniques HEVC, like it predeccessor H.264, is an hybrid-video coding scheme that comprises of a block-based prediction followed by a transform coding; however, the block-based pardigm in HEVC has drastically improved by the introduction of the quadtree-based techniques that gives more flexibility for both the prediction and transform coding. In HEVC input video signal is split into smaller partitions called coding tree units (CTU) by quadtree signaling process. The CTU has one luma CTB (coding tree block) whose size can be 64x64, 32x32 or 16x16 of luma sample data, corresponding chroma CTBs and syntax data. Larger size typically gives better compression. Figure 2 shows an example of a 128x128 pixels area divided into variable-sized Coding Units (CUs). The first 64x64 area (I) is divided into four 32x32 CUs (second depth). The second 64x64 area (II) is encoded as just one 64x64 CU (first depth). The third 64x64 area (III) is divided into four 32x32 CUs, two of which are divided into four 16x16 CUs, one of which is divided into four 8x8 CUs (fourth depth). Finally, the fourth 64x64 area (IV) is divided into four 32x32 CUs and two of them are divided into four 16x16 CUs (third depth) [15].

Figure 2: HEVC Quadtree Partition [15]

-4-

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

Another example of the quadtree process [23] is shown in Figure 3.

Figure 3: Example of a coding quadtree with a 64x64 coding tree block that is divided into smaller, variable-size coding blocks (above). As an example of further subdivisions, the 3232 coding block number 14 is partitioned one into two prediction blocks (below left) and one into variable-size transformblocks using the residual quadtree (below right). [23]

The quadtree-based techniques offers great flexibility and efficiency for the prediction and transform coding; however, it also significantly raises the computational complexity, especially in the motion estimation [6].

Motion Estimation Motion estimation is the process of finding the motion vectors to reduce the temporal redundancy among frames in the video signal. To obtain such vector, blocks in the current frame are compared with those in the reference frames (the number of the frames to be compared can be one or more); an area for where the search is performed is defined to narrow down the finding and ease the processing computation. A motion vector is formed when a block in the reference frame -5-

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

found to be best-matched, i.e. having the most similarity with the one being searched. Figure 4 gives an illustration for the motion estimation process.

Figure 4: Motion Estimation Illustration [3]

Recently, to enable the real-time application of the beyond-HD encoding, the video codec has been widely adopted at hardware implementation level. Therefore, it is crucial to find fast and power-efficient hardware-friendly algorithm for video codec, especially for motion estimation. To achieve good results, many search algorithms have been proposed for H.264 to find best motion vector for the largest 16x16 block of samples. As the largest block size in HEVC increases upto 64x64, it drastically raises the workload for the estimation process compared to H.264. Research [16,17] shows that the running time of motion estimation takes upto 81% portion of the entire encoding process. As a result, fast algorithms for HEVC motion estimation are essential to reduce the entire computational complexity.

Figure 5: These execution timeresults were obtained by profiling the encoding of the“BasketballDrive” Full HD (1920×1080) sequence. [17]

-6-

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

As it is necessary to perform the search/compare for all the blocks inside a frame within the reference picture, which can be as large as HD size 1920x1080, the motion estimation consumes lots of computation time for processing. This makes it the most exhausted process in the video encoding. As the thirst for higher resolution increases drastically in video codecs, especially HEVC, motion estimation's computational complexity becomes extremely critical.

Interpolation HEVC interpolation process is decribed in Figures 6 - 8.

Figure 6: HEVC Interpolation: illustration of a block; A(i,j) are the integer pixels, a(i,j) and c(i,j) are half pixels, b(i,j) are quarter pixels [1]

Figure 7: HEVC Interpolation Filter Coefficients [1]

-7-

Analysis of Motion Estimation Algorithm in HEVC

Figure 8: HEVC Interpolation Filters [1]

-8-

Tuan Ho

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

Approaches Optimized Search Algorithms instead of Full-search for ME Despite of its quality, the full-search algorithm is time-consuming and not suitable for implementation in real-time process. Lots of alternative methods have been proposed in the recent years [6]. Figures 9 and 10 shows some of the fast search algorithms.

Figure 9: Fast search algorithms - (a) Two dimensional logarithmic (2DLOG), (b) Three steps search (TSS), (c) New three steps search (NTSS) [6]

Figure 10: Fast search algorithms - (a) Four steps search (FSS), (b) One-at-a-time search (OTA), (c) Orthogonal search algorithm (OSA) [6]

Hierarchical Search for ME To increase the efficiency of search algorithm and obtain higher quality, the searches can be further divided by two main steps. The first is to achieve an

-9-

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

integer best-match using fast search algorithms. From this new anchor the subpixel search is performed to create more accurate motion vector. This hirarchical search is shown in Figure 11 [6].

Figure 11: Hierarchical Search for ME [6]

Complexity Control Employment By limiting the number of the depths in CU partition, the computational complexity can be reduced. [15] Figure 12 shows the average percentage of computational complexity demanded for encoding CUs belonging to each depth of the tree structure. CU depths are presented on the y-axis of Figure 12 and the computational complexity is presented on the x-axis. As mentioned before the nature of the data structures used in HEVC leads to nested encoding loops, such that CUs at higher tree depths are encoded inside CUs at smaller tree depths. This is shown in Figure 12 as follows: for each CU depth (vertical axis), the computational complexity is divided into three components: (1) the complexity of performing inter prediction for CUs at the current depth (in grey); (2) the complexity of coding the same image area as

- 10 -

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

CUs at a higher depth (in white); and (3) the complexity of other operations (in black) [15].

Figure 12: Complexity of processing regarding to CU Depth [15]

- 11 -

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

References [1] G. Sullivan et al, "Overview of the High Efficiency Video Coding (HEVC) Standard," IEEE Transactions On Circuits and Systems For Video Technology, Vol. 22, No. 12, pp. 1649 – 1668, Dec. 2012. [2] T. Sotetsumoto et al, "Low complexity algorithm for sub-pixel motion estimation of HEVC," 2013 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC), pp. 1-4, Aug. 2013. [3] N. Purnachand et al, "Fast Motion Estimation Algorithm for HEVC," 2012 IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin), pp. 34 – 37, Aug. 2012. [4] C. Yan et al, "Highly Parallel Framework for HEVC Motion Estimation on Many-Core Platform", IEEE Data Compression Conference (DCC), pp. 63 – 72, Mar. 2013. [5] G. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC),” IEEE Journal of Selected Topics in Signal Processing, Vol. 7, No. 6, pp. 1001 – 1016, Dec. 2013. [6] M. Jakubowski and G. Pastuzak, “Block based motion estimation algorithms – a survey,” Opto- Electronics Review, Vol. 21, No.1, pp. 86-102, Mar. 2013. [7] HEVC development at Fraunhofer Heinrich Hertz Institute: http://hevc.hhi.fraunhofer.de/ [8] HEVC Bitstreams: ftp://ftp.kw.bbc.co.uk/hevc/ [9] HEVC Model (HM) repository: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [10] HEVC HM Software Manual: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/branches/HM-9.2dev/doc/software-manual.pdf [11] HEVC Development Guidelines: http://phenix.intevry.fr/jct/doc_end_user/documents/8_San%20Jose/wg11/JCTVC-H1001-v1.zip [12] HEVC Specification:Complexity control of high efficiency video encoders for power-constrained devices http://phenix.it- 12 -

Analysis of Motion Estimation Algorithm in HEVC

Tuan Ho

sudparis.eu/jct/doc_end_user/current_document.php?id=7243 [13] T. Wiegand et al, “Overview of the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, Vol.13, No. 7, pp. 560 - 576, July 2003. [14] G. Correa et al, “Complexity control of high efficiency video encoders for power-constrained devices,” IEEE Transactions on Consumer Electronics, Vol. 57, No. 4, pp.1866 – 1874, Nov. 2011. [15] G. Correa et al, “Coding Tree Depth Estimation for Complexity Reduction of HEVC,” IEEE Data Compression Conference (DCC), pp. 43 – 52, Mar. 2013. [16] S. Kim et al, “A Novel Fast and Low-complexity Motion Estimation for UHD HEVC,” Picture Coding Symposium (PCS), pp. 105 – 108, Dec. 2013. [17] M. Grellert et al, “An adaptive workload management scheme for HEVC encoding,” 20th IEEE International Conference on Image Processing (ICIP), pp. 1850 – 1854, Sep. 2013. [18] D. Han; A. Kulkarni and K. R. Rao, “ Fast inter-prediction mode decision algorithm for H.264 video encoder,” 9th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pp. 1 – 4, May 2012. [19] K. R. Rao et al, “Video coding standards - AVS China, H.264/MPEG-4 PART 10, HEVC, VP6, DIRAC and VC-1,” Springer, 2014. [20] PSNR article in Wikipedia: http://en.wikipedia.org/wiki/Peak_signal-tonoise_ratio [21] Zhou Wang et al, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600 – 612, Apr. 2004. [22] G. Bjontegaard (2008), “Improvements of the BD-PSNR model,” Document of ITU-T Q.6/SG16, Berlin, Germany, July 16-18, 2008. [23] Tung Nguyen et al, “Transform Coding Techniques in HEVC,” IEEE Journal of Selected Topics in Signal Processing, Vol. 7, No. 6, pp. 978 – 989, Dec. 2013

- 13 -

Analysis of Motion Estimation Algorithm in HEVC

- 14 -

Tuan Ho