IJCST Vol. 3, Issue 1, Jan. - March 2012

ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print)

Research on Distortion Correction of QR Code Images 1 1,2,3

Jiejing Zhou, 2Yunfei Liu, 3Amit Kumar

College of Information Science and Technology, Nanjing Forestry University, Nanjing, China

Abstract The QR code (Quick Response code) is a two-dimensional barcode which can be stored quite complex and large data and be decoded very fast. In this paper, a new distortion correction method is proposed to correct distorted QR code images taken by mobile phones adaptively. Firstly, the QR code image is filtered. Then, the gradient-based Hough Transform (HT) is modified to shorten the processing time and preserve enough information when detecting the contour lines of QR code symbol. An algorithm for acquiring the four vertices of QR code is proposed during the process. At last, the Inverse Perspective Transformation (IPT) and gray level interpolation are performed. Experiments show that this algorithm can achieve satisfactory results under various acquisition situations. Keywords Quick Response Code, Distortion Correction, Hough Transform, Inverse Perspective Transformation, Threshold I. Introduction QR Code is a matrix type symbol with a cell structure arranged in a square. It consists of the functionality patterns for reading easy and the data area where the data is stored. It includes finder patterns, alignment patterns, timing patterns, data areas and quiet zones. Finder patterns are the patterns used for detecting the position of the QR Code. They are arranging at the three corners of the QR code symbol. The length of the finder pattern is seven times of the module width. Fig. 1, shows an example of the QR code. When QR code images are obtained from mobile phones, the images may suffer perspective distortion in most cases. Therefore, in order to ensure the correctness of QR code decoding, some transformation should be done. According to the general steps of digital image processing, the distortion correction of the 2D barcode can be summarized as follows before the decoding of 2D barcode [1, 2]: 1) the Hough Transform (HT) algorithm, 2) the geometric space transformation and the gray level interpolation. The HT is usually suggested to detect the four contour lines of the 2D barcode. But in the practical application, the four contour lines of QR code symbol usually do not exist as the quiet zone is often ignored when printed. Also, the internal information of QR code symbol may affect the detecting result in some cases. Therefore, the traditional algorithm is needed to be modified in order to adapt to this new situation.

Fig. 1: An Example of the QR Code, (a). A QR Code Symbol with Quiet Zone, (b). A QR Code Symbol Without Quiet Zone w w w. i j c s t. c o m

Reviewing the existing methods, the Inverse Perspective Transformation (IPT) algorithm was approved an effective method to correct the distortion in real environment. Before the IPT is performed, the four corner points of QR code image are needed. There exist several algorithms to solve this problem, including: an Intersection-based Corner Detection (ICD) algorithm [3], an algorithm which combines Canny edge detection with contour finding algorithms [4], an algorithm which first obtains three corner points according to the pattern features of QR code, then gets the fourth corner point by removing all the internal pixels and performs the standard Hough transform algorithm [5], another algorithm which uses the best linear approximation to find the QR code outline [6]. Above methods either take too much computation time or make calculation more complex than HT algorithm. Usually HT tends to be most successfully applied to line finding and has the merit that it is little affected by collinear point’s interspaces and noise, but it is greatly affected by the internal pixels of the QR code symbol. Therefore, it is needed to find a suitable adaptive threshold which can either minimize the impact of the internal pixels or save the computation time. As a result, the gradient-based HT [7-8], method is modified according to the structure of QR code. This method is simple and effective and don’t need to add additional algorithms such as the contours finding algorithms. Testing results show this method is fast and effective. II. An Modified Method A. Adaptive Distortion Correction Method The suggested distorted correction method consists of the following steps. 1. To do gray conversion to the original image if necessary. 2. To filter the gray image. 3. To perform the modified HT to find out the four vertices of the QR code symbol. 4. To perform the IPT according the vertexes. 5. To perform bilinear interpolation to the transformed image. B. Filter After performing the gray conversion to the original image, a filter is needed to smooth the image. The filter should be set to an appropriately size so that it's able to suppress the dominant noise. In this paper, a 5-by-5 filter is used to smooth out the small scale irregularities. The masking processing parameters can be set to be 11111, 12221, 12121, 12221, and 11111 [9]. C. Modified Hough Transform HT is a kind of linear describing method patented by Hough [10] in 1962. It is used to find out lines in all orientations and positions and is suitable for detecting the contour lines when the image is skewed. A line is easily parameterized as a collection of points (x, y) such that: (1) Now any a pair of represents a unique line, where, is the perpendicular distance from the original to the line and is the sweep angle from axis X to the direction that is perpendicular to the line. International Journal of Computer Science And Technology 

415

IJCST Vol. 3, Issue 1, Jan. - March 2012

For all points of a contour line in a QR code symbol, the of this line is unique, so all the contour lines in the QR code image can be located according to the statistics after counting the number of corresponding points of a pair of and using an 2D accumulation array A to record . Standard HT (SHT) [11], is based on the edge detection when processing the segmentation. It can be used to find the contours of the two-dimensional barcodes by Chinese National Standard: • To perform the Canny method to remove the details of the object. • To perform SHT to the input image to find the lines in the image. • To calculate the length of the lines. • To choose the above four lengthiest lines as the four contours of the two-dimensional barcodes. When performing the SHT algorithm, some parameters must be set probably, they are [12]: a threshold which is used to perform the voting process of HT; it is defined as: k*M (M is the maximum value in the accumulation array A); a positive scalar which is used to merge two line segments into a single line segment; a positive real scalar which discards the merged line segments shorter than this parameter. These parameters are all set by the user and any improper values will greatly affect the result. The SHT algorithm takes a large computation and has the fixed thresholds. As a result, HT must be modified. One way of reducing the computation is to make use of gradient information such as the gradient-based HT [13]. In the gradient-based HT, the features of gradient operator are used. That is, the gradient magnitude is small or zero in the slow changing gray or gray-scale invariant region but large in the mutable ones. The gradient-based HT algorithm mainly consists of the following steps [13]. Firstly: to built the accumulation array. • To compute the gradient and the magnitude of gradient. • To clear the margins of the gradient field. • To get the pixels whose gradient magnitudes are larger than the given threshold G. for all the voting • To compute the line parameters pixels. • To build the accumulation array using the gradient magnitude as weight. Secondly: To locate the peaks in the accumulation array. • To smooth the accumulation array. • To find the maximum value in the accumulation array. • To apply the local maxima filter to the array. • To perform segmentation for the array according to k. • To clear the margins of the array. • To group the local maxima candidates by adjacency. Finally: to get the parameters of the detected lines. where G is a threshold of the gradient magnitude in order to remove the background of image before the voting process of the HT. k is a value that controls the line detection sensitivity as mentioned before. In the above algorithm, due to the two parameters G and k are all fixed values, they are not suitable to different QR image. So we need to find the adapted values. In this paper, the specifications of QR code symbol are exploited to adjust the two parameters. The QR code symbol width L is defined as [14]: L = (17 + 4 ⋅ V ) ⋅ d (2) where, V is the version of QR code symbol. Its value is from 1 to 40 and d is the module width. Supposing the module width d takes one pixel, the maximum

416

International Journal of Computer Science And Technology 

ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print)

storage T of QR code symbol can be obtained as: T = L2 = (17 + 4 ⋅ V ) 2 ⋅ d 2 = (17 + 4 × 40) 2 × 1 = 31329 (3) It means that at least 31329 points are needed for QR code symbol description in the maximum version. The value T can be used as a threshold to control the size of the accumulation array when the version number V is not known before. When the size of the accumulation is larger than T, the value of G is increased to remove the excess points of the accumulation array; otherwise, the value is decreased. In this way, the efficiency of HT for QR code image can be greatly improved. Briefly, values of pixels with gradient magnitudes below than G are not considered belong to any line. G is changed until enough QR code points are obtained. The value of G can be defined as:   G0 G= G  N

where,

if the size of the image < T and the size of M 0 < T otherwise G + N ⋅ tG GN =  0 G0 − N ⋅ tG



(4)

if the size of M N > T + CT if the size of M N < T − CT

(5)

M0 is the initial number of pixels whose gradient magnitudes values are larger than the primary value of G, that is G0. N is operation times needed. tG is the tolerance of G. MN is the number of pixels whose gradient magnitudes values are larger than GN, C is a coefficient set by the user to define the permitted range of MN. It is set with 0.05 in our tests. As a result, some additional pixels can be removed when the size of the QR code image is too large, see Eq. (5). But if the size of the QR code image is smaller than T, G will be set with G0 according to your need. In order to decide the value of k, we can use the features of the finder patterns. We know that the width of the finder patterns is usually the maximum bar width according to the module configuration of QR code symbol [14]. Hence, the eight boundary lines of the three finder patterns are the longest lines in most cases. The location of these eight lines is shown in fig. 2.

Fig. 2: Ideal Eight Detected Lines of the Finder Patterns These eight lines are sufficient to find out the contour lines of the QR code symbol. A larger k value may take a large computation time and reduce the accuracy; a smaller k value may not obtain all the four contour lines. As a result, the value of k is changed until the number of detected lines number R is eight. Values in the accumulation array A which are lager than k*M will be considered to be belong to lines when performing the voting process of HT. The value of k can be defined as: k + N ⋅ tk if the number of detected lines > 8 k= 0  k0 − N ⋅ tk if the number of detected lines < 8 (6) where, k0 is the primary value of k. N is operation times needed. tk is the tolerance of k. w w w. i j c s t. c o m

ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print)

You can give the values of G0 and k0, and set the tolerance tG and tk depending on your own needs. In our tests, G0 is set to be 1 and the tolerance 0.1; k0 is set with 0.1 and its tolerance 0.01. To decide the parameters G and k, the gradient-based HT algorithm is modified, see Algorithm 1. After that, some different lines defined as are obtained. To these lines, perform Algorithm 2. Hereto the coordinates of the four corners of QR code image are all obtained.

ALGORITHM 1 Adaptive thresholds of HT for QR code image Input: X, A, G, k, tG, tk, C, T, Img_size, R. {X presents the input image. A is the accumulation array which records the points in any detected lines. Img_size is the size of input image. R presents the number of Hough detected lines.} Output: Y1, Y2. {Y1 presents the array of the pixels whose gradient magnitudes are larger than the threshold G. Y2 presents the array after thresholding the accumulation array A.} 1: qr_step = 1; G=G0; G_old =G; k=k0; k _old = k 2: Calculate the gradient magnitudes of the input image X 3: if Img_size > T 4: Find the array Y1 with the primary value of G 5: if the size of Y1 > T then 6: while the size of Y1 -T > C *T do 7: sign1 = sign(the size of Y1-T) 8: sign2 = sign(qr_ step) 9: if sign1*sign2 == -1 then 10: qr_ step = qr_ step *(-1) 11: end if 12: G = G + qr_step* tG 13: if G_old == G then 14: break 15: end if 16: Find the array Y1 with the new value of G 17: G _old= G 18: end while 19: end if 20: else 21: Find the array Y1 with the primary value of G 22: end if 23: return Y1 24: Calculate the maximum value in the accumulation array A 25: Find the array Y2 in array A which are larger than k*M 26: Get the number of Hough detected lines R with the initially value of k 27: while R-8 > 0.5 do 28: sign3 = sign(R-8) 29: if sign2*sign3 == -1 then 30: qr_ step = qr_ step *(-1) 31: end if 32: k = k + qr_step* tk 33: if k_old == k then 34: break 35: end if 36: Get the number of Hough detected lines R with the new value of k 37: k _old= k 38: end while 39: end if 40: return Y2 w w w. i j c s t. c o m

IJCST Vol. 3, Issue 1, Jan. - March 2012

ALGORITHM 2 Looking for four vertices Input: R pairs of of Hough detected lines. {R presents the number of Hough detected lines.} Output: P0 ( x 0 , y 0 ) , P1 ( x1 , y1 ) , P2 ( x 2 , y 2 ) and P3 ( x 3 , y 3 ) . begin Step1: To translate the lines into the x and y coordinates system. When , then



(7)

where, i ≠ j and they are in the region of [1, R], is the intersection point of line L1, L2. Step2: To calculate all the intersection points of all these lines and remove the pseudo points such as the points which are beyond the boundaries of the image. Step3: To extract the lines which have the minimum x or y, and the maximum x or y. Step4: To remove the duplicated lines and finally get the external contour lines of the QR code. Step5: To obtain the intersection points P0 ( x 0 , y 0 ), P1 ( x1 , y1 ) , P2 ( x 2 , y 2 )and P3 ( x 3 , y 3 ) of the four external contour lines, where, P0 is the top-left point, P1 is the top-right point, P2 is the bottom-right point and P3 is the bottom-left point. Firstly the four pairs of intersection points ( x r , y r ), (r = 0,1,2,3) are classified into two parts, X : {x 0 , x1 , x 2 , x 3 } and Y : { y 0 , y1 , y 2 , y 3 } Then, X is divided into two parts, the larger ones X L : {x a , x b } and the smaller ones X S : {x c , x d }. At last, the corresponding y values for each x value in X L are found. After that the y values y a and y b are compared, Do: if y a < y b Point ( x a , y a ) is considered as P1 Point ( x b , y b ) is considered as P2 else if y a > y b Point ( x a , y a ) is considered as P2 Point ( x b , y b ) is considered as P1 end if P0 and P3 can be obtained in the same way. if y c< y d Point ( x c , y c ) is considered as P0 Point ( x d , y d ) is considered as P3 else if y c > y d Point ( x c , y c ) is considered as P3 Point ( x d , y d ) is considered as P0 end if end D. Geometric Space Transform Perspective distortion is a common geometric distortion which is caused by the camera's digital sensor not being parallel to the QR code image's surface. IPT which maps an arbitrary 2D quadrilateral (named as Q1) into another (named as Q2) can be used to normalize International Journal of Computer Science And Technology 

417

IJCST Vol. 3, Issue 1, Jan. - March 2012

ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print)

the code shape into a square. Define ( x, y ) is a point in the original coordinate plane XOY, (u , v) is the corresponding image point in the normal coordinate plane UOV, then the IPT formula between the two coordinate planes can be presented as follows [11]: h00 u + h01v + h02   h00 x = h u + h v +1  20 21 and H =  h10   y = h10 u + h11v + h12  h20  h20 u + h21v + 1 

h01 h11 h21

h02  h12  ≠ 0 1 

(8) where, H is the transformation matrix. In order to calculate the matrix H, four pairs of control points are needed. There are the coordinates ( xi , yi ) of the four vertexes of the Q1 and the corresponding ones (ui , vi ) of Q2 (i = 0,1, 2,3). Then, the coefficients can be calculated by solving the linear system as [11]:  u0 u  1 u2  u3 0  0 0   0

v0

1

0

0

0 −u0 x0

v1 v2 v3 0 0 0 0

1 1 1 0 0 0 0

0 0 0 u0 u1 u2 u3

0 0 0 v0 v1 v2 v3

0 0 0 1 1 1 1

−u1 x1 −u2 x2 −u3 x3 −u0 y0 −u1 y1 −u2 y2 −u3 y3

−v0 x0  −v1 x1  −v2 x2   −v3 x3  −v0 y0   −v1 y1  −v2 y2   −v3 y3 

 h00   x 0  h     01   x1   h02   x 2       h10   x 3   h11   y 0       h12   y1  h   y   20   2   h21   y 3  =

(9) According to the structure of QR code, we choose the four vertexes of QR code symbol P0 ( x 0 , y 0 ), P1 ( x1 , y1 ), P2 ( x 2 , y 2 ) and P3 ( x 3 , y 3 ) as the control points and set the points in the corrected image by:  u 0 = 0, v 0 = 0 u = W , v = 0  1 1  , u = W v 2 =W  2  u 3 = 0, v 3 = W (10) where, W = max{image _ width, image _ height} . The fig. 3, illustrates the situation:

Fig. 3: Inverse Perspective Transformation, (a). Original Image, (b). Corrected Image Where, a quadrilateral Q1 is shown on the coordinate plane XOY and a square Q2 is on the coordinate plane UOV.

(11) where, g (u , v) is the non-grid pixel and f ( x, y ), f ( x + 1, y ), f ( x, y + 1) and f ( x + 1, y + 1) are the neighboring pixels around. III. Experiment Result Image Source: Google open-Source 1D/2D barcode image processing library Zxing-1.5 CPU: Pentium(R) 4, 2.66GHz Software: MATLAB 7.1 The tested image shown in Fig. 4 (a) is the first QR code image named “1.jpg” from the qrcode-1 folder of Zxing-1.5. Fig. 4 (d) displays the result after performing the SHT. Fig. 5 displays the tested results of the adaptive method. The calculated adaptive value of G for the modified HT is 14.3 and k is 0.28. The tested result shows that due to the interference of the internal pixels of the QR code symbol, the eight ideal lines presented in fig. 2, will not be all detected in some cases. That is also the reason that eight lines are needed when performing the modified HT method, not four lines. We have also tested another 42 images from the qrcode-3 folder of Zxing-1.5. From our tests, we find that under in some cases, some un-uniform background can influence the detecting results, so we need to do some processing to the original image lines and finally get the external contour lines of the QR code, see fig. 6. 90% of the images can be corrected successfully.

Fig. 4: Tested Results of SHT, (a), An Original Image, (b). Result after Canny Edge Detection, (c). Corresponding Lines in ρoθ Plane With Points in XOY Plane (8 Squares Corresponding with the 8 Longest Lines), (d). The Location of the Longest Lines in the QR Code Image

E. Bilinear Interpolation After the geometric transformation (here is IPT) is performed to the distorted image, there must be non-grid points. As a result, an interpolation method is needed to keep the continuity of image feature. In this paper, the bilinear interpolation method is used to solve this problem. It can be described by the equation as below:

418

International Journal of Computer Science And Technology 

w w w. i j c s t. c o m

IJCST Vol. 3, Issue 1, Jan. - March 2012

ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print)

(a) 50 100

(b)

P0

P1

150 200 250 300 350 400

P3

P2

450 100

200

300

(c)

400

500

600

(d)

Fig. 5: Tested Results of the Adpative Method. The Calculated G is 14.3, k is 0.28, (a). The Gray Image of an Original Image in fig. 4, (b). The Result After Filtering, (c). The Result After Performing Algorithm 1 and Algorithm 2, (d). The Corrected Image P0

P3

(a)

(b) P1

P0

P2

P3

(c)

(d)

Fig. 6: The Tested Results of the Special Samples. (a) Test Image; (b) Result of Fig. 6 (a) After the Modified HT; (c)Test Image After the Cutting Processing; (d) Result of Fig. 6 (c) After the Modified HT IV. Conclusion In this paper, an algorithm for the distorted QR code image correction is presented. The gradient-based HT is modified when finding out the vertices of QR code symbol. It is achieved by 1) calculating the adaptive parameters G and k and getting the 8 longest lines (see Algorithm 1) and 2) finding out the four vertices of QR code symbol (see Algorithm 2). During the process, the QR code information is fully utilized. Compared with other methods, our method is simple and takes less computation time. It can achieve satisfactory results under various acquisition situations yet doesn’t need to add other complex algorithms. It is little affected by collinear point’s interspaces, noise, internal pixels of the QR code symbol and effectively increases the correctness when finding the contour lines of QR code symbol. As a result, the QR code symbol can be corrected precisely with the IPT algorithm. This modified method for the threshold G and k decision of HT (see Eq. (4, 5, 6)) is only applied to the situations that the size of QR code image is larger than T (see Eq. (3)). Further research will focus on the performance improvement of the small images.

w w w. i j c s t. c o m

References [1] ISO/IEC 18004:2006,“Information technology - Automatic identification and data capture techniques-QR Code 2005 bar code symbology specification”, 2006. [2] Article Numbering Centre of China,“Two-dimensional barcode technology and application”, China Metrology Publishing House, Beijing, (in chinese), pp. 106-108, 2007. [3] Yu-Hsuan Chang, Chung-Hua Chu, Ming-Syan Chen., “A General Scheme for Extracting QR Code from a non-uniform background in Camera Phones and Applications”, Ninth IEEE International Symposium on Multimedia, pp. 126128, 2007. [4] Aidong Sun, Yan Sun,“The QR-code reorganization in illegible snapshots taken by mobile phones”, Fifth International Conference on Computational Science and Application, pp. 532-536, 2007. [5] Ming Sun, Long-Shen Fu, Xin-Yan Yang, Shu-huai Zhang,“Image Analysis Method for QR Code’s Automatic Recognition”, Journal of University of Electronic Science and Technology of China (in chinese), Vol. 2, No. 6, pp. 1018-1019, Nov. 2009. [6] DU Xiu Wei, ZHANG Jing, ZHAI Jing Tong,“Algorithm of location and extraction of the QR code based on contour features”, Application of Electronic Technique (in chinese), Vol. 35, No. 6, pp. 47-49, 2009. [7] F. O’Gorman, M.B. Clowes,“Finding picture edges through collinearity of feature points”, IEEE Trans. Comput., Vol. 25, No. 4, pp. 449–456, 1976. [8] T.M. van Veen, F.C.A. Groen,“Discretization errors in the Hough transform”, Pattern Recognition 14, pp. 137–145, 1981. [9] JOHN S. DaPONTE, MARTIN D. FOX,“Enhancement of Chest Radiographs with Gradient Operators”, IEEE Transactions On Medical Imaging, Vol. 7, No. 2, pp. 110112, June 1988. [10] Hough, P.V.C.,“Methods and Means for Tecognizing Complex Patterns”, U.S. Patent 3 069 654, 1962. [11] W. K. Pratt.,“Digital Image Processing”, Computer Science Press, Inc., 1982. [12] Rafael C. Gonzalez, Richard E. Woods, Steven L. Eddins,“Digital Image Processing Using MATLAB”, Prentice Hall, 2004. [13] Tao Peng,“Detect lines in grayscale image using Hough Transform”, [Online] Availble: http://www.mathworks.com/ matlabcentral/fileexchange/9226-detect-lines-in-grayscaleimage-using-hough-transform. [14] GB/T 18284-2000,“National standard of the People’s Republic of China: Quick Response Code”, Issued by China State Bureau of Quality and Technical Supersion, 2000.

International Journal of Computer Science And Technology 

419

IJCST Vol. 3, Issue 1, Jan. - March 2012

ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print)

Yun-fei Liu received his B.S. degree in physics from Zhenjiang Normal College, Zhenjiang, China, in 1984, the M.S. degree in optics from Sichuan Normal University, Chengdu, China, in 1991, and the Ph.D. degree in testing measuring technology and instrument from Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2005. He was a teaching assistant, lecturer, associate professor, with Department of Basic Courses, Nanjing Forestry University, in 1991, 1994 and 2000 respectively. He was an associate professor, professor, College of Information Science and Technology, Nanjing Forestry University, in 2000 and 2007 respectively. His research interests include digital signal processing, electronic measurement techniques, microwave and optical technique. At present, He is engaged in Gaussian optics and terahertz technique in forestry application. Amit Kumar received his bachelor’s degree in Mathematics from the Himachal Pradesh University, Shimla, India, in 2002 and Masters’ degree in Computer Application from Kurukshetra University, Kurukshetra, India, in 2006. He completed his M.Phil. in Computer Science from Annamalai University, Annamalai nagar, Tamilnadu, India, in 2010. He is currently pursuing his Ph.D. in Computer Science. He is working as a Lecturer in the Department of Computer Science, College of Information Science and Technology, Nanjing Forestry University, Nanjing, China. He has many publications in National /International Conference proceedings and International Journals. He is a reviewer for many international Journals. His current interest includes Techno-Economic Analysis of Broadband Wireless Networks viz. WiMAX-m, HSPA+ and LTE-Advanced. His future focus is to explore the Green Wireless Technologies and their Sustainable development.

420

International Journal of Computer Science And Technology 

w w w. i j c s t. c o m