MOBILE COLOR BARCODE STREAMING FRAMEWORK

by JEREMY YANIK Advisor HALA ELAARAG

A senior research proposal submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in the Department of Mathematics and Computer Science in the College of Arts and Science at Stetson University DeLand, Florida

Fall Term 2012

TABLE OF CONTENTS

TABLE OF CONTENTS....................................................................................................ii ABSTRACT........................................................................................................................1 1. INTRODUCTION.......................................................................................................2 2. Related Work...............................................................................................................4 3. Mobile Color Barcode Streaming Framework.............................................................6 3.1 MoCoBar Encoding Process..................................................................................7 Barcode Specification..............................................................................................8 Blur Prevention......................................................................................................10 Frame Display........................................................................................................12 3.2 MoCoBar Decoding Process................................................................................13 Blur and Motion Detection.....................................................................................14 Corner Detection....................................................................................................15 Code Extraction......................................................................................................16 4. Conclusion..................................................................................................................17 REFERENCES..................................................................................................................19

ii

ABSTRACT

In this paper we present the Mobile Color Barcode (MoCoBar) Streaming Framework, a foundation for visible light communications for mobile devices. The framework encodes information into a high-density 2D barcode specifically designed to be decoded easily by mobile phones with cameras. Various image processing techniques are used in both the encoding and decoding processes in order to achieve higher transfer rates with fewer decoding errors. The framework is designed to decode barcodes in real time to allow for streaming up to 15 frames per second. The framework was developed and tested on the Android operating system, allowing for widespread adaptation.

1

1. INTRODUCTION

Today, consumers have readily available smart phones that are capable of reading many different barcode formats with ease. With the advent of open formats like QR Code[1] and Data Matrix[4], the use of barcodes is becoming an increasingly popular medium for one-way, short range communications. This is achieved by storing bit patterns in a specified format and decoding that format with an optical device, like a camera. The amount of data stored in a single barcode restricts their storage capabilities to short strings such as URLs or contact information [1]. To increase the storage density, all sorts of different formats have been proposed, typically involving the addition of colors and shapes. However, other implementations take this much further. One such implementation is Color Barcode Streaming for Smartphone Systems (COBRA) [2]. Communication via COBRA requires two devices, a sender and a receiver, which typically consist of two off-the-shelf smartphones. The sending device relays a message by displaying a stream of barcode images while the receiving device decodes the images to reconstruct the message. In this paper we propose a new short range communication scheme that is based on mobile phones barcode streaming. Our technique is similar to COBRA, but applies different barcode formats, image processing algorithms and measures for error prevention and correction. Our goal is to form a point of comparison for other projects, improve data transfer rates and reduce transmission errors. Specifically our framework will contribute the following:

2

1. We develop a barcode specification to maximize data density while maintaining readability. The framework also utilizes open standard barcode specifications such as QR and DataMatrix. A point of comparison using a specification that has already been widely adopted is important to consider while evaluating our results. 2. We examine color usage and error correction schemes and study how these changes improve data transfer rates and and reduce the frequency of errors; two important performance metrics. 3. We implement widely adopted corner detection algorithms in an attempt to achieve more accurate or faster corner detection. Real-time corner detection is achieved by utilizing the last known location for corners as a starting point for each barcode frame. This methodology is referred to as “fast corner detection.” 4. We apply algorithms for motion detection and blur correction. Most smartphones have accurate accelerometer sensors that can be used to predict movement. If we know that the device is moving, we may apply such blur correction to improve the readability of an image before we attempt to decode it. We use accelerometer data to estimate the potential new location of corners for fast corner detection. 5. We evaluate the reliability of low-quality cameras and develop the functionality of a two-way communication scheme

The rest of this paper is organized as follows: in Section 2 we cover related works in regard to Visible Light Communication. Section 3 presents an overview of MoCoBar’s framework followed by details of the encoding and decoding processes on mobile devices. In Section 4, we provide a conclusion of the paper.

3

2. Related Work

Visible Light Communication (VLC) has become a topic of importance as it is poised to compete with existing technologies for short-range communications such as radio frequency. In fact, the Visible Light Communication Consortium (VLCC) has been meeting annually since 2003. However, the focus of VLC in embedded devices is a recent emergence. Smart phones are becoming more powerful; the costly computations for image processing are now an easy feat for today’s average smartphone. The most commonly known and globally adopted barcode is the Universal Product Code (UPC). They are composed of vertical lines that represent a numeric code up to 12 digits [3]. These are what we will refer to as 1-Dimensional barcodes; they store data in a single dimension. Special barcode readers are purchased by retailers to read the codes. The UPC can be seen in use every day at supermarkets and retailers. However, their storage limitations prohibit their use for much else than simple numeric tracking. Additionally, these numeric codes are meaningless without some sort of database containing information for corresponding product codes. More recently, 2D barcode standards have emerged. Some of the popular formats are QR Code [1], Data Matrix [4] and Aztec Code [5]. The introduction of 2D barcodes allows for raw binary data to be stored. In most cases, data is not restricted by type (numeric, string, etc.) but by a maximum number of bits. The second dimension also means that the barcodes have a greater data density. Fig. 1 compares a 12-digit UPC to the same code represented as a QR Code.

4

Figure 1: UPC-A (left) and QR Code (right) Visual Comparison.

Many standards have sprung to life in order to achieve increased storage capacity in 2D barcodes. Numerous high-density barcode formats have been proposed, but none have been as widely adopted as formats such as QR Code. COBRA’s barcode specification would be considered high-density [2]. Multiple colors are used and the colors are arranged in a way to reduce blur. However, COBRA takes this a step further by streaming many barcode images to transmit a dense message [2]. One of the problems with high density barcodes is that they become difficult to decode, particularly in embedded devices. The most expensive operations in decoding an image of a high-density barcode are all related to correcting distortions via image processing [6]. Some of these operations include perspective correction, blur correction and various color corrections. The goal is to take a camera image and convert it, via image processing, into a usable form. There are a number of related projects that research the best way to achieve this. Aside from COBRA, there is little research on the topic of streamed barcode images, particularly in the mobile environment. PixNet’s research is an early example of research in the field of VLC. PixNet is a system for transmitting data over LCD screens and camera links [6]. The project was not intended to be utilized on mobile platforms, but 5

it does provide incredibly useful insight into the world of VLC. It discuss useful perspective correction techniques and algorithms that our framework makes use of. The research by Chu, Yang and Chen for image stabilization on mobile devices is incredibly important to our framework. In [14], the authors present algorithms for motion correction of images coming from camera links on smart phones. The algorithms use data from accelerometer sensors in order to estimate and restore an image that was blurred by the movement of the device [14]. Similar research for blur detection and estimation is presented in [15] and [16]. However, the algorithms themselves are not suitable for real time processing. A lot of research has been done for corner detection of 2D barcodes. Many works propose corner detection algorithms that use algorithms such as hough transformations [11] or various edge detection algorithms [8]. Analyzing these techniques, we have determined that algorithms such as these are not suitable real time detection. Rather than using expensive image processing algorithms, researchers were able to build fast corner detection systems based on pattern detections [2]. However, the implementation of pattern-based corner detection requires reliable, known visual structures to be present in images.

3. Mobile Color Barcode Streaming Framework

In this section we present our MoCoBar framework. The entire process takes place as follows. A header is attached to an arbitrary data message before it is encoded by the sending device. The encoding process generates multiple frames containing barcode 6

images. The images of each barcode are displayed on the device’s screen in rapid succession, an animation. A receiving device’s camera is pointed at the screen of the sending device. Images of the barcodes are processed and decoded on the receiving device. Once all frames have been received, the data message will be reconstructed. The most challenging aspect for reading and writing barcodes is held within two major processes: encoding and decoding. Encoding is the generation and display of a barcode that depicts a piece of data [6]. Decoding is capturing the encoded image via a camera link and converting that image back into data [6]. Each process has its own difficulties and there are a number of ways to overcome them. In the remainder of section 3, we discuss the details of MoCoBar’s encoding and decoding processes. Within the encoding process there are three main components, determining which barcode variant to use, blur prevention and frame display. The image processing algorithms included in the decoding process is the most expensive computational component. The low computer power of mobile devices must be taken into consideration, especially within real-time applications. There are also three main components for MoCoBar’s decoding process; blur and motion detection, corner detection and code extraction.

3.1 MoCoBar Encoding Process

Encoding is a more trivial task than decoding. However, encoding data properly is crucially important to aid the decoding process. Particularly if the decoding process must be done in real time. As mentioned earlier, one major difficulty presented by encoding is 7

choosing which barcode specification to use. Each barcode has different features when it comes to readability, data density and error correction. In Section 3.1.1, we compare three widely adopted barcode standards we consider for this research. After choosing a specification, the image must be displayed on the screen. There are some steps that can be taken to treat the image before it is displayed. We present these steps in section 3.1.2. Finally we discuss the frame speed component in section 3.1.3.

3.1.1 Barcode Specifications

There are a number of popularized barcode specifications that we consider for our implementation. Codes such as QR [1], DataMatrix [4] and Aztec [5] are either in the public domain or have un-enforced patents. Each barcode specification has different levels of data density and error correction. In many cases, even slightly damaged barcodes can be read. In Table 1, compare the three aforementioned popular barcode specifications in terms of maximum storage, error correction and block count. In this table, Block Count refers to the number of data blocks used to display a barcode given our sample. The Block Count is calculated by counting every “block” or “module” within each barcode using the lowest error correction possible provided by the specification. This includes modules for corners, timing and other features.

8

Barcode

Max Storage

Error Correction

Block Count

QR Code

2953 bytes

7 – 30%

8281

DataMatrix

1555 bytes

0 – 25%

8464

Aztec

1914 bytes

0%

3249

Table 1: Popular barcode specification information.

Our framework also supports our high-density MoCo Barcode (Figure 2). We have designed the barcode with streaming in mind. As such, we have used some of the key features found in QR and DataMatrix to create a robust code with high data density. Three corners feature the Corner Module, a colored square that can easily be located and used to identify orientation. The barcode also features Timing Modules that assist in the decoding process to help form a sort of grid. A majority of the barcode is the Data Area, an area filled with colored data modules that represent bits (encompassed by a translucent line in Figure 2). Each color module in the Data Area represents 2 bits for a RGB color scheme. In the specification, the colors white, red, blue and green represent the bits 00, 01, 10, and 11 respectively. The specification also supports an alternate color scheme for 3 bits per module using additional colors. However, we explain later why less colors may result in a higher data transfer rate. The read order of data modules is horizontal, starting at the leftmost module on each line. The data density of the MoCo Barcode is limited by the resolution at which it is being displayed. As such, it makes sense that if we use the smallest possible module size (1 pixel) we can achieve a relatively high data density. Such a small module size would not be possible in reality. The quality of smartphone cameras and the level of blur require 9

the size of modules to be greater [2][6]. Module size may vary depending on hardware specifications such as screen size, screen resolution and camera quality. The design of the MoCo Barcode is flexible enough to support any size module.

Figure 2: A sample MoCo Barcode and its features.

3.1.2 Blur Prevention

Although the topic is unimportant for processing single barcode, the blur effect is very important to keep in mind when the intent is to rapidly decode many high density barcodes. Blur can lead to inaccurate detection and reduced decoding rates [2]. Within the encoding process, we are not concerned with blur caused by the movement of a device. Instead, we are more interested in preventing blur caused between data blocks of different color. This is of particular importance when using color barcodes and high resolutions [2]. 10

The most obvious way to deter the effects of blurring between different data modules would be to adjust the size of the block. To have a minimal amount of blur, we use larger block sizes. However, a larger block size would cause data transfers rates to drop. We must utilize the smallest block size that does not cause too great of a blur effect. This is a challenge as it is highly dependent on two variables: the distance between the sender and receiver and the quality of the camera link. Without the use of two-way communication, it is difficult to evaluate these variables in the real world. With this in mind, we’ve developed the framework to work with a 5 Megapixel camera at a distance of 1 to 2 feet.

Figure 3: Example of data represented in its original color order (left) and with a stride of 3 (right).

As we have stated earlier, blur is caused between data blocks that are different colors. By reducing the length of the border between colors overall, it is possible to reduce the potential blur an image may contain once received [2]. We can achieve this by setting using data strides. This will cause the order in which bits are stored to be altered, possibly resulting in an image with a shorter border length between different colors. 11

Figure 3 demonstrates how the stride will affect the output. Unlike the technique in [2], we apply our algorithm in 2 dimensions and do not rely on random strides. When N represents the number of blocks in the data area, we can compare all possible strides from 2 to N/2, as well as the original. We can quickly compare each stride by counting the number of neighbors each block has that are of a different color. We call this total the Blur Potential (BP) of the frame. BP can be calculated using the algorithm below in Fig. 4. The framework uses the stride that contains the lowest BP.

BP = 0 for i = 0 to N - 1 do if i has a right neighbor AND it is a different color then BP += 1 end if if i has a bottom neighbor AND it is a different color then BP += 1 end if end for

Figure 4: Blur Potential calculation.

3.1.3 Frame Display

The final step for encoding is the actual display of a barcode. As stated earlier in Section 3, once barcode images are generated, they will be displayed in an animation on the sending device’s screen. In a practical scenario, the speed of the animation should be 12

much lower than that of the processing speed of a single frame. There is great variability in the computing capabilities of different mobile devices, so the exact display frame rate and decoding rate will vary. Generally, it is agreed that the sending device display at roughly half the framerate that is possible on the decoding device [2, 6]. In our tests, the sending device updates its image at 15 frames-per-second (FPS) and the receiving device attempts to read at 30 FPS.

3.2 MoCoBar Decoding Process

Decoding has greater computational difficulties than encoding. The goal here is taking a camera-link image and process it to best match the original encoded image. It is not feasible to form a pixel-to-pixel match [6]. As a result, the goal is to get close enough to the original image so that we can use the barcode specification to retrieve the encoded data. This can be achieved by applying a number of image pre-processing techniques. Before attempting to decode an incoming frame, it is important to determine how blurred the image is and if blur correction algorithms should be used. If the frame is suitable for decoding, the actual barcode must be located within the image. This is normally achieved through corner detection algorithms. Once the barcode has been isolated, the code may be extracted.

13

3.2.1 Blur and Motion Detection

Blur detection is a rather expensive and computationally heavy process. The authors in [14] have come up with a very accurate algorithm to detect the blur effect caused by movement and loss of focus. However, the algorithm does not appear simple enough to be run in real-time. Rather than run expensive algorithms, we can assume that if corner detection fails that the image may be blurred. Blur correction algorithms, such as the barcode restoration algorithm presented in [14], have made their way to mobile environments. These algorithms use light traces within images in order to estimate parameters. Even better, accelerometer sensors allow us to detect motion. If motion can be detected, blur effects can be reduced. The authors in [14] have also published an algorithm to counteract blur based on the movement of a device. An additional blur correction and detection algorithm is presented in [16]; however, this algorithm was only tested for single-direction linear movement. Autofocus is becoming increasingly available on mobile devices. This feature allows the camera video feed to automatically focus to help prevent blurring. If autofocus is available, it should be used to alleviate unneeded computational overhead. Moreover, if autofocus features are not available, it may be feasible to request camera hardware to perform a focus operation as the ZXing library demonstrates [13].

14

3.2.2 Corner Detection

Corner detection is a vital process for the receiving device. The corner detection algorithm needs to be accurate and fast. The topic is not a new one, but attempting to locate corners in real-time has some challenges. The complexity of corner detection algorithms are exacerbated by the larger image resolutions on mobile devices today. One of the most popular corner detection algorithms is the Hough Transformation. As noted by [2] and made obvious in ZXing’s corner detection [13], algorithms that detect all edge features are not suitable for real-time decoding. Both [2] and [13] implement unique detection algorithms specific for the barcode specification in question. This ensures that computing time is not wasted finding every edge in an image. For example, ZXing [13] begins corner detection from the center of an image and moves outward searching for specific features based on barcode symbology. Another algorithm published for real-time detection was based on contour tracing. Contour tracing can be used in combination with an algorithm by D. M. Tsai et al [8]. The goal of the authors in [8] was to achieve robust detection of corners based on shapes (as opposed to edges). Our framework takes corner detection a bit further. Because images are being decoded at high rates, it can be assumed that the corners in the current frame are very close to location of the corners in the previous frame. Corner detection begins at the location of the previous corners instead of the center or corners of the entire image. This greatly reduces the time needed to locate barcode corners. Of course, this means the first frame to be decoded will require more computation. In order to ensure speedy decoding, 15

an accurate be inexpensive corner detection algorithm should be used for real-time processing [2].

3.2.3 Code Extraction

The final step in the decoding process is code extraction or signal detection [6]. Once an image is processed and corners are found, a perspective transformation is performed. The perspective transformation uses the corners detected previously in order to resize the code area of a barcode into a rectangle [7][9]. Figure 5 demonstrates the perspective transformation.

Figure 5: QR Code image from camera (left) and the same image after perspective transformation (right).

After the image is transformed into a usable grid, the signals must be interpreted and read. One popular way to read 2D barcodes is achieved via binarization. With a monochromatic barcode image, binarization is used to reduce the image to black and white extremes [7]. Liu showed that some binarization algorithms may be used for real16

time processing of barcodes [7]. Now, the signal of each data module may easily be detected and the barcode is decoded. However, binarization only works for monochromatic barcodes. High density color barcodes rely heavily on timing modules. Some monochrome 2D barcodes, like DataMatrix, include timing modules. Figure 5 (right) shows a DataMatrix barcode with the data area dimmed so that timing blocks can be seen easily. These timing blocks are essential for decoding while streaming high density barcode images [2].

4. Conclusion

In this paper we propose a framework for Mobile Color Barcode Streaming. Our framework utilizes open barcode formats like QR, Aztec and DataMatrix. We design and implement a high density barcode specification with attributes that make it easier to decode in real time. This format will include easily identifiable corners and timing mechanisms as well as a large data area. We will employ error correction techniques and experiment with the alternate use of color selection and error correction techniques to maximize data density while maintaining readability. To achieve corner detection in real time, our framework will utilize pattern detection as opposed to expensive algorithms like Hough transformations or edge detection. With the use of colors in the barcode specification, searching for a pattern containing a specified color will yield faster results. We will utilize a fast corner detection approach to further enhance the corner detection process.

17

We will analyze the use of motion detection via accelerometer sensors to estimate the new locations of corners and provide parameters to be used in reducing blur. Primarily, we will use motion detection to run algorithms that reduce blur. In addition, we will also utilize error correction within the barcode specification to fight blur. We will evaluate our framework on low-quality cameras. Many smartphones today have both front and rear facing cameras. However, front-facing cameras are typically a much lower resolution than rear facing cameras. The use of front-facing cameras allow us to achieve two-way communications using our framework. We will develop a simple communication protocol to achieve this.

18

REFERENCES

[1]

I. 18004:2006. Automatic identification and data capture techniques - QR code 2005 bar code symbology specification.

[2]

I. 16022:2006. Automatic identification and data capture techniques - Data Matrix barcode symbology specification.

[3]

T. Hao, R. Zhou, G. Xing. COBRA: color barcode streaming for smartphone systems. In Proceedings of the 10th international conference on Mobile systems, applications and services, 2012.

[4]

I. 15420:2009. Information technology - Automatic identification and data ` capture techniques – EAN/UPC bar code symbology specification.

[5]

I. 24778:2008. Automatic identification and data capture techniques - Aztec Code bar code symbology specification.

[6]

S. Perli, N. Ahmed, and D. Katabi. Pixnet: interference-free wireless links using lcd-camera pairs. In Proceedings of the sixteenth annual international conference on Mobile computing and networking. ACM, 2010.

[7]

C. Chu, D. Yang, and M. Chen. Image stabilization for 2D barcode in handheld devices. In Proceedings of the 15th international conference on Multimedia. Multimedia, 2007.

[8]

H. Tong, M. Zhang, and C. Zhang. Blur Detection for Digital Images using Wavelet Transformation. Multimedia and Expo, 2004. 19

[9]

D. Nguyen, D. Bui, and T. Ha Le. Blur Estimation for barcode Recognition in Out-of-Focus Images. Pattern Recognition and Machine Intelligence, 2011.

[10]

H. Wang and

M. Brady. Real-time corner detection algorithm for motion

estimation. Image and Vision Computing, 1995. [11]

X. Li, M. Zou, and J. Zhou. The multi-QR codes extration method in illegible image based on contour tracing. Anti-Counterfeiting, Security and Identification, 2011.

[12]

ZXing ("Zebra Crossing"). http://code.google.com/p/zxing/.

[13]

Y. Liu, J. Yang, and M. Liu. Recognition of QR Code with mobile phones. Control and Decision Conference, 2008. IEEE, 2008.

[14]

A. Sun, Y. Sun, and C. Liu. The QR-code reorganizatio in illegible snapshots taken by mobile phones. Computation Science and its Applications, 2007. IEEE, 2007.

20