http://www.paper.edu.cn

USB Image Acquisition System Based on CMOS Image Sensor Wang Kejun，Zhang Xiaolei Pattern Recognition Laboratory，Harbin Engineering University，Harbin（150001） E-mail：[email protected]

Abstract Using the OmniVision OV7620 CMOS image sensor as photoelectric imaging device, glueless connecting was achieved between the USB2.0 controller CY7C68013A and the sensor by general programmable interface (GPIF). The simple high efficiency and low-cost hardware which be apt for high speed image capture was designed and the software system was also compiled based on the multi-thread technology. The image processing system with acquisition, transmission and display was designed based on the new type CMOS image sensor and USB2.0 controller. Keywords：CMOS image sensor，USB，CY7C68013A，GPIF

1 Introduction Because of the rapid development of the semiconductor technics, Complementary Metal Oxide Semiconductor (CMOS) [1-3] image sensor makes progress quickly in recent years because of its low cost, small volume, light weight, low power consumption and single chip integration compared with traditional CCD [4-5] sensor. It has become the key facility of image processing technology and has been applied to a great many fields such as video camera, digital camera, mobile communication, safeguard, and pattern recognize. In illustration of USB Image acquisition system, OmniVision single chip OV7620 is described in this paper, including its features, operate mode and usages. The Universal Serial Bus (USB) has gained wide acceptance as the connection method of choice for PC peripherals. Equally successful in the Windows and Macintosh worlds, USB has delivered on its promises of easy attachment, an end to configuration hassles, and true plug-and-play operation. USB brings the ultimate in implicity when connecting peripheral devices. The longer needs to determine the right connection for the mouse versus the keyboard. There isn’t a different cable for the monitor, printer, or high storage capacity external disk drive. The user never has to determine if the connection is a parallel or serial port. In the world of USB, there are no longer dip switches, jumpers, IRQ conflicts, and DMA conflicts. In fact, users will always be able to connect or disconnect a new device when the PC is up and running. Now that’s true Plug and Play! Because of this consistency and simplicity of connection, USB will make it easy to add peripheral image acquisition system that was previously thought to be too difficult for the average user. To sum up, it has been very important and necessary to integrate all the advantages of CMOS image sensor and USB interface into the image acquisition system. In this paper, we use the OmniVision OV7620 CMOS [6] image sensor and Cypress FX2LP high-speed USB2.0 CY7C68013A [7-8]as the main chips to achieve our design.

2 Hardware Design 2.1 CMOS Image Sensor OmniVision OV7620 is a highly integrated high resolution (640x480) Interlaced / Progressive Scan CMOS digital color /black&white video camera chip. The digital video port supports 60Hz YCrCb -1-

http://www.paper.edu.cn

4:2:2 16Bit / 8 Bit format [9], ZV Port output format, RGB raw data 16Bit/8Bit output format and CCIR601/CCIR656[10-11] format. The builtin SCCB interface provides an easy way of controlling the builtin camera functions.

2.2 USB2.0 Controller Cypress Semiconductor Corporation’s EZ-USB FX2LP™ (CY7C68013A) is a low-power version of the EZ-USB FX2™ (CY7C68013), which is a highly integrated, low-power USB 2.0 microcontroller. By integrating the USB 2.0 transceiver, serial interface engine (SIE), enhanced 8051 microcontroller, and a programmable peripheral interface in a single chip, CY7C68013A has created a very cost-effective solution that provides superior time-to-market advantages with low power to enable bus powered applications. The ingenious architecture of CY7C68013A results in data transfer rates of over 53 Mbytes per second in theory, the maximum-allowable USB 2.0 bandwidth, while still using a low-cost 8051 microcontroller in a small package. Because it incorporates the USB 2.0 transceiver, the CY7C68013A is more economical, providing a smaller footprint solution than USB 2.0 SIE or external transceiver implementations. The Smart SIE handles most of the USB 1.1 and 2.0 protocols in hardware, freeing the embedded microcontroller for application-specific functions and decreasing development time to ensure USB compatibility. The General Programmable Interface (GPIF) [12]and Master/Slave Endpoint FIFO (8- or 16-bit data bus) provides an easy and glueless interface to popular interfaces such as ATA, UTOPIA, EPP, PCMCIA, and most DSP processors.

2.3 Principle of USB Image Acquisition System There are three parts of the USB image acquisition system.

Referring to Fig 1.

Figure 1: USB Acquisition System Block Diagram

2.3.1 Acquisition Part: The light from outside scenery is converged to CMOS pixel array through lens. Each compact active pixel element has high photo-sensitivity and converts photon energy to analog voltage signal. The sensor OV7620 has on-chip Analog to Digital Converter and comparators to digitize the pixel output. The output rate of CMOS image sensor is relatively high. It is required that USB controler's rate match the sensor's. There are several signals to be considered in system design: 8 bit image data, PCLK , VSYNC , HREF, and I2C bus etc. The majority of signal processing in OV7620 is performed in the analog processing block, which does color separation, matrixing, AGC, gamma correction, color correction, color balance, black level calibration, knee smooth, aperture correction, controls for the luminance and chrominance picture and anti-alias filtering. The analog video signals are based on the formula: -2-

http://www.paper.edu.cn

Y = 0.59G + 0.31R + 0.11B; U = R-Y; V = B-Y; [13] In our design, It is not necessary to capture the color image.The Y channel is enough to satisfy the system requirement, So there are only image data of Y channel can be transmit to the USB controller with the 8 bit mode on the data bus. PCLK , VSYNC , and HREF signals are the main hand-shake signals of control interfaces. Standard video timing signals include VSYNC, HREF, PCLK, FODD and CHSYNC. Here VSYNC is asserted high during several scan lines in the vertical sync period, after that will be the HREF, while image data clocking out at the pace of PCLK, HREF will assert high till all the data in a frame are transmitted out. By default, data is updated at the falling edge of PCLK and is stable at its rising edge. PCLK runs at the pixel rate (normally at 13.5M when crystal clock frequency is 27MHz). Data bus including 8 data lines, the luminance (Y channal) data is clocked out of this bus at the rate of one byte per pixel. I2C bus (inter integrate SCCB bus) is a communication technology adopting two lines (SDAT and SCLK). I2C bus transmitting rate is up to 400kbps. It supports CMOS process, thereby CMOS image sensor contains I2C bus interface and need not design it.

2.3.2 Transmission Part: To achieve the maximum sustained throughput in USB 2.0 High Speed designs, the physical interconnect should never be the primary bottleneck in the system. CY7C68013A 's GPIF (General Programmable InterFace) provides a highly configurable and flexible glueless peripheral interface that allows the highest possible bandwidth to be achieved over the physical layer. In our design, the OV7620’s PCLK pin will connect to GPIF’s IFCLK pin which can be either an input or output signal and is determined by what the system requirements are; The OV7620’s VSYNC and HREF will connect to GPIF’s RDY0 and RDY1 respectively. To achieve the maximum bandwith of the transmission, it will configure the GPIF’s CTL0 signal as the master strobe of FLOWSTATES.

2.3.3 Display Part: The development environment of software is Borland C++ Builder 6.0 under Windows. The application software capture the image data from the USB cable and display on the screen.

3. Software Design 3.1 Firmware In order to simplify the design and make the engineers concentrate their attention on the function not to waste their time on the complex protocol, it is necessary and convenient to use the integrate framework. The framework is helpful to the research worker to realize the most parts of the USB protocol. According to peripheral device and requirement, we just need to write different functions to finish our design. To be aimed at the high-speed data and the USB 2.0 protocol, there are several aspects in our firmware software design: 3.1.1 Initial The USB Port and Fifo Register: Set the 8051 CPU clock to 48MHz. Set the endpoint2 IN, bulk, size 512, 4x buffered. Set NAKALL bit to NAK all transfers from host. Reset EP2 FIFO and clear NAKALL bit to resume normal operation. Set EP2FIFOCFG register auto in mode, disable PKTEND zero length send, word ops. -3-

http://www.paper.edu.cn

Initialize GPIF registers For EP2IN, GPIF uses FF flag. Initialize global FLOWSATES register. Parts of the program are as follows. CPUCS = ((CPUCS & ~bmCLKSPD) | bmCLKSPD1); SYNCDELAY; EP2CFG = 0xA0; SYNCDELAY; FIFORESET = 0x80; SYNCDELAY; FIFORESET = 0x02; SYNCDELAY; FIFORESET = 0x00; SYNCDELAY; EP2FIFOCFG = 0x10; GpifInit (); EP2GPIFFLGSEL = 0x01; SYNCDELAY; EP6GPIFFLGSEL = 0x02; FLOWLOGIC = FlowStates[19]; FLOWSTB = FlowStates[22]; GPIFHOLDAMOUNT = FlowStates[26]; FLOWSTBEDGE = FlowStates[24]; FLOWSTBHPERIOD = FlowStates [25]; SYNCDELAY; …… 3.1.2 Setup The GPIF Interface Engine: Except the initial code in the function Gpifinit(), the function code void TD_Poll(void) must be written in the firmware program. It is the kernel code to ensure the GPIF engine operates smoothly. In the TD_Poll (void) function, we define the condition of when the GPIF engine start and how many times it operate. Parts of the program are as follows. void TD_Poll(void) { if( GPIFTRIG & 0x80 ) // if GPIF interface IDLE { if(in_enable) // if IN transfers are enabled { if ( GPIFTRIG & 0x80 ) // if GPIF interface IDLE { if ( EXTFIFONOTEMPTY ) // if external FIFO is not empty { if ( !( EP24FIFOFLGS & 0x01 ) ) { if(enum_high_speed) -4-

http://www.paper.edu.cn

{ GPIFTCB1 = 0x01; SYNCDELAY; GPIFTCB0 = 0x00; SYNCDELAY; } Setup_FLOWSTATE_Read(); SYNCDELAY; GPIFTRIG = GPIFTRIGRD | GPIF_EP2; SYNCDELAY; while( !( GPIFTRIG & 0x80 ) ) { ; } SYNCDELAY; …… There is another software GPIF TOOL use to generate the C code file which was compiled in the firmware project. Referring to Fig 2.

Figure 2: GPIF TOOL Software Interface

GPIF is an internal master to the CY7C68013A’s endpoint FIFOs. It replaces the external “glue” logic which might otherwise be required to build an interface between the CY7C68013A and the outside world. At the GPIF’s core is a programmable state machine which generates up to six “control” and nine “address” outputs, and accepts six external and two internal “ready” inputs. Four user-defined Waveform Descriptors control the state machine; generally (but not necessarily), one is written for FIFO reads, one for FIFO writes, one for single-byte/word reads, and one for single-byte/word writes.Only the FIFO reads waveform have been used in our design. Referring to Fig 3.

-5-

http://www.paper.edu.cn

Figure 3: FIFO Reads Waveform Interface

In order to achieve the maximum USB 2.0 High-speed bandwidth, the CPU (in this case an enhanced 8051) should never be directly involved in moving the payload data from the external slave device to the USB host and vice versa. The CPU would clearly be the largest bottleneck in a High-speed design. So instead we adopt an Auto-Transfer mode, whereby the payload data is “auto-committed” from the external slave device to the USB host and likewise in the other direction. The GPIF engine uses this Auto-Transfer mode to move data to and from the external slave device. As mentioned above, the GPIF engine generates the GPIF waveform descriptors that implement the physical bus timing. The GPIF engine can implement multiple waveform behaviors, which can then be assigned to four waveform types. The assignment is made through the GPIFWFSELECT register setting. Each descriptor is 32 bytes long and resides in a special GPIF waveform descriptor area in on-chip memory space, once loaded by the CPU. When we finish the GPIF design, these descriptors will be add to firmware project and compiled soon.

3.2 Application Program: When first plugged into USB, the CY7C68013A enumerates automatically and downloads firmware and USB descriptor tables over the USB cable. Next, the CY7C68013A enumerates again, this time as a device defined by the downloaded information. This patented two-step process, called ReNumeration™, happens instantly when the device is plugged in, with no hint that the initial download step has occurred. At the same time, pc get the PID/VID code which use to identify the USB device through CYUSB.INF [14] and CYUSB.SYS [15]. The CYUSB.INF file can be modified to accomplish several different objectives.To make the driver match to a specific device, the device's vendor ID and product ID need to be added to the CYUSB.INF file. For instance, :%VID_VVVV&PID_PPPP.DeviceDesc%=CyUSB,USB\VID_ VVVV&PID_PPPP. Change the VVVV to contain the hexadecimal value of the VendorID for the device. Change the PPPP to contain the hexadecimal value of the ProductID for the device. CYUSB.SYS is a USB device driver. The API library is only able to communicate with USB devices that are served by this driver. we must force Windows to use the CYUSB.SYS driver with the device.If Windows prompts for a driver or indicates that it needs a driver, direct the PC to use the CYUSB.SYS driver by steering it to the CYUSB.INF file in the Driver directory.If Windows does not -6-

http://www.paper.edu.cn

prompt for a driver, it has already matched the device to a driver itself. In this case, manually instruct Windows to use that driver. The application software was compiled by Borland C++ Builder 6.0. CYAPI.H [16] and CYAPI.LIB was cited in the software for opening the device and transmiting data. Take accout of the high-speed data rates, double buffer technology and scanline method were adopt in the multi-thread software. Parts of the program are as follows. …… Graphics::TBitmap*tmpBitmap=newGraphics::TBitmap(); tmpBitmap->Assign((TPersistent*)Image1->Picture->Graphic); tmpBitmap->PixelFormat=pf24bit; …… CCyUSBDevice *USBDevice = new CCyUSBDevice (Handle) ; //Get the devive handle and open the device. USBDevice->~CCyUSBDevice(); //Close the USB device. …… USBDevice->BulkOutEndPt->XferData(buf, length); //Asynchronous transfer mode. BeginDataXfer(buffer, length, &outOvLap); …… BeginDataXfer is an advanced method for performing asynchronous IO. This method sets-up all the parameters for a data transfer, initiates the transfer, and immediately returns, not waiting for the transfer to complete. BeginDataXfer allocates a complex data structure and returns a pointer to that structure. FinishDataXfer de-allocates the structure. Therefore, it is imperative that each BeginDataXfer call have exactly one matching FinishDataXfer call. There are two threads in the application software. Capturing the image data occupy one thread specially. With this method, we can get the image data smoothly and avoid no respond to system when transmit the USB data. The application software display interface. Referring to Fig.4

Figure 4: Application Software Interface -7-

http://www.paper.edu.cn

4. Summary This USB image acquisition system adopt the new type CMOS image sensor OV7620 and USB 2.0 highspeed controller CY7C68013A to realize the design which has created a very cost-effective, simple high efficiency and low-cost solution compare with the traditional method. It can be applied to a great many fields such as video camera, automatic device, robot, mobile communication, safeguard, and pattern recognize conveniently and flexibly.

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]

[13] [14] [15]

Abbas El, Gamal Helmy Eltoukhy, “CMOS image sensors,” IEEE Circuits & Devices, pp. 6-20, May/June 2005. B. Fowler, A. El Gamal, and D.X.D. Yang, “A CMOS area image sensor with pixel-level A/D conversion,” IEEE Int. Solid-State Circuits Tech, Dig., pp. 226–227, 1994. N. Stevanovic, M. Hillegrand, B.J. Hostica, and A. Teuner, “A CMOS image sensor for high speed imaging,” in ISSCC Tech. Dig., vol. 43, pp. 104–105, 2000. N. S. Saks, “A technique for suppressing dark current generated by interface states in buried channel CCD imagers,” IEEE Electron Device Lett, vol. 1, pp. 131–133, July 1980. OV7620 Product Specifications-Rev. 1.3 2000.http://www.ovt.com. CY7C68013 EZ-USB FX2 USB Microcontroller Data Sheet Rev. *B. Cypress Semiconductor Corporation, June 2002. The EZ-USB FX2 Technical Reference Manual Version 2.1. Cypress Semiconductor Corporation, 2001. Cliff Reiter, "With J: image processing 2: color spaces," ACM SIGAPL APL Quote Quad, Vol.34, Issue 3, pp3-12, June 2004. James M. Kasson, Wil Plouffe, "An analysis of selected computer interchange color spaces," ACM Transactions on Graphics (TOG), Vol.11, Issue 4, pp373-405, October 1992. Didier Le Gall, "MPEG: a video compression standard for multimedia applications," Communications of the ACM, Vol.34, Issue 4, pp47-58, April 1991. EZ-USB FX2 GPIF Primer. Cypress Semiconductor Corporation, March 2003. B. Hill, Th. Roger, F. W. Vorhagen, "Comparative analysis of the quantization of color spaces on the basis of the CIELAB color-difference formula,"ACM Transactions on Graphics (TOG), Vol.16, Issue 2, pp109-154, April 1997. Cypress CyUsb.sys Programmer's Reference.2003 Cypress USB Console Users' Guide.2003. Cypress CyAPI Programmer's Reference.2003.

-8-