Kinect Based Obstacle. Avoider Robot

Kinect Based Obstacle Avoider Robot Kinect Based Obstacle Avoider Robot Final Report Prepared At: Innovation Cell, IIT Bombay Team Members:  Nilesh...
Author: Lucy Cummings
5 downloads 0 Views 979KB Size
Kinect Based Obstacle Avoider Robot

Kinect Based Obstacle Avoider Robot Final Report

Prepared At: Innovation Cell, IIT Bombay Team Members:  Nilesh Ugale  Shubham Adhau 1

Kinect Based Obstacle Avoider Robot

Acknowledgement We would like to thanks Prof. Abhishek Gupta for giving us opportunity for this project.

We would like to extend our sincere thanks to Arpit Gupta for his guidance and support in completion of this project.

We would like to express my deepest appreciation to all those who provided me the possibility to complete this Project. We are highly indebted to Innovation Cell IIT, Bombay for their guidance and constant supervision as well as for providing necessary information regarding the project & also for their support in completing the project .

2

Kinect Based Obstacle Avoider Robot

Table Of Contented Abstract………….………………………………………………………..5 Introduction………………………………………………………...6 1.Components……………………………………………………....7 1.1Kinect Sensor………………………………………………7 1.2OpenCV………………………………………………………8 1.3Atmega128 & RS232 …………………………………..9 1.4Actuators & Motor Drivers ………………………...10 1.5Visual Studio ……………………………………………..11 1.6Programmers Notepad ………………………………12 2. Project Discussion ………………………………………….13 2.1Kinect Feed ……………………………………………….13 2.2Obstacle Detection …………………………………….14 2.3Calibration ………………………………………………..15 2.4Decision Making ………………………………………..15 2.5Serial Communication ……………………………….17 2.6Specification of Robot ………………………………..18

3

Kinect Based Obstacle Avoider Robot

3. Code ………………………………………………………………19 3.1OpenCV function Used………………………………..19 3.2CL NUI function Used…………………………………19 3.3Serial Port Function Used…………………………...19 3.4Code for Image Processing………………………….20 3.4.1Main Code…………………………………………….20 3.4.2SerialPort.CPP………………………………………27 3.5Micro-Controller Code………………………………29 3.6Code Discussion ……………………………………….31 4.Future Work…………………………………………………..32 5.Team Info……………………………………………………….33

4

Kinect Based Obstacle Avoider Robot

Abstract To prepare a mobile autonomous robot which can navigate independently taking depth feed from Kinect sensor, hence to prepare an obstacle avoider robot using Kinect sensor.

5

Kinect Based Obstacle Avoider Robot

Introduction Kinect based obstacle avoider robot uses the Kinect Xbox 360 sensor for getting the feed of the environment around the robot in 3D raw data. These raw data is transformed in RGB image with the help of CL NUI Kinect drivers. Robot gets the controlling signals after processing the RGB image using OpenCV libraries OpenCV is mostly used for image processing and object tracking. The IDEs used for writing programs are Visual Studio and Programmer’s Notepad. The Serial communication between compute and microcontroller was established using RS232 the micro-controller used is Atmega128

6

Kinect Based Obstacle Avoider Robot

1. Components 1.1 Kinect Sensor

Fig.1.1 The Kinect is constructed to stand in front or on top of a television and thus has a single foot for stable positioning. Inside the foot is a joint which makes it possible to tilt the head of the Kinect. The tilt function is motorized, and can adjust the vertical direction of the sensor head approximately ±27o. 7

Kinect Based Obstacle Avoider Robot The several printed circuit boards inside the cabinet are cooled with a small fan, and equipped with an accelerometer for detection of the sensor head alignment. Along the bottom side of the sensor head is an array of multiple microphones, and on front is the IR transmitter, multicolour LED, VGA camera and IR camera.

Fig.1.2 Figure shows the placing of RGB camera, 3D Depth sensor, motor and multi array mic.

1.2 OpenCV OpenCV (Open Computer Vision) is a library that implements many algorithms commonly used in the field of computer vision. Computer vision is the area of computer science that focuses on extracting structured information from images. Images as they are stored on computers are large, unstructured, two dimensional arrays of pixels. Computer vision techniques can 8

Kinect Based Obstacle Avoider Robot also be applied to videos, which are stored as sequences of images. OpenCV provides algorithms that can be used for tasks such as locating faces in an image, recognizing predefined objects and shapes and detecting movement in a video. OpenCV also provides the infrastructure necessary for working with images and videos.

1.3 Atmega128 & RS232 Atmega128 is micro-controller used for controlling the motion of robot. It is 8 bit programmable micro controller with 7 input output ports and 2 UART (Universal Asynchronous Transmission and Receiver) channels with clock frequency of 14.7456 MHz

Fig.1.3 Atmega128 Development board

9

Kinect Based Obstacle Avoider Robot

Fig.1.4 Atmega128 Pin Out RS232 is used for serial asynchronous communication between computer and micro-controller. It sends data bit wise and one bit at a time.

1.4 Actuators & Motor Drivers Actuator used for motion of robot is Jonson motor. It works on 12V DC supply with 15 rpm rotational speed at 12V supply. Four motors are used for differential controlling of robot 10

Kinect Based Obstacle Avoider Robot Motor Drivers are necessary for controlling of motors through micro-controller as micro-controller can’t provide required voltage to motors. We here used MC33886 motor driver for providing required voltage to motors.

Fig.1.5 Block Diagram of MC33886

1.5 Visual Studio Visual Studio is IDE used for programming purpose the Kinect depth data is captured, converted and processed in it and sends controlling signal to micro-controller through serial communication with RS232. All the programming is done here it the Microsoft product and can be downloaded from official Microsoft site.

11

Kinect Based Obstacle Avoider Robot

1.6 Programmers Notepad Programmers Notepad is used to write and program the Atmega128 using usbasp programmer. It includes all header files and makefile to generate “.hex” which can be burned in to microcontroller directly.

12

Kinect Based Obstacle Avoider Robot

2. Project Discussion 2.1 Kinect Feed The first task was to access Kinect and get its feed to Microsoft Visual Studio. This was achieved using CL NUI drivers and SDK’s for Kinect sensor. The CL NUI driver accesses the Kinect depth data and provides us the RGB image in OpenCV. Cl NUI uses GETNUICAMERADEPTHFRAME ( ) to capture data from Kinect sensor. The value of the particular pixel at a position in the RGB image is due the distance of the object from the camera. The depth data from Kinect was converted into RGB image using OpenCV function cvSetData( ). The cvSetData( ) converts the raw depth data into 3 channel 8bit RGB image.

13

Kinect Based Obstacle Avoider Robot

Fig.2.1 2.2 Obstacle Detection Kinect actually starts detection from 45cm from camera. So we cannot see anything in between 0-45cms. So if the obstacle comes very close to robot within 0 to 45cms we cannot see it. So decided to use this blind region for detecting obstacle can be detected very close to robot. Kinect was put on a height from ground such the nearest point on the ground was 45cm from camera. Anything which comes in between floor and camera is obstacle hence obstacle detection was possible. The obstacle in the image appears black.

Fig.2.2 The obstacle can be detected as black colour detection in the image which can be done using OpenCV function cvInRangeS().

14

Kinect Based Obstacle Avoider Robot cvInRangeS() is use for colour detection in image so detecting black colour obstacle detection was possible.

2.3 Calibration The next task was to region of interest of the image so that the bot could only see the region which exactly in front of it an of dimension slightly garter then the dimension of bot. It was solve by OpenCV function cvSetImageROI() setting the ROI(Region Of Interest) we need only to process that much part of the whole image. It decreased the processing par of image taking less time to execute. The ROI was called centre image in the program discussed later.

Fig.2.3 As seen in the image the processing part is only up to ROI the whole image need not to be processed reducing processing.

15

Kinect Based Obstacle Avoider Robot

2.4 Decision Making The final task for the project was the decision making to take turn left or right or to move forward. It’s simple when there is no object in front of bot it can move forward. But problem comes when obstacle is detected now it has to go right or left. So for this purpose we took the remaining part of the image and divide into two part Right & Left just for decision making purpose. So if there is obstacle in front of it check for right image or left image in whichever image there is no obstacle turn that side.

Fig.2.4 But if there is obstacle in both the image we have problem. So we calculated centroid of the obstacle and whichever is away turn that side. The last case comes while turning if centroid of centroid of right image goes away as it is turning to left at time centroid of right will 16

Kinect Based Obstacle Avoider Robot be away from left and it will turn right according to code. And it will go turning right left right for ever. This error was removed by remembering last turn taken if it is right and now it turns left without going forward and starts turning right again then the bot will not turn right and will continue left turn till it finds space to move forward.

2.5 Serial Communication Serial Communication between computer and micro-controller was established using RS232. The data was sent through this channel in char form ‘L’ for Left, ‘R’ for Right, ‘F’ for Forward and ‘T’ for stopping bot at the end. The serial communication was establishedusing “SerialPort.h” header file

17

Kinect Based Obstacle Avoider Robot .

Fig.2.5

2.6 Specification of Robot The bot dimensions are 21x25cm. The Kinect is placed at the top at height of 30cm from ground. Kinect is oriented towards the ground making an angle of 30 deg. The bot consist of four wheel powered by 12V DC Jonson motors. Kinect needs power supply which is provided by adaptor. The robot motor and micro controller are powered by 12V battery. 18

Kinect Based Obstacle Avoider Robot

3. Code 3.1 OpenCV Function Used cvShowImage() : Display’s the image on the screen. cvSetImageROI() : Sets image ROI. cvInRangeS() : Sets the threshold to detect colour. cvCreateTrackbar() : Creates the Trackbar for calibration purpose. cvNot() : Creates the negative of binary image. cvErode() : Filters the Image to minimise error. cvDilate() :Filters the Image to minimise error. cvWaitKey() : Waits for the specified time. cvDesrtoyWindow() : Destroys the window in which image is shown. cvSetdata() : Sets the raw depth data to RGB image.

3.2 CL NUI Function Used GetNUICameraDepthFrameRGB32() : Gets the raw from Kinect StartNUICamera() : Starts the Kinect depth camera.

3.3 Serial Port Function Used Serial.OpenSerial() : Opens the serial connection. Serial.Send() : Sends the data through serial port. 19

Kinect Based Obstacle Avoider Robot

3.4 Code for Image Processing 3.4.1 Main Code #include "stdafx.h" #include #include #include #include #include #include "SerialPort.h" #include typedef struct sense { IplImage* Image; int x; int y; }cen; using namespace std; void kinectinit(); void Kinectfeed(); void getfromfile(); void writeinfile(); void trackbar(); void area_check(cen* ); void set_dist(); int lh=0,hh=255,ls=0,hs=255,lv=0,hv=255,x=0,y=0,h=480,w=640,wt; char ch,ab; double area ; cen* Right = (cen *)malloc(sizeof(cen)); cen* Left = (cen *)malloc(sizeof(cen)); cen* Centre = (cen *)malloc(sizeof(cen)); CvMoments *moments = (CvMoments*)malloc(sizeof(CvMoments));

20

Kinect Based Obstacle Avoider Robot IplImage* grey = cvCreateImage(cvSize(640, 480), 8, 1); IplImage* depth32 = cvCreateImage(cvSize(640, 480), 8, 4);

DWORD* depth32_data; CLNUICamera cam = CreateNUICamera(GetNUIDeviceSerial(0)); CLNUIMotor motor = CreateNUIMotor(GetNUIDeviceSerial(0)); CSerialPort Serial;

int _tmain(int argc, _TCHAR* argv[]) { int q =14; try { kinectinit(); printf("Set distance\n\n\n"); set_dist(); printf("start the bot 'y/n' and enter \n"); scanf("%c",&ch); if(ch =='y') { printf("To stop the bot press 's'\n"); while(1) { Kinectfeed(); cvInRangeS(depth32,cvScalar(lh,ls,lv),cvScalar(hh,hs,hv),grey); cvNot(grey,grey); cvErode(grey,grey,NULL,4); cvDilate(grey,grey,NULL,3); /* cvSetImageROI(grey,cvRect(14,y,x-14,h)); cvShowImage("left",grey); cvSetImageROI(grey,cvRect(0,0,640,480)); cvSetImageROI(grey,cvRect(w+x,y,640-(x+w+7),h)); For error solving purpose cvShowImage("right",grey); cvSetImageROI(grey,cvRect(0,0,640,480)); */ //

21

cvSetImageROI(grey,cvRect(x,y,w,h)); cvShowImage("grey",grey); Centre->Image = grey; area_check(Centre);

//

Kinect Based Obstacle Avoider Robot if(72 < area) { cvSetImageROI(grey,cvRect(14,y,x-14,h)); Left->Image = grey; area_check(Left); cvSetImageROI(grey,cvRect(0,0,640,480)); cvSetImageROI(grey,cvRect(w+x,y,640-(w+x+7),h)); Right->Image = grey; area_check(Right); if(((x-Left->x-14) > Right->x) && Left->x != 0) { if(q == 1) { Serial.Send(&ab); } else { ab = 'R'; Serial.Send(&ab); if(q==7) { q=14; } q=q/2; } } else if(x-Left->x-14 < Right->x|| Left->x == 0) { if(q == 1) { Serial.Send(&ab); } else { ab = 'L'; Serial.Send(&ab); if(q==2) { q=14; } q=q/7; }

22

Kinect Based Obstacle Avoider Robot } else { if(Centre->x > w/2) { if(q == 1) { Serial.Send(&ab); printf("%c e %d\n",ab,q); } else { ab = 'L'; Serial.Send(&ab); if(q==2) { q=14; } q=q/7; } } else if(Centre->x < w/2) { if(q == 1) { Serial.Send(&ab); } else { ab = 'R'; Serial.Send(&ab); if(q==7) { q=14; } q=q/2; } } else { Serial.Send(&ab); } } } else

23

Kinect Based Obstacle Avoider Robot { ab = 'F'; Serial.Send(&ab); q=14; } if(ch=='s') { ch='T'; Serial.Send(&ch); break; } ch=cvWaitKey(34); cvSetImageROI(grey,cvRect(0,0,640,480)); } cvDestroyWindow("grey"); cvDestroyWindow("left"); cvDestroyWindow("right"); ch='t'; } if(ch=='n') { printf("\n................ Abort ................."); } else if(ch == 't'); { printf("\n................ Process Terminated ................."); }

/*

getchar(); getchar(); free(&depth32_data); free(&Left); free(&Right); free(&Centre); */ return 0;

} catch( cv::Exception& e ) { const char* err_msg = e.what(); getchar(); } }

24

Kinect Based Obstacle Avoider Robot void Kinectfeed() { depth32_data = (DWORD*) realloc(depth32_data,640*480*4); GetNUICameraDepthFrameRGB32(cam, depth32_data); cvSetData(depth32, depth32_data, depth32->widthStep); } void writeinfile() { std::fstream fout; fout.open("threshold.txt",ios::out,ios::binary); fout

Suggest Documents