University of Texas at Arlington

CSE 4392/5369

Robotic Vision: Sensing, Localization and Control Dr. Gian Luca Mariottini, Ph.D. Department of Computer Science and Engineering University of Texas at Arlington WEB : http://ranger.uta.edu/~gianluca

“ The most beautiful thing that we can experience is the Mystery. It is the true source of all art and all science.” - Albert Einstein

Gian Luca Mariottini, Univ. Texas at Arlington

Robotic Vision: Sensing...

No. of objects; Area; Center; Object Orientation;

Gian Luca Mariottini, Univ. Texas at Arlington

... Localization ...

Camera orientation w.r.t. Table; Distance to each object;

No. of objects; Area; Center; Object Orientation;

Gian Luca Mariottini, Univ. Texas at Arlington

and Control

FANUC robot vision system

Gian Luca Mariottini, Univ. Texas at Arlington

Applications Stair-Climbing & Descending for Rescue Operations

Gian Luca Mariottini, Univ. Texas at Arlington

Applications

Personal and Health-care Robotics

Courtesy from Charlie Kemp, director of the Healthcare Robotics Center at Georgia Tech;

Omnivision sensing

3-D object locations

Robot control

Gian Luca Mariottini, Univ. Texas at Arlington

Applications Multi-robot Navigation

Team of Robots Formation Control

Gian Luca Mariottini, Univ. Texas at Arlington

Applications Laparoscopic Surgery Courtesy from Univ. Strasbourg

Gian Luca Mariottini, Univ. Texas at Arlington

Applications Autonomous Navigation

Blue Team (U.Berkley, TA&M)

Carnegie Mellon

Gian Luca Mariottini, Univ. Texas at Arlington

Applications Sensing, Cognition and Humanoid-Robot Control

NAO (Aldebaran Robotics)

http://www.youtube.com/watch?v=HS Gian Luca Mariottini, Univ. Texas at Arlington

Space Exploration

Applications

Gian Luca Mariottini, Univ. Texas at Arlington

Applications Health-care RObotics

MOVAID Project to aid disabled people (Scuola Superiore St.Anna, Pisa, ITALY) PAM-AID (Trinity College, Ireland) Gian Luca Mariottini, Univ. Texas at Arlington

Visual Servoing

Applications

Courtesy from SIRS Lab, University of Siena

Gian Luca Mariottini, Univ. Texas at Arlington

Applications

Autonomous Aerial-Vehicle Sensing and Navigation

http://www.youtube.com/watch?v=WoMrPKCpWXw Gian Luca Mariottini, Univ. Texas at Arlington

Applications

3-D Dense Reconstruction with Commodity Cameras

Video Kinect Fusion Gian Luca Mariottini, Univ. Texas at Arlington

Applications System for Wearable and Audio Navigation @ GaTech Goal:

• Determine user’s location • Identify surroundings (parks, bench, trees, etc.) • Represent objects with unique sounds • Audio waypoints (e.g. metasounds) to destination path

3D (top)

...

•N-views Structure-from-Motion

•MAP localization (RANSAC/LM) Gian Luca Mariottini, Univ. Texas at Arlington

Am I a robot ? (Yes/No)

A

B

D

C

E

F Gian Luca Mariottini, Univ. Texas at Arlington

Course Topics Robotics and Vision-sensing: Basics - Introduction to Robotics; - The human eye and image formation; - Camera models; – Pinhole, Stereo and Omnidirectional - Feature detection – Edge, Corner, Color, SIFT - Feature tracking

Vision-based Localization - Reference frames and rigid body transformations - Full pinhole and stereo camera models - Camera Calibration - Camera Pose Estimation - Two-view geometry

Vision-based Control - Position- and Image-based Visual Servoing - Epipole-based Visual Servoing

Probabilistic Robotics - Basics of probability reasoning in robotics - Bayesian filtering for localization Gian Luca Mariottini, Univ. Texas at Arlington

Class Schedule and Office Hours Class Schedule: Mon. & Wed., 5:30-6:50pm in COBA 141 Office Hours:

Mon. & Wed., 1:30-3:30pm

I am generally available before or after class and by appointment, as well as at the office hours scheduled above. Since this is a special-topic course in robotics, and since each presented topic contains many exciting sub-fields, the interested students who want to know more about a specific problem are encouraged to schedule an appointment with Dr. Mariottini for additional material

Gian Luca Mariottini, Univ. Texas at Arlington

Course Prerequisites CSE 4392/5369 is self-contained and does not need special prerequisites. MATLAB programming experience is required. If you are unfamiliar, please check this: http://www.math.ucsd.edu/~bdriver/21d-s99/matlab-primer.html. During the course we will also make use of the MATLAB's Epipolar Geometry Toolbox, which can be efficiently used for creating vision-based robot simulations.

Gian Luca Mariottini, Univ. Texas at Arlington

Course Text There is no required textbook for this course. Suggested references: B. Siciliano, L. Sciavicco, L. Villani, G. Oriolo "Robotics: Modelling, Planning and Control", 3rd Edition, Springer, 2009 

E. Trucco, A. Verri "Introductory Techniques for 3-D Computer Vision", Prentice Hall, 1998 

R. Hartley, A. Zisserman, "Multiple View Geometry", 2nd Edition, Cambridge Univ. Press 2004 

Y. Ma, S. Soatto, J. Kosecka, S. Sastry "An Invitation to 3-D Vision. From Images to Geometric Models", Springer 2000 



Roberto Cipolla, “Robot and Computer Vision Course”, Cambridge University, UK

Gian Luca Mariottini, Univ. Texas at Arlington

Homeworks and Final Project (1) Students will be graded based on homework assignments and a major course project. Homeworks: theoretical questions, programming assignments (in MATLAB), etc. related to the topics of the course. Dates will be announced in class. Tentatively, course grades will based on the following: Assignments % of final grade Homeworks (total) 60 % Course Project 30 % - Proj. Presentation 10 %

Percent Grade 90 %-100 % 80 %-89.9 % 70 %-79.9 % 60 %-69.9 % Yes

E=Yes

F=No

Gian Luca Mariottini, Univ. Texas at Arlington