Welcome to Accurate Robotic 3D Vision
An educational webinar sponsored by Universal Robotics and Yaskawa Motoman Robotics
Speakers Hob Wubbena, Director of Marketing, Universal Robotics • • • •
Engineering, technical planning and product marketing for Hewlett-Packard and Agilent Technologies for 25 years Aerospace & defense, telecommunications, test & measurement, electronic manufacturing, and the chemical process industries 3 patents, numerous marketing awards, ~12 published articles B.S. Civil Engineering, University of Wisconsin; Masters Business, Denver University
Erik Nieves, Technology Director, Yaskawa Motoman Robotics •
• • •
Management and engineering, for Motoman Robotics for 20 years; currently focused on corporate strategic technology roadmap and emerging applications Standards Development & Education Committees for the Robotics Industries Association (RIA), “Ask the Experts” forum, RIA Many published robot technology articles including control and metrology B.S. Mathematical Physics, Southwestern Adventist University 7/21/2011
Copyright © 2011 Universal Robotics
Page 2
Experts Aditya Nawab, R&D Manager, Universal Robotics • • •
•
Extensive design experience in computer vision, hardware/software integration, autonomous vehicle development and industrial robotics Worked on numerous Department of Defense and NASA robotic projects His work at Universal focuses on dexterous manipulation, force control, sensory-motor coordination, forward and inverse kinematics, dynamics of articulated manipulators, and R&D project management B.S. Mechanical Engineering, Florida Atlantic University; M. S. Mechanical Engineering, University of Florida
Greg Garmann, Technology Leader, Yaskawa Motoman Robotics
• • •
Technology Leader at Yaskawa America, Motoman Robotics Division, and has been involved in automation for more than 25 years. Developed vision capabilities for robotics guidance using 2D and 3D technologies B.S. Computer Engineering, Wright State University
7/21/2011
Copyright © 2011 Universal Robotics
Page 3
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision • Types of 2D, 2½D & 3D robotic vision systems • Elements of Accuracy • Choosing a 3D vision system • Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 4
Robotic 3D Vision Introduction • Beyond scope of webinar: – –
– – – –
–
3D vision (movies) anaglyphs for depth perception Photometric stereo from telecentric camera Depth From Focus (DFF) 3D Mosaicking Vision analysis tools (blob analysis, face recognition) Time of Flight (TOF) Future webinar Laser line projector
• Robotic vision delivers real-time data about objects – –
Part Inspection Vision Guidance (X, Y, Z position & Rx, Ry, Rz pose)
• 3D accuracy requires both intrinsic camera calibration and hand-eye calibration
7/21/2011
Copyright © 2011 Universal Robotics
Page 5
Repeatability vs. Accuracy • Repeatability is important for automated tasks where robot picking and placing to same locations • Relative Accuracy is important for random tasks where the spatial location is constantly changing: – Random box depalletization – Random box moving – Random part picking – Random bin picking 7/21/2011
Copyright © 2011 Universal Robotics
Page 6
Intrinsic Camera Calibration Calibration process • • •
Camera sensor information Fiducial in full FOV 15 varying fiducial images
Y Z
Data in 3 coordinate systems •
Image, Camera, Object
X
Image rectification (ΔY) •
Removes perspective and/or lens distortions
If multiple cameras •
•
Computes disparity, distance, 3D coordinates Aligns image on common plane
7/21/2011
Camera
Object
Y Z
Y Z X
Image
Copyright © 2011 Universal Robotics
X Focal length
Page 7
FOV Field of View
3D Vision & Robotic 3D Vision 3D Vision: stereo vision with depth (z) perception resulting from the disparity (Δx) of different images of the same object 3D Accuracy: resolution in the Δz depth of an object viewed from stereoscopic vision. It is based on quality and geometry of cameras. Δy distance difference between the cameras (placement and orientation) is corrected during intrinsic camera calibration, resulting in a rectified image. The Δx disparity of each camera's image of same pixel point in space is computed through 3D software algorithms.
Robotic 3D Vision •
3D Vision with vision guidance for a robot delivers in real-time: – – – –
7/21/2011
Copyright © 2011 Universal Robotics
Position (X, Y, Z) Pose/orientation (Rx, Ry, Rz) Interactively, with tool offsets Choice of robot tool affects vision requirements (vacuum and grippers can work within 2-3 mm)
Page 8
Hand–Eye Calibration • First, calibrate cameras • Then Hand-Eye calibration enables data transform across coordinate systems –
Robot “hand” tool guided by camera “eye”
• RESULTS FOR: – – –
VISION GUIDANCE Part Inspection (optional) 3D Part Model creation (optional)
7/21/2011
Copyright © 2011 Universal Robotics
Page 9
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision • Types of 2D, 2½D & 3D robotic vision systems • Elements of Accuracy • Choosing a 3D vision system • Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 10
Types of 2D, 2½D & 3D Vision Systems
2½D
2D Rz
Rz
X
3D Z
Rz
#1
#2
Z
X
X Rx
•Single Camera
•Single camera with known calibration plate, OR multiple images to obtain depth, OR add TOF/laser
•Move camera OR add 2nd camera with shape-matching (model) OR TOF/laser with surface-matching (point cloud)
•Vision data in same plane
•Z data added
•Rx, Ry data added
•X, Y, Rz(angle) only; no Z
•Postion X, Y, Z, Rz (angle) only
•Position X, Y, Z, & Pose Rx, Ry, Rz
•No tilt (Rx, Ry)
•No tilt (Rx, Ry)
•Allows for part tilt
•Camera distance constant
•Camera distance can change
•Camera distance can change
7/21/2011
Copyright © 2011 Universal Robotics
Page 11
Types of 3D Vision
Camera System
Vision
Methodology Required
Comments
Single Stationary
2D, 2½D, 3D
2½: ONLY if height is known & fixed. Height can be variable with TOF 3D: Shape-match
Good if plane is fixed
Single Moving
3D
3D: with shape-match or surface-match
Can measure depth, Slower than Stereo
Binocular Stereo
3D
Shape-match (model) or surface-match (point cloud)
Surface-match requires TOF or laser
3D
Mosaicking Shape-match (model)
Surface-match requires TOF or laser, Slower than binocular
Multiple Stereo
7/21/2011
Copyright © 2011 Universal Robotics
Page 12
Single Stationary Camera • Using 3D Model • Shape-Based 3D Matching –
Speed up by using only a region of interest, eliminate unnecessary edges
• Surface-Based 3D matching
7/21/2011
Copyright © 2011 Universal Robotics
Page 13
Single Camera, Multiple Positions • Parts must be stationary for measurement (e.g. Stacked parts) • May require multiple images of same object from different camera locations • Good for vision guidance and part inspection
7/21/2011
Copyright © 2011 Universal Robotics
Page 14
Binocular Cameras • Only one image required • Faster than multiple image approach • Can find randomly located objects • Use of standard off-theshelf cameras • Can provide full field of view of robot operating envelope
7/21/2011
Copyright © 2011 Universal Robotics
Page 15
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision • Types of 2D, 2½D & 3D robotic vision systems • Elements of Accuracy • Choosing a 3D vision system • Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 16
3D Vision Accuracy and Robot Operating Envelope Geometry 3D Accuracy, Δz, is affected by:
Object
Z = object distance
Δz
Virtual Image 1
P1 (x1,y1,z1) P2 (x2,y2,z2)
Z X
Camera 1
Virtual Image 2
f=focal length
Camera 2
b=baseline distance
7/21/2011
•Distance, b, between pair of cameras •Distance, Z, from object to cameras •Disparity, Δd, offset in x between image points, P1 & P2 •Focal length, (pixels, not mm) 2
Δz = Z f·b
· Δd
where f (pixels) = f of Lens (mm) * Camera Horiz. Resolution (px) / CCD Horiz. sensor width (mm)
Copyright © 2011 Universal Robotics
Page 17
3D Camera Positions for Objects on Flat Surface 2nd pair orthogonal ~100mm apart @ 45˚
1st pair cameras just above robot at ~100mm apart, 45˚ below horizontal
NOTE: Irregular objects require about 300px by 300px for determining position & pose
Robot
±150mm depending on part & envelope dimensions
Robot Work Envelope
Robot Envelope Height
Robot Envelope Depth
Parts / Boxes on Flat Surface
7/21/2011
Copyright © 2011 Universal Robotics
Page 18
NOTE: Rectangular objects require about 200px by 200px for determining position & pose
Camera Positions for Objects in Bin 1st pair cameras just above robot at ~100mm apart, 60˚ below horizontal
2nd pair orthogonal ~100mm apart @ 60˚
NOTE: Irregular objects require about 300px by 300px for determining position & pose
Robot Operating Envelope
Robot operating envelope Height
±150mm depending on part & envelope dimensions
Robot
Envelope Depth
Parts or Boxes in Bin 7/21/2011
Copyright © 2011 Universal Robotics
Page 19
NOTE: Rectangular objects require about 200px by 200px for determining position & pose
Robot Accuracy & Repeatability • • •
Accuracy: how close a robot can reach a commanded 3D position Accuracy varies with robot speed, robot reach, and with payload Accuracy of a robot is determined by three elements of the system: – Resolution of the control system – Error operating the robot arm under closed loop servo operation – Imprecision of mechanical linkages, gears & deflections under load
• •
Repeatability: the ability to duplicate an action or a result every time See ISO 9283 7/21/2011
Copyright © 2011 Universal Robotics
Page 20
Typical Robot Repeatability •
Plotted over 100 robots from 3 companies – –
•
Repeatability loosely a function of: – – –
•
2Kg to 1,200Kg SCARA, material handling, welding, assembly…
Payload (Kg) Reach (M) Speed (˚/sec)
Best repeatability approaches limit line
7/21/2011
Copyright © 2011 Universal Robotics
Page 21
Robot Accuracy • Accuracy is typically worse than repeatability, not constant over workspace • Full robot calibration: –
Accuracy = 2X – 4X of Repeatability; can approach 1X
• Typical robot calibration: –
Accuracy = 3X – 5X of Repeatability
Best (2X) Repeatability Accuracy 0.015mm 0.03mm 0.06mm 0.1mm 0.2mm
0.03mm 0.06mm 0.12mm 0.2mm 0.4mm 7/21/2011
Typ. (4X) Accuracy 0.06mm 0.12mm 0.24mm 0.4mm 0.8mm
Payload, Reach, Speed 2kg 5kg 35kg 50kg 275kg
Copyright © 2011 Universal Robotics
Page 22
0.6M 0.8M 1.3M 1.6M 2.5M
375 deg/s 270 deg/s 170 deg/s 170 deg/s 90 deg/s
Overall 3D Robotic Vision System Accuracy System Accuracy results (go to Universal Robotics 3D Made Easy Calculator at www.universalrobotics.com/calc
7/21/2011
Copyright © 2011 Universal Robotics
Page 23
Camera Selection Scalable vision systems enable a wide variety of cameras to fit the robot operating envelope geometry and accuracy needs USB Webcams & GigE Camera Selection Medium Size Parts & Boxes (workspace < 1.0M wide x 1.0M deep x 1.0M high)
Boxes, Parts, & Pallets (workspace < 1.5M x 1.5M x 1.5M)
Working Distance from Cameras to Objects (Camera selection provides
