Stereo Vision for High Resolution 3D-Imaging
Ensenso N10 und N20 Stereo 3D Cameras 1
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
• • • • • •
2
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Overview of 3D sensing The stereo vision principle Comparison to other methods Our new FlexView capture technique Calibration between cameras and robots Applications and examples
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
3D Acquisition Techniques - Overview
Triangulation of Measurement of laser line with light propagation a camera time in each pixel using modulated light
Triangulation by image comparison
Triangulation via Triangulation stripe and sine via known wave projection projected pattern
Microsoft Kinect 3
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Triangulation by image comparison + projection of unknown static pattern
Ensenso
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
3D Acquisition Techniques - Properties
Laser Triangulation Moving Objects? Multi-view capable? Object requirements? 4
!
Time-ofFlight
Stereo Vision Fringe (passive) Projection
Kinect
Stereo Vision (active)
needs controlled movement
only textured objects Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Multi-view stitching / calibration
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Stereo Vision – Rediscovered • Stereo Vision has been around in research since decades! • Almost no industrial applications • New algorithms and increased computing power boost usability • Collision avoidance in cars – Daimler E & S-Class
• New sensor technology for industrial applications – Automation, Robotics 5
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision – Without Projection • Image comparision (“Stereo matching”) computes disparity for each pixel
?? incomplete depth image
• Problem: Disparity cannot be uniquely determined on untextured surfaces
6
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision – With Projection • Solution: add artificial texture via a pattern projector
Shadow
• Result: (almost) complete depth information
7
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
complete depth image
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
45mm
Stereo Vision – Ensenso N10 Camera Cameras with 3.6 - 16mm lenses
USB2.0 (lockable) : Image transfer to PC
LED as diffuse light source for calibration
Static pattern projector 8
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
12-24V GPIO: Trigger input or output
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision – Ensenso N20 Camera Gigabit Ethernet, (lockable): image transfer to PC
LED + Pattern projector Cameras with 6 - 16mm focal length
9
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
12-24V GPIO: Trigger input or output
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision – Ensenso N20 Camera • Gigabit Ethernet – Up to 100m cable connections – Power supply over PoE
• 1.3MP resolution – more/smaller parts are detectable in the image frame
N10 WVGA
N20 10
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision – Pros and Cons • Pros • • • •
Fast one-shot measurement with only one image pair Easy to set up Works for moving and still standing objects Reasonably priced due to standard hardware components • Flexible in working volume • Simple, synchronous multi-view acquisition
• Cons • Less detailed 3D data • Computationally demanding 11
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision - Raw Data • Stereo Vision: two images ≈ 0.7MB • Fringe projection: Multiple (20+) images with different sine waves / Gray codes ≈ 7MB • Laser triangulation: one image per 3D profile/line (z.B. 480 images!) ≈ 23MB 12
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Stereo Vision - Framerate • 30Hz live 3D data • Up to 80Hz with offline image processing, binning, AOI etc.
13
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Stereo Vision – With HDR • Stereo Vision only relies on local contrasts; brightness characteristics of sensor pixels may be unknown • Can use HDR functionality of image sensors • HDR expands possibilities to triangulate uncooperative surfaces 14
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
FlexView Technique • Stereo Vision: Two images ≈0.7MB • with different projection patterns: ≈ 𝑛 x 0.7MB • Fringe Projection: Multiple (20+) images with different patterns ≈ 7MB • Laser triangulation: One image per 3D profile/line (e.g. 480 images!) ≈ 23MB 15
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
FlexView Technique • Shiftable Projection Pattern – piezo mechanical – integrated into N20 cameras
• Computation of 3D point cloud from multiple stereo image pairs – Much higher Z accuracy in static scenes – opens up new possibilities in bin picking and object identification
16
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
FlexView Technique • finer object details • more robust 3D data on slanted surfaces • more precise object contours • significant noise reduction
one-shot Aufnahme 4 image pairs
8x gobo shift 17
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Software – Multi-View • Set up arbitrary combinations of stereo- and monocular cameras • Calibration of any setup within a few minutes
One stereo camera
18
Multiple stereo cameras to reduce shadow regions or enlarge field of view
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
Combination of monocular and stereo cameras for synchronous multi-view RGBD imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Software – Multi-View • Problem of triangulation techniques: shadows • Merging multiple perspectives – Reduces shadows – Enlarges field of view
Camera 1 19
Camera 2
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
Camera 1 + 2
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Software – Calibration • Requirements for processing of 3D data
?
– Conversion between camera and workspace coordinates – Conversion between camera and robot coordinates ?
20
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Software – Hand-Eye Calibration • Computation of camera pose – from measured calibration plate pose – and known robot hand pose
• Problem: Pose of calibration plate inside robot hand must be known – measurement is time consuming / expensive 21
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
? ?
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Software – Hand-Eye Calibration • Solution: Use multiple observations to estimate the missing pose • Different Configurations – Camera on robot hand – Camera fixed in workspace
• Calibration can be automated easily – No manual measuring – Quick setup – Short down times 22
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Software – Hand-Eye Calibration
23
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications • Handling – Material feeding, assembly
• Volume measurement – Luggage, Parcels, Dimensional Weight
• Quality Control – Checking assemblies, blisters
• Security – Access control – Tracking people movement
• Medical – Shape of body parts, custom splints – Head movement during tomography 24
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications – Counting Parts • Counting of chocolates • Independent of chocolate color
25
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Applications – Part Defects • Some part defects are not simple to detect in 2D, but become obvious in the height image
26
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Applications – Pick&Place Demo
27
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications – Palletizing • Loading and unloading of euro palettes, 1200x800mm
28
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Applications – Food Salad
• Identification of vegetables • Color camera adds texture cues for classification • Depth data allows size binning
Onions
Oranges
Tomatoes
29
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Applications – Product Identification • Identification of Dish • Combination of 3D and Synchronous Texture Data • Monitoring of conveyor belt in 3D at 20Hz, without external trigger
30
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications
Application – Tree Trunk Measurement • Outdoor application – Wood structure allows triangulation despite sunlight – Passive stereo vision
• Trunk value estimation • Cut planning for sawmill
www.je-gmbh.de JÖRG Elektronik GmbH 31
Speaker: Rainer Voigt, Ensenso GmbH | Talk: Industrial 3D Imaging with Stereo Vision
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications – Bin Picking • One-shot image acquisition allows faster pick rates • Multiple cameras – increase spatial resolution for smaller parts – allow robust part localization by using multiple perspectives 32
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
Applications – Bin Picking
33
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Applications
3D Techniques
Stereo Vision
Pros/Cons
FlexView
Calibration
• Questions
• Further information: www.ids-imaging.de
ensenso… 34
www.ensenso.de
Rainer Voigt, Ensenso GmbH | Stereo Vision for High Resolution 3D-Imaging
Applications
