Real-time 3D Tracking with Camera Phones D i l Wagner Daniel W
Real-time 3D Tracking with Camera Phones Slide 2
DANIEL WAGNER
[email protected]
Augmented Reality on Mobile Phones Low cost, widely spread platform
Billions of phones deployed People know how to use them Strong demand from commercial side Huge chance for AR!
Target practical applications Easy to use 15-30 15 30 H Hz overallll fframe rate t Robust tracking
Real-time 3D Tracking with Camera Phones Slide 3
DANIEL WAGNER
[email protected]
Mobile Phone Camera Tracking Tracking for Augmented Reality always means Pose Tracking (6DOF) Optical O ti l Fl Flow Very simple, but does not give a pose
Marker Tracking Works well, but limited in its applications Hardly a research topic anymore
Natural Feature Tracking Currently a hot topic!
Real-time 3D Tracking with Camera Phones Slide 4
DANIEL WAGNER
[email protected]
History of AR Tracking on Phones (1) 2003 ARToolKit on PDA Wagner et at. 2004 3D Marker on Phone Möhring et al. 2005 ARToolKit on Symbian Henrysson et al.
Real-time 3D Tracking with Camera Phones Slide 5
DANIEL WAGNER
[email protected]
History of AR Tracking on Phones (2) 2005 Visual Codes Rohs R h ett at. t
2006 Studierstube Tracker Wagner et al. 2007 ye WikEye Schöning et al.
Real-time 3D Tracking with Camera Phones Slide 6
DANIEL WAGNER
[email protected]
History of AR Tracking on Phones (3) 2008 Advanced Marker Tracking Wagner et al. 2008 Natural Feature Tracking Wagner et al. 2009 High speed Natural Feature Tracking
Real-time 3D Tracking with Camera Phones Slide 7
DANIEL WAGNER
[email protected]
CPU/Memory Limitations of Mobile Phones Small memory Even though phones today have 64-128MB RAM consider 2-5 Megabytes as maximum
Weak processing power 200-600 MHz, Single core Typically no FPU (floating point ~40x slower than integer) Slow memory access, access small caches
Æ Code optimized for phones runs 5 5-10x 10x slower on a high-end phone than on an average PC (2GHz, single core) going g to change g q quickly y due to battery yp power Æ Not g
Real-time 3D Tracking with Camera Phones Slide 8
DANIEL WAGNER
[email protected]
Tracking by Detection This is what most „trackers trackers“ do… do Targets are detected e er frame every Popular because tracking and detection are solved simultanously
Camera Image
Keypoint detection
Descriptor creation and matching
Outlier Removal
P Pose estimation ti ti and refinement
Pose
Real-time 3D Tracking with Camera Phones Slide 9
DANIEL WAGNER
[email protected]
Natural Feature Tracking by Detection SIFT
Ferns
St State t off th the artt ffor object bj t recognition Known to be slow (best implementation for phones is ~10-100x 10 100x too slow for real-time use) Typically yp y used off-line
St State t off the th artt for f fast f t pose tracking Known to be memory intensive (requires ~10x 10x too much memory for phones) Long g training gp phase
SIFT: [Lowe, 2004]
Ferns: [Ozuysal, 2007]
Real-time 3D Tracking with Camera Phones Slide 10
DANIEL WAGNER
[email protected]
Performance of SIFT and Ferns modified for mobile phone tracking
O Overall tra acking tim me
100ms 80ms FERNS Ad 60ms
FERNS Cars FERNS Vienna
40ms
SIFT Ad 20 20ms
SIFT Cars SIFT Vienna
0ms PC
iPAQ
N95 Float
N95 Fixed
Moto Q9
Real-time 3D Tracking with Camera Phones Slide 11
DANIEL WAGNER
[email protected]
NFT with SIFT on a Mobile Phone
Real-time 3D Tracking with Camera Phones Slide 12
DANIEL WAGNER
[email protected]
Doing it better: Dedicated Detection and Tracking
Real-time 3D Tracking with Camera Phones Slide 13
DANIEL WAGNER
[email protected]
Detection and Tracking
Real-time 3D Tracking with Camera Phones Slide 14
DANIEL WAGNER
[email protected]
High Speed Tracking on the Mobile Phone
Real-time 3D Tracking with Camera Phones Slide 15
DANIEL WAGNER
[email protected]
Workflow of our Tracker During D i startup: fi find d ffeatures iin a reference f iimage At runtime: ti Take previous pose and apply motion model Get estimate for what we are looking for
Create affine warped p p patches of reference features Closely resemble how the feature should look in the camera image
Project patches into camera image and use normalized cross correlation ((NCC)) to match
Real-time 3D Tracking with Camera Phones Slide 16
DANIEL WAGNER
[email protected]
PatchTracker in Action (1)
Real-time 3D Tracking with Camera Phones Slide 17
DANIEL WAGNER
[email protected]
PatchTracker in Action (2)
Real-time 3D Tracking with Camera Phones Slide 18
DANIEL WAGNER
[email protected]
How fast is it? PhonyFerns
45,0
Average ms p A per frame
40,0
38 3 38,3
PhonySIFT with PatchTracker 4,0
41,3
3,5
35,0 30,0 25 0 25,0 20,0 15,0 10,0
8,4
8,3
Average ms p A per frame
PhonySIFT
3,8 3,2
3,0 2,5 2,0 1,5
1,0
1,0
5,0
0,5
0,0
0,0 Performance on the phone
PhonyFerns with PatchTracker
Performance on the PC
1,0
Real-time 3D Tracking with Camera Phones Slide 19
DANIEL WAGNER
[email protected]
Orthogonal Strengths and Weaknesses SIFT/Ferns
PatchTracker
Recognize many targets
5
4
Detect target
5
4
Initialize tracking
5
4
Speed
4
5
Robust to blur
4
5
Robust to tilt
4
5
Robust to lighting changes
5
Real-time 3D Tracking with Camera Phones Slide 20
DANIEL WAGNER
[email protected]
M More Results… R lt (1)
Real-time 3D Tracking with Camera Phones Slide 21
DANIEL WAGNER
[email protected]
Creatted by the e group of Blair Ma acIntyre a and others s…
More Results… Results (2)
Real-time 3D Tracking with Camera Phones Slide 22
DANIEL WAGNER
[email protected]
Conclusions & Future Work NFT sometimes more robust than markers Bad lighting (blur) Occlusions
Many open issues
Non planar targets Non-planar Large targets (rooms, building, cities) A t Automatic ti ttargett acquisition i iti (SLAM) GPU implementations
Real-time 3D Tracking with Camera Phones Slide 23
DANIEL WAGNER
[email protected]
Thank you for listening listening…