Real-time 3D Tracking. Daniel Wagner

Real-time 3D Tracking with Camera Phones D i l Wagner Daniel W Real-time 3D Tracking with Camera Phones Slide 2 DANIEL WAGNER [email protected] ...
Author: Albert Baker
3 downloads 2 Views 2MB Size
Real-time 3D Tracking with Camera Phones D i l Wagner Daniel W

Real-time 3D Tracking with Camera Phones Slide 2

DANIEL WAGNER [email protected]

Augmented Reality on Mobile Phones ƒ Low cost, widely spread platform ƒ ƒ ƒ ƒ

Billions of phones deployed People know how to use them Strong demand from commercial side Huge chance for AR!

ƒ Target practical applications ƒ Easy to use ƒ 15-30 15 30 H Hz overallll fframe rate t ƒ Robust tracking

Real-time 3D Tracking with Camera Phones Slide 3

DANIEL WAGNER [email protected]

Mobile Phone Camera Tracking Tracking for Augmented Reality always means Pose Tracking (6DOF) ƒ Optical O ti l Fl Flow ƒ Very simple, but does not give a pose

ƒ Marker Tracking ƒ Works well, but limited in its applications ƒ Hardly a research topic anymore

ƒ Natural Feature Tracking ƒ Currently a hot topic!

Real-time 3D Tracking with Camera Phones Slide 4

DANIEL WAGNER [email protected]

History of AR Tracking on Phones (1) ƒ 2003 ƒ ARToolKit on PDA ƒ Wagner et at. ƒ 2004 ƒ 3D Marker on Phone ƒ Möhring et al. ƒ 2005 ƒ ARToolKit on Symbian ƒ Henrysson et al.

Real-time 3D Tracking with Camera Phones Slide 5

DANIEL WAGNER [email protected]

History of AR Tracking on Phones (2) ƒ 2005 ƒ Visual Codes ƒ Rohs R h ett at. t

ƒ 2006 ƒ Studierstube Tracker ƒ Wagner et al. ƒ 2007 ye ƒ WikEye ƒ Schöning et al.

Real-time 3D Tracking with Camera Phones Slide 6

DANIEL WAGNER [email protected]

History of AR Tracking on Phones (3) ƒ 2008 ƒ Advanced Marker Tracking ƒ Wagner et al. ƒ 2008 ƒ Natural Feature Tracking ƒ Wagner et al. ƒ 2009 ƒ High speed Natural Feature Tracking

Real-time 3D Tracking with Camera Phones Slide 7

DANIEL WAGNER [email protected]

CPU/Memory Limitations of Mobile Phones ƒ Small memory ƒ Even though phones today have 64-128MB RAM consider 2-5 Megabytes as maximum

ƒ Weak processing power ƒ 200-600 MHz, Single core ƒ Typically no FPU (floating point ~40x slower than integer) ƒ Slow memory access, access small caches

Æ Code optimized for phones runs 5 5-10x 10x slower on a high-end phone than on an average PC (2GHz, single core) going g to change g q quickly y due to battery yp power Æ Not g

Real-time 3D Tracking with Camera Phones Slide 8

DANIEL WAGNER [email protected]

Tracking by Detection ƒ This is what most „trackers trackers“ do… do ƒ Targets are detected e er frame every ƒ Popular because tracking and detection are solved simultanously

Camera Image

Keypoint detection

Descriptor creation and matching

Outlier Removal

P Pose estimation ti ti and refinement

Pose

Real-time 3D Tracking with Camera Phones Slide 9

DANIEL WAGNER [email protected]

Natural Feature Tracking by Detection SIFT

Ferns

ƒ St State t off th the artt ffor object bj t recognition ƒ Known to be slow (best implementation for phones is ~10-100x 10 100x too slow for real-time use) ƒ Typically yp y used off-line

ƒ St State t off the th artt for f fast f t pose tracking ƒ Known to be memory intensive (requires ~10x 10x too much memory for phones) ƒ Long g training gp phase

SIFT: [Lowe, 2004]

Ferns: [Ozuysal, 2007]

Real-time 3D Tracking with Camera Phones Slide 10

DANIEL WAGNER [email protected]

Performance of SIFT and Ferns modified for mobile phone tracking

O Overall tra acking  tim me

100ms 80ms FERNS Ad 60ms

FERNS Cars FERNS Vienna

40ms

SIFT Ad 20 20ms

SIFT Cars SIFT Vienna

0ms PC

iPAQ

N95  Float

N95  Fixed

Moto Q9

Real-time 3D Tracking with Camera Phones Slide 11

DANIEL WAGNER [email protected]

NFT with SIFT on a Mobile Phone

Real-time 3D Tracking with Camera Phones Slide 12

DANIEL WAGNER [email protected]

Doing it better: Dedicated Detection and Tracking

Real-time 3D Tracking with Camera Phones Slide 13

DANIEL WAGNER [email protected]

Detection and Tracking

Real-time 3D Tracking with Camera Phones Slide 14

DANIEL WAGNER [email protected]

High Speed Tracking on the Mobile Phone

Real-time 3D Tracking with Camera Phones Slide 15

DANIEL WAGNER [email protected]

Workflow of our Tracker ƒ During D i startup: fi find d ffeatures iin a reference f iimage ƒ At runtime: ti ƒ Take previous pose and apply motion model ƒ Get estimate for what we are looking for

ƒ Create affine warped p p patches of reference features ƒ Closely resemble how the feature should look in the camera image

ƒ Project patches into camera image and use normalized cross correlation ((NCC)) to match

Real-time 3D Tracking with Camera Phones Slide 16

DANIEL WAGNER [email protected]

PatchTracker in Action (1)

Real-time 3D Tracking with Camera Phones Slide 17

DANIEL WAGNER [email protected]

PatchTracker in Action (2)

Real-time 3D Tracking with Camera Phones Slide 18

DANIEL WAGNER [email protected]

How fast is it? PhonyFerns

45,0

Average ms p A per frame

40,0

38 3 38,3

PhonySIFT with PatchTracker 4,0

41,3

3,5

35,0 30,0 25 0 25,0 20,0 15,0 10,0

8,4

8,3

Average ms p A per frame

PhonySIFT

3,8 3,2

3,0 2,5 2,0 1,5

1,0

1,0

5,0

0,5

0,0

0,0 Performance on the phone

PhonyFerns with PatchTracker

Performance on the PC

1,0

Real-time 3D Tracking with Camera Phones Slide 19

DANIEL WAGNER [email protected]

Orthogonal Strengths and Weaknesses SIFT/Ferns

PatchTracker

Recognize many targets

5

4

Detect target

5

4

Initialize tracking

5

4

Speed

4

5

Robust to blur

4

5

Robust to tilt

4

5

Robust to lighting changes

…

5

Real-time 3D Tracking with Camera Phones Slide 20

DANIEL WAGNER [email protected]

M More Results… R lt (1)

Real-time 3D Tracking with Camera Phones Slide 21

DANIEL WAGNER [email protected]

Creatted by the e group of Blair Ma acIntyre a and others s…

More Results… Results (2)

Real-time 3D Tracking with Camera Phones Slide 22

DANIEL WAGNER [email protected]

Conclusions & Future Work ƒ NFT sometimes more robust than markers ƒ Bad lighting (blur) ƒ Occlusions

ƒ Many open issues ƒ ƒ ƒ ƒ

Non planar targets Non-planar Large targets (rooms, building, cities) A t Automatic ti ttargett acquisition i iti (SLAM) GPU implementations

Real-time 3D Tracking with Camera Phones Slide 23

DANIEL WAGNER [email protected]

Thank you for listening listening…

Suggest Documents