Link¨oping Studies in Science and Technology Dissertation No. 1449

Motion and Structure Estimation From Video Johan Hedborg

Department of Electrical Engineering Link¨ opings universitet, SE-581 83 Link¨oping, Sweden Link¨oping May 2012

Motion and Structure Estimation From Video c 2012 Johan Hedborg

Department of Electrical Engineering Link¨ oping University SE-581 83 Link¨ oping Sweden

ISBN: 978-91-7519-892-7

ISSN 0345-7524

Link¨ oping Studies in Science and Technology Dissertation No. 1449

iii

Dedicated to Ida and Edvin, my family

iv

v

Abstract Digital camera equipped cell phones were introduced in Japan in 2001, they quickly became popular and by 2003 outsold the entire stand-alone digital camera market. In 2010 sales passed one billion units and the market is still growing. Another trend is the rising popularity of smartphones which has led to a rapid development of the processing power on a phone, and many units sold today bear close resemblance to a personal computer. The combination of a powerful processor and a camera which is easily carried in your pocket, opens up a large field of interesting computer vision applications. The core contribution of this thesis is the development of methods that allow an imaging device such as the cell phone camera to estimates its own motion and to capture the observed scene structure. One of the main focuses of this thesis is real-time performance, where a real-time constraint does not only result in shorter processing times, but also allows for user interaction. In computer vision, structure from motion refers to the process of estimating camera motion and 3D structure by exploring the motion in the image plane caused by the moving camera. This thesis presents several methods for estimating camera motion. Given the assumption that a set of images has known camera poses associated to them, we train a system to solve the camera pose very fast for a new image. For the cases where no a priory information is available a fast minimal case solver is developed. The solver uses five points in two camera views to estimate the cameras relative position and orientation. This type of minimal case solver is usually used within a RANSAC framework. In order to increase accuracy and performance a refinement to the random sampling strategy of RANSAC is proposed. It is shown that the new scheme doubles the performance for the five point solver used on video data. For larger systems of cameras a new Bundle Adjustment method is developed which are able to handle video from cell phones. Demands for reduction in size, power consumption and price has led to a redesign of the image sensor. As a consequence the sensors have changed from a global shutter to a rolling shutter, where a rolling shutter image is acquired row by row. Classical structure from motion methods are modeled on the assumption of a global shutter and a rolling shutter can severely degrade their performance. One of the main contributions of this thesis is a new Bundle Adjustment method for cameras with a rolling shutter. The method accurately models the camera motion during image exposure with an interpolation scheme for both position and orientation. The developed methods are not restricted to cellphones only, but is rather applicable to any type of mobile platform that is equipped with cameras, such as a autonomous car or a robot. The domestic robot comes in many flavors, everything from vacuum cleaners to service and pet robots. A robot equipped with a camera that is capable of estimating its own motion while sensing its environment, like the human eye, can provide an effective means of navigation for the robot. Many of the presented methods are well suited of robots, where low latency and real-time constraints are crucial in order to allow them to interact with their environment.

vi

vii

Popul¨ arvetenskaplig sammanfattning Mobiltelefoner utrustade med kameror introducerades i Japan 2001, och blev snabbt popul¨ ara. Redan 2003 s˚ aldes fler kamerautrustade mobiltelefoner ¨an digitalkameror. 2010 hade det s˚ alts 1 miljard kameratelefoner och f¨ors¨aljningen har inte minskat sedan dess. En annan trend inom mobiltelefoni ¨ar “smarta” mobiltelefoner som n¨ armast kan likst¨allas med mindre datorer i b˚ ade funktionalitet och ber¨ akningskraft. Kombinationen av en kraftfull processor och en kamera som l¨att kan b¨ aras i fickan ¨ oppnar upp f¨or en m¨angd intressanta datorseende applikationer. Huvudbidraget i denna avhandling ¨ar metoder som m¨ojligg¨or att en kamerautrustad enhet, s˚ asom en mobiltelefon, kan ber¨akna sin egenr¨orelse och genom detta ˚ aterskapa tre-dimensionell strukturer av vad kameran ser. Med hj¨alp av dessa tekniker skulle man kunna komplettera sina semesterbilder med 3D modeller av t.ex. statyer och byggnader. D˚ a man kan ˚ aterskapa tre-dimensionella information finns det ocks˚ a m¨ojlighet att skapa stereobilder fr˚ an sin mobiltelefonkamera, som sedan kan visas p˚ a t.ex. en 3D TV. Ett tredje exempel ¨ar s˚ a kallad f¨ orst¨ arkt verklighet d¨ ar virtuella objekt kan placeras i kamerabilden. Med denna teknik skulle man kunna g¨ora datorspel som man spelar i “verkligheten” eller ha en navi-gator som placerar ut skyltar och pilar p˚ a v¨agen eller p˚ a en fasad i den verkliga bilden. Inom datorseende ¨ ar struktur fr˚ an r¨ orelse ett aktivt forskningsomr˚ ade d¨ar m˚ alet ¨ ar att utveckla metoder som ber¨aknar 3D struktur genom att observera den r¨orelse som uppkommer i bildplanet n¨ar en kamera i r¨orelse registrerar en statisk scen. I denna avhandling presenteras en metod som fr˚ an fem korresponderande punkter i tv˚ a bilder skattar den relativa positionen och orienteringen mellan dessa tv˚ a vyer. Denna typ av metod anv¨ands vanligtvis inom ett RANSAC -ramverk f¨or att g¨ ora en robust skattning. H¨ar har vi utvecklat en strategi som kan f¨ordubbla prestandan hos ett s˚ adant ramverk. F¨or att behandla sekvenser med fler ¨an tv˚ a bilder har vi utvecklat en ny bundle adjustment metod, speciellt l¨ampad f¨or nyare bildsensorer. Krav p˚ a l¨ agre str¨ omf¨ orbrukning, minskad storlek, och ett l¨agre pris har lett till en designf¨ or¨ andring hos bildsensorerna f¨or n¨astan alla nya typer av kameror. Denna design¨ andring har medf¨ort att bildexponeringen ¨andrats fr˚ an en global slutare till en rullande slutare, d¨ar den rullande slutaren exponerar bilden rad f¨or rad. Klassiska struktur fr˚ an r¨ orelse metoder ¨ar baserade p˚ a ett antagande om en global slutare och om de anv¨ands p˚ a en rullande slutare kan resultatet kraftigt f¨ ors¨ amras. Ett viktigt bidrag i denna avhandling ¨ar en bundle adjustmentmetod f¨ or kameror med rullande slutare. Metoden modellerar noggrant r¨orelsen hos kameran f¨ or varje bildrad med b˚ ade position och orientering. De utvecklade metoderna a¨r inte enbart till¨ampbar p˚ a mobiltelefoner, utan f¨or alla typer av mobila plattformar som ¨ar utrustade med kameror, t.ex. en robot eller en autonom bil. En kamerautrustad robot eller bil kan navigera och interagera i sin omgivning, och i likhet med en m¨anniska kan en robot f˚ a en b¨attre uppskattning av sin omgivning genom att r¨ora sig och ¨andra sin vy.

viii

ix

Acknowledgments I would like to thank all current and former members of the Computer Vision Laboratory in Link¨ oping, contributing to a very friendly and inspiring working environment, and a special thanks goes to: • Michael Felsberg for sharing his wisdom, expertise, giving the very best possible support, and finally for showing tolerance and keeping confidence in me. • Per-Erik Forss´ en for invaluable discussions and insights, whiteout it this work would never have been the same. • G¨ osta Granlund, Eirk Jonsson, Fredrik Larsson, Klas Nordberg, Bj¨ orn Johansson, Erik Ringaby for many and very interesting discussions. • Johan Wiklund for mediating between me and my hardware/software. • Per-Erik Forss´ en, Klas Nordberg and Ida Friedleiff for proofreading this thesis. I would also like to thank my family and friends, most notably: • My mother and Peter for all love, support and encouragement throughout my life, whit out this support this work would not have been possible • Last but certainly not least my family, Ida for love and massive support and great patience during this period and Edvin for always greeting me with a smile after a hard day at work.

Acknowledged goes out to the funders of this research: European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n◦ 215078 DIPLECS, the European Community’s Sixth Framework Programme (FP6/20032007) under grant agreement n◦ 004176 COSPAL, and ELLIIT, the Strategic Area for ICT research, funded by the Swedish Government

Johan Hedborg May 2012

x

Contents 1 Introduction 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Included Publications . . . . . . . . . . . . . . . . . . . . .

1 1 2 2

I

9

Background Theory

2 Least Squares Problems 2.1 Problem Formulation . . . . . . . . . . . . 2.2 Linear Least Squares . . . . . . . . . . . . 2.3 Non-Linear Least Squares . . . . . . . . . 2.3.1 The Gauss-Newton Method . . . . 2.3.2 The Levenberg-Marquardt Method

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

11 11 12 12 13 14

3 Image Correspondences 3.1 Blob Detection . . . . 3.2 Corner Detection . . . 3.3 Similarity Metrics . . 3.4 Correspondence Search

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

17 17 19 19 21

4 Pose Estimation 4.1 Overview . . . . . . . . . . . . . . . . . 4.2 The Pinhole Camera and its Calibration 4.3 Five Point Solution . . . . . . . . . . . . 4.4 Perspective-N-Point Pose Estimation . . 4.5 Costs . . . . . . . . . . . . . . . . . . . . 4.6 Pose Search . . . . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

23 23 24 25 28 28 29

5 Bundle Adjustment 5.1 Overview . . . . . . . . . . . . . . . . . . 5.2 Rolling Shutter . . . . . . . . . . . . . . . 5.3 Bundle Adjustment Problem Formulation 5.4 The Jacobian Structure . . . . . . . . . . 5.5 Rolling Shutter Camera . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

31 31 32 33 33 34

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

xi

. . . .

. . . .

. . . .

xii

CONTENTS 5.6 5.7

Rolling Shutter Rotation Rectification . . . . . . . . . . . . . . . . Rolling Shutter Bundle Adjustment . . . . . . . . . . . . . . . . . .

36 36

6 Concluding Remarks 6.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

39 39 39

II

43

Publications

A Real-time view-based pose recognition and interpolation for tracking initialization 45 B KLT Tracking Implementation on the GPU

69

C Fast and Accurate Structure and Motion Estimation

79

D Fast Iterative Five point Relative Pose Estimation

95

E Structure and Motion Estimation from Rolling Shutter Video

113

F Rolling Shutter Bundle Adjustment

131

Chapter 1

Introduction 1.1

Motivation

Digital camera equipped cell phones were introduced in Japan in 2001, they quickly became popular and by 2003 outsold the entire stand-alone digital camera market. In 2010 sales passed one billion units and the market is still growing. This development has led to a very capable imagine device that is small, power efficient and has a low cost. for example, the iPhone 4 camera has a 5Mpix 1/3.2”, backilluminated CMOS sensor with autofocus, is no bigger than half of a cubic cm, and at a price of only 9 Euro. The core contribution of this thesis is the development of methods that allow an imaging device such as the cell phone camera to estimate its own motion and to capture the observed scene structure. A portable device such as the cell phone could be used not only for taking pictures, but also to capture the geometry. Besides a collection of holiday photos, whole scenes or statues could be captured and could be viewed as realistic textured 3D models. Another aspect of being able to capture geometry is that, if the geometry is known there is a possibility to change from where an object is viewed. This gives a cell phone camera the ability to create 3D stereoscopic for a 3D capable TV. Augmented reality has recently captured the interest of many, here a cell phone can be used to augment virtual objects to the real-world environment when looking through the cell phone screen. There are quite a few applications that show this ability, most of them however use a predefined pattern, usually a paper with a special pattern printed on it. In this thesis we investigate markerless methods, which allows us to use the observed scene as “markers” and no special pattern is needed. A mobile platform such as an autonomous car or a robot, equipped with cameras is capable of estimating its own motion while sensing its environment. However using cameras for navigation is a challenging task, as was shown in one of the largest competitions for unmanned cars in urban environments, the Darpa Urban Challenge 2007. Here all vision sensors where turned off during the final. The vision sensor gave too much noise and were simply too unreliable, however when 1

2

CHAPTER 1. INTRODUCTION

a human drives a car more or less all data comes via their vision. This thesis have a focus towards real-time applications, which is well suited for navigation where low latency and real-time constraints are crucial in order to allow fast response. Pose estimation methods presented in this thesis have high accuracy, this can be crucial for mobile platforms e.g. when moving forward. A challenge that has recently appeared is the introduction of a rolling shutter on cameras. A rolling shutter camera captures the image row by row, and is especially challenging for computer vision methods that handles moving cameras and are based on the assumption of a static scene. Small cheap cameras has had a rolling shutter for a longer time now, but they are also starting to appear in more expensive cameras, even the ones used by professionals (e.g. RED or Canon C300). In this thesis we aim to solve some of the problems associated with a rolling shutter and structure from motion.

1.2

Outline

This thesis is divided into two parts, one introductory part and one publication part consisting of six previous published papers. The introductory part serves as a brief overview of the area of structure from motion which is a research field within computer vision. Underlying theory and concepts are presented, which hopefully will help the reader to better understand the papers in the second part. Additionally, in the introduction the author highlights common parts of the papers and positions them in a larger context. The papers can be divided into two sub categories, where the first four papers are concerned with pose estimation with a focus on real-time applications. The last two papers contribute with methods used for solving the structure from motion problem on a CMOS rolling shutter sensor, a sensor type which is used in nearly all new consumer imaging devices.

1.2.1

Included Publications

Here is a brief description of the six included papers, followed by an author list and abstract for each of the papers. Paper A uses a set of a priori known camera poses to train a system to fast retrieve a camera pose for a new image. The method is based on P-Channels which is a kind of feature vector and is compared with SIFT. Paper B is concerned with the task of finding points in a sequence of images that corresponds to one and the same 3D point in the world. The method is known as the KLT-tracker [23] and the papers core contribution is way of accelerating the search using the GPU. Paper C presents an extension to the RANSAC framework, which lowers the number of iteration by half, and hence doubles performance. The contribution is a strategy for how the minimal set is chosen. Paper D contains work on a new five point relative pose estimation method, the method is based on the Powell’s dog leg method which is a nonlinear least squares solver. Two possible ways of initialize the solver is also presented.

1.2. OUTLINE

3

Paper E focuses on the problem of estimating structure from motion from a camera with a rolling shutter. The paper assumes small translation between frames, and uses a rotation only rectification scheme to rectify points in the image. The points are then use with classical structure from motion methods to reconstruct camera trajectory and structure. Paper F extends the work of doing structure from motion on rolling shutter images, however in this paper a tailor made bundle adjustment method is presented which fully models the rolling shutter of the camera.

Paper A: Real-Time View-Based Pose Recognition and Interpolation for Tracking Initialization M. Felsberg and J. Hedborg. Real-time view-based pose recognition and interpolation for tracking initialization. Journal of real-time image processing, 2(2-3):103–115, 2007.

Abstract: In this paper we propose a new approach to real-time view-based pose recognition and interpolation. Pose recognition is particularly useful for identifying camera views in databases, video sequences, video streams, and live recordings. All of these applications require a fast pose recognition process, in many cases video real-time. It should further be possible to extend the database with new material, i.e., to update the recognition system online. The method that we propose is based on P-channels, a special kind of information representation which combines advantages of histograms and local linear models. Our approach is motivated by its similarity to information representation in biological systems but its main advantage is its robustness against common distortions such as clutter and occlusion. The recognition algorithm consists of three steps: (1) low-level image features for color and local orientation are extracted in each point of the image; (2) these features are encoded into P-channels by combining similar features within local image regions; (3) the query P-channels are compared to a set of prototype P-channels in a database using a least-squares approach. The algorithm is applied in two scene registration experiments with fisheye camera data, one for pose interpolation from synthetic images and one for finding the nearest view in a set of real images. The method compares favorable to SIFT-based methods, in particular concerning interpolation. The method can be used for initializing pose-tracking systems, either when starting the tracking or when the tracking has failed and the system needs to re-initialize. Due to its real-time performance, the method can also be embedded directly into the tracking system, allowing a sensor fusion unit choosing dynamically between the frame-by-frame tracking and the pose recognition. Contributions: The author contributed with implementation, partly theory and partly writing, while Felsberg contributed with the idea, theory and writing.

4

CHAPTER 1. INTRODUCTION

Paper B: KLT Tracking Implementation on the GPU J. Hedborg, J. Skoglund, and M. Felsberg. KLT tracking implementation on the GPU. In Proceedings SSBA 2007, 2007. Abstract: The GPU is the main processing unit on a graphics card.A modern GPU typically provides more than ten times the computational power of an ordinary PC processor. This is a result of the high demands for speed and image quality in computer games. This paper investigates the possibility of exploiting this computational power for tracking points in image sequences.Tracking points is used in many computer vision tasks, such as tracking moving objects, structure from motion, face tracking etc. The algorithm was successfully implemented on the GPU and a large speed up was achieved. Contributions: The author contributed with the idea, theory, implementation and writing. The co-authors contributed with writing and partly theory. Paper C: Fast and Accurate Structure and Motion Estimation J. Hedborg, P.-E. Forss´en, and M. Felsberg. Fast and accurate structure and motion estimation. In International Symposium on Visual Computing, volume Volume 5875 of Lecture Notes in Computer Science, pages 211–222, Berlin Heidelberg, 2009. Springer-Verlag.

Abstract: This paper describes a system for structure-and-motion estimation for real-time navigation and obstacle avoidance. We demonstrate a technique to increase the efficiency of the 5-point solution to the relative pose problem. This is achieved by a novel sampling scheme, where we add a distance constraint on the sampled points inside the RANSAC loop, before calculating the 5-point solution.

1.2. OUTLINE

5

Our setup uses the KLT tracker to establish point correspondences across time in live video. We also demonstrate how an early outlier rejection in the tracker improves performance in scenes with plenty of occlusions. This outlier rejection scheme is well suited to implementation on graphics hardware. We evaluate the proposed algorithms using real camera sequences with fine-tuned bundle adjusted data as ground truth. To strenghten our results we also evaluate using sequences generated by a state-of-the-art rendering software. On average we are able to reduce the number of RANSAC iterations by half and thereby double the speed. Contributions: The author contributed with the idea, theory, implementation and writing. The co-authors contributed with writing and partly theory. Paper D: Fast Iterative Five point Relative Pose Estimation J. Hedborg and M. Felsberg. Fast iterative five point relative pose estimation. Journal of real-time image processing, Under review, 2012. Abstract: Robust estimation of the relative pose between two cameras is a fundamental part of Structure and Motion methods. For calibrated cameras, the five point method together with a robust estimator such as RANSAC gives the best result in most cases. The current state-of-the-art method for solving the relative pose problem from five points is due to [28], because it is faster than other methods and in the RANSAC scheme one can improve precision by increasing the number of iterations. In this paper, we propose a new iterative method, which is based on Powell’s Dog Leg algorithm. The new method has the same precision and is approximately twice as fast as Nist´er’s algorithm. The proposed algorithm is systematically evaluated on two types of datasets with known ground truth. Contributions: The author contributed with the idea, theory, implementation and writing. The co-author contributed with writing and partly theory. Paper E: Structure and Motion Estimation from Rolling Shutter Video J. Hedborg, E. Ringaby, P.-E. Forss´en, and M. Felsberg. Structure and motion estimation from rolling shutter video. In ICCV 2011 Workshop, 2nd IEEE Workshop on Mobile Vision, 2011. Abstract: The majority of consumer quality cameras sold today have CMOS sensors with rolling shutters. In a rolling-shutter camera, images are read out row by row, and thus each row is exposed during a different time interval. A rollingshutter exposure causes geometric image distortions when either the camera or the scene is moving, and this causes state-of-the-art structure and motion algorithms to fail. We demonstrate a novel method for solving the structure and motion problem for rolling-shutter video. The method relies on exploiting the continuity of the camera motion, both between frames, and across a frame. We demonstrate

6

CHAPTER 1. INTRODUCTION

the effectiveness of our method by controlled experiments on real video sequences. We show, both visually and quantitatively, that our method outperforms standard structure and motion, and is more accurate and efficient than a two-step approach, doing image rectification and structure and motion. Contributions: The author contributed with the idea, theory, implementation and writing. The co-authors contributed with writing and partly theory. Paper F: Rolling Shutter Bundle Adjustment J. Hedborg, P.-E. Forss´en, M. Felsberg, and E. Ringaby. Rolling shutter bundle adjustment. In IEEE Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, USA, June 2012. IEEE Computer Society, IEEE. Accepted.

1.2. OUTLINE

7

Abstract: This paper introduces a bundle adjustment (BA) method that obtains accurate structure and motion from rolling shutter (RS) video sequences: RSBA. When a classical BA algorithm processes a rolling shutter video, the resultant camera trajectory is brittle, and complete failures are not uncommon. We exploit the temporal continuity of the camera motion to define residuals of image point trajectories with respect to the camera trajectory. We compare the camera trajectories from RSBA to those from classical BA, and from classical BA on rectified videos. The comparisons are done on real video sequences from an iPhone 4, with ground truth obtained from a global shutter camera, rigidly mounted to the iPhone 4. Compared to classical BA, the rolling shutter model requires just six extra parameters. It also degrades the sparsity of the system Jacobian slightly, but as we demonstrate, the increase in computation time is moderate. Decisive advantages are that RSBA succeeds in cases where competing methods diverge, and consistently produces more accurate results. Contributions: The author contributed with the idea, theory, implementation and writing. The co-authors contributed with writing and partly theory.

8

CHAPTER 1. INTRODUCTION

Part I

Background Theory

9

Chapter 2

Least Squares Problems A large set of computer vision problems can be formulated within some minimization framework. Methods developed in this thesis are, to a large extent, fitted into a least squares optimization framework. As indicated by the name, a least squares problem has a cost function (or error function) which is a sum of a set of squared costs, and although not being robust against data which has a large deviation from the model (outliers), the least squares methods are usually efficient and have a high rate of convergence. This chapter is a brief introduction to some of the least squares methods used in this thesis. For a more in-depth presentation, the reader is referred to the report [24].

2.1

Problem Formulation

A least squares problem aims to find the n dimensional vector x that minimizes I

F (x) =

1X fi (x)2 , 2 i=1

(2.1)

where fi :