AN INTRODUCTION TO 3D COMPUTER VISION TECHNIQUES AND ALGORITHMS Bogustaw Cyganek Department of Electronics, AGH University of Science and Technology, Poland

J. Paul Siebert Department of Computing Science, University of Glasgow, Scotland, UK

©WILEY A John Wiley and Sons, Ltd., Publication

Contents Preface

xv

Acknowledgements

xvii

Notation and Abbreviations

xix

Part I

1

1 1.1 1.2 1.3 1.4

Introduction Stereo-pair Images and Depth Perception 3D Vision Systems 3D Vision Applications Contents Overview: The 3D Vision Task in Stages

3 4 4 5 6

2 2.1 2.2 2.3

Brief History of Research on Vision Abstract Retrospective of Vision Research Closure 2.3.1 Further Reading

9 9 9 14 14

Part II

15

3 3.1 3.2 3.3

17 17 18 23 24 24 26 27 28 29 30

2D and 3D Vision Formation Abstract Human Visual System Geometry and Acquisition of a Single Image 3.3.1 Projective Transformation 3.3.2 Simple Camera System: the Pin-hole Model 3.3.2.1 Extrinsic Parameters 3.3.2.2 Intrinsic Parameters 3.3.3 Projective Transformation of the Pin-hole Camera 3.3.4 Special Camera Setups 3.3.5 Parameters of Real Camera Systems

viii

3.4 Stereoscopic Acquisition Systems 3.4.1 Epipolar Geometry 3.4.1.1 Fundamental Matrix 3.4.1.2 Epipolar Lines and Epipoles 3.4.2 Canonical Stereoscopic System 3.4.3 Disparity in the General Case 3.4.4 Bifocal, Trifocal and Multifocal Tensors 3.4.5 Finding the Essential and Fundamental Matrices 3.4.5.1 Point Normalization for the Linear Method 3.4.5.2 Computing F in Practice 3.4.6 Dealing with Outliers 3.4.7 Catadioptric Stereo Systems 3.4.8 Image Rectification 3.4.9 Depth Resolution in Stereo Setups 3.4.10 Stereo Images and Reference Data 3.5 Stereo Matching Constraints 3.6 Calibration of Cameras 3.6.1 Standard Calibration Methods 3.6.2 Photometric Calibration 3.6.3 Self-calibration 3.6.4 Calibration of the Stereo Setup 3.7 Practical Examples 3.7.1 Image Representation and Basic Structures 3.7.1.1 Computer Representation of Pixels 3.7.1.2 Representation of Images 3.7.1.3 Image Operations 3.8 Appendix: Derivation of the Pin-hole Camera Transformation 3.9 Closure 3.9.1 Further Reading 3.9.2 Problems and Exercises 4 Low-level Image Processing for Image Matching 4.1 Abstract 4.2 Basic Concepts 4.2.1 Convolution and Filtering 4.2.2 Filter Separability 4.3 Discrete Averaging 4.3.1 Gaussian Filter 4.3.2 Binomial Filter 4.3.2.1 Specification of the Binomial Filter 4.3.2.2 Spectral Properties of the Binomial Filter 4.4 Discrete Differentiation 4.4.1 Optimized Differentiating Filters 4.4.2 Savitzky-Golay Filters 4.4.2.1 Generation of Savitzky-Golay Filter Coefficients

Contents 31 31 34 35 36 38 39 41 44 46 49 54 55 59 61 66 70 71 73 73 74 75 75 76 78 87 91 93 93 94 95 95 95 95 97 99 100 101 101 102 105 105 108 114

Contents

IX

4.5 Edge Detection 4.5.1 Edges from Signal Gradient 4.5.2 Edges from the Savitzky-Golay Filter 4.5.3 Laplacian of Gaussian 4.5.4 Difference of Gaussians 4.5.5 Morphological Edge Detector 4.6 Structural Tensor 4.6.1 Locally Oriented Neighbourhoods in Images 4.6.1.1 Local Neighbourhood with Orientation 4.6.1.2 Definition of a Local Neighbourhood of Pixels 4.6.2 Tensor Representation of Local Neighbourhoods 4.6.2.1 2D Structural Tensor 4.6.2.2 Computation of the Structural Tensor 4.6.3 Multichannel Image Processing with Structural Tensor 4.7 Corner Detection 4.7.1 The Most Common Corner Detectors 4.7.2 Corner Detection with the Structural Tensor 4.8 Practical Examples 4.8.1 C++Implementations 4.8.1.1 Convolution 4.8.1.2 Implementing the Structural Tensor 4.8.2 Implementation of the Morphological Operators 4.8 J Examples in Matlab: Computation of the SVD 4.9 Closure 4.9.7 Further Reading 4.9.2 Problems and Exercises

115 117 119 120 126 127 127 128 130 130 133 136 140 143 144 144 149 151 151 151 155 157 161 162 163 163

5 Scale-space Vision 5.1 Abstract 5.2 Basic Concepts 5.2.1 Context 5.2.2 Image Scale 5.2.3 Image Matching Over Scale 5.3 Constructing a Scale-space 5.5.7 Gaussian Scale-space 5.3.2 Differential Scale-space 5.4 Multi-resolution Pyramids 5.4.7 Introducing Multi-resolution Pyramids 5.4.2 How to Build Pyramids 5.4.J Constructing Regular Gaussian Pyramids 5.4.4 Laplacian of Gaussian Pyramids 5.4.5 Expanding Pyramid Levels 5.4.6 Semi-pyramids 5.5 Practical Examples 5.5.7 C++Examples 5.5.7.7 Building the Laplacian and Gaussian Pyramids in C++

165 165 165 165 166 166 168 168 170 172 172 175 175 Yll 178 179 181 181 181

Contents 5.5.2 Matlab Examples 5.5.2.1 Building the Gaussian Pyramid in Matlab 5.5.2.2 Building the Laplacian of Gaussians Pyramid in Matlab 5.6 Closure 5.6.1 Chapter Summary 5.6.2 Further Reading 5.6.3 Problems and Exercises

186 190 190 191 191 191 192

6

193 193 193 194 194 198 199 201 202 205 206 209 212 214 215

Image Matching Algorithms 6.1 Abstract 6.2 Basic Concepts 6.3 Match Measures 6.3.1 Distances of Image Regions 6.3.2 Matching Distances for Bit Strings 6.3.3 Matching Distances for Multichannel Images 6.3.3.1 Statistical Distances 6.3.4 Measures Based on Theory of Information 6.3.5 Histogram Matching 6.3.6 Efficient Computations of Distances 6.3.7 Nonparametric Image Transformations 6.3.7.1 Reduced Census Coding 6.3.7.2 Sparse Census Relations 6.3.7.3 Fuzzy Relationships Among Pixels 6.3.7.4 Implementation of Nonparametric Image Transformations 6.3.8 Log-polar Transformation for Image Matching 6.4 Computational Aspects of Matching 6.4.7 Occlusions 6.4.2 Disparity Estimation with Subpixel Accuracy 6.4.3 Evaluation Methods for Stereo Algorithms 6.5 Diversity of Stereo Matching Methods 6.5.7 Structure of Stereo Matching Algorithms 6.5.1.1 Aggregation of the Cost Values 6.5.1.2 Computation of the Disparity Map 6.5.1.3 Disparity Map Postprocessing 6.6 Area-based Matching 6.6.7 Basic Search Approach 6.6.2 Interpreting Match Cost 6.6.3 Point-oriented Implementation 6.6.4 Disparity-oriented Implementation 6.6.5 Complexity of Area-based Matching 6.6.6 Disparity Map Cross-checking 6.6.7 Area-based Matching in Practice 6.6.7.7 Intensity Matching 6.6.7.2 Area-based Matching in Nonparametric Image Space 6.6.7.3 Area-based Matching with the Structural Tensor

216 218 222 222 224 226 229 233 234 235 237 238 239 241 245 250 256 257 259 260 260 262

Contents

xi

6.7 Area-based Elastic Matching 6.7.1 Elastic Matching at a Single Scale 6.7.1.1 Disparity Match Range 6.7.1.2 Search and Subpixel Disparity Estimation 6.7.2 Elastic Matching Concept 6.7.3 Scale-based Search 6.7.4 Coarse-to-fine Matching Over Scale 6.7.5 Scale Subdivision 6.7.6 Confidence Over Scale 6.7.7 Einal Multi-resolution Matcher 6.8 Feature-based Image Matching 6.8.1 Zero-crossing Matching 6.8.2 Corner-based Matching 6.8.3 Edge-based Matching: The Shirai Method 6.9 Gradient-based Matching 6.10 Method of Dynamic Programming 6.10.1 Dynamic Programming Formulation of the Stereo Problem 6.11 Graph Cut Approach 6.11.1 Graph Cut Algorithm 6.11.1.1 Graphs in Computer Vision 6.11.1.2 Optimization on Graphs 6.11.2 Stereo as a Voxel Labelling Problem 6.11.3 Stereo as a Pixel Labelling Problem 6.12 Optical Flow 6.13 Practical Examples 6.13.1 Stereo Matching Hierarchy in C++ 6.13.2 Log-polar Transformation 6.14 Closure 6.14.1 Further Reading 6.14.2 Problems and Exercises

273 273 IIA 275 278 280 283 284 285 286 288 289 292 295 296 298 301 306 306 309 310 311 312 314 318 318 319 321 321 322

7 Space Reconstruction and Multiview Integration 7.1 Abstract 7.2 General 3D Reconstruction 7.2.7 Triangulation 7.2.2 Reconstruction up to a Scale 7.2.3 Reconstruction up to a Projective Transformation 7.3 Multiview Integration 7.3.1 Implicit Surfaces and Marching Cubes 7.3.1.1 Range Map Pre-segmentation 7.3.1.2 Volumetric Integration Algorithm Overview 7.3.1.3 Hole Filling 7.3.1.4 Marching Cubes 7.3.1.5 Implementation Considerations 7.3.2 Direct Mesh Integration

323 323 323 324 325 327 329 330 331 332 332 333 338 338

7.4 Closure 7.4.7 Further Reading 8

Case Examples 8.1 Abstract 8.2 3D System for Vision-Impaired Persons 8.3 Face and Body Modelling 8.3.1 Development of Face and Body Capture Systems 8.3.2 Imaging Resolution, 3D Resolution and Implications for Applications 8.3.3 3D Capture and Analysis Pipeline for Constructing Virtual Humans 8.4 Clinical and Veterinary Applications 8.4.1 Development of 3D Clinical Photography 8.4.2 Clinical Requirements for 3D Imaging 8.4.3 Clinical Assessment Based on 3D Surface Anatomy 8.4.4 Extraction of Basic 3D Anatomic Measurements 8.4.5 Vector Field Surface Analysis by Means of Dense Correspondences 8.4.6 Eigenspace Methods 8.4.7 Clinical and Veterinary Examples 8.4.8 Multimodal 3D Imaging 8.5 Movie Restoration 8.6 Closure 8.6.1 Further Reading

342 342 343 343 343 345 345 346 350 352 352 353 353 354 357 359 362 367 370 374 374

Part III

375

9 Basics of the Projective Geometry 9.1 Abstract 9.2 Homogeneous Coordinates 9.3 Point, Line and the Rule of Duality 9.4 Point and Line at Infinity 9.5 Basics on Conies 9.5.1 Conies in p2 9.5.1.1 The Dual Conic 9.5.1.2 Circular Points 9.5.2 Conies in p2 9.5.2.1 The Absolute Conic 9.5.2.2 The Dual Absolute Conic 9.6 Group of Projective Transformations 9.6.7 Projective Base 9.6.2 Hyperplanes 9.6.3 Projective Homographies 9.7 Projective Invariants 9.8 Closure 9.8.1 Further Reading

377 377 377 379 380 382 382 383 383 384 384 385 385 385 386 386 387 388 389

Contents 10 Basics of Tensor Calculus for Image Processing 10.1 Abstract 10.2 Basic Concepts 10.2.1 Linear Operators 10.2.2 Change of Coordinate Systems: Jacobians 10.3 Change of a Base 10.4 Laws of Tensor Transformations 10.5 The Metric Tensor 10.5.1 Covariant and Contravariant Components in a Curvilinear Coordinate System 10.5.2 The First Fundamental Form 10.6 Simple Tensor Algebra 10.6.1 Tensor Summation 70.6.2 Tensor Product 10.6.3 Contraction and Tensor Inner Product 10.6.4 Reduction to Principal Axes 10.6.5 Tensor Invariants 10.7 Closure 10.7.1 Further Reading

хш 391 391 391 392 393 394 396 397 397 399 399 399 400 400 400 401 401 401

11 11.1 11.2 11.3 11.4 11.5

Distortions and Noise in Images Abstract Types and Models of Noise Generating Noisy Test Images Generating Random Numbers with Normal Distributions Closure 77.5.7 Further Reading

403 403 403 405 407 408 408

12 12.1 12.2 12.3

Image Warping Procedures Abstract Architecture of the Warping System Coordinate Transformation Module 72.5.7 Projective and Affine Transformations of a Plane 12.3.2 Polynomial Transformations 12.3.3 Generic Coordinates Mapping Interpolation of Pixel Values 12.4.1 Bilinear Interpolation 12.4.2 Interpolation of Nonscalar-Valued Pixels The Warp Engine Software Model of the Warping Schemes 12.6.1 Coordinate Transformation Hierarchy 12.6.2 Interpolation Hierarchy 12.6.3 Image Warp Hierarchy Warp Examples Finding the Linear Transformation from Point Correspondences 12.8.1 Linear Algebra on Images

409 409 409 410 410 411 412 412 412 414 414 415 415 416 416 419 420 424

12.4

12.5 12.6

12.7 12.8

Contents

XIV

12.9 Closure 12.9.1 Further Reading

427 428

13 Programming Techniques for Image Processing and Computer Vision 13.1 Abstract 13.2 Useful Techniques and Methodology 13.2.1 Design and Implementation 13.2.1.1 Comments and Descriptions of 'Ideas' 13.2.1.2 Naming Conventions 13.2.1.3 Unified Modelling Language (UML) 13.2.2 Template Classes 13.2.2.1 Expression Templates 13.2.3 Asserting Code Correctness 13.2.3.1 Programming by Contract 13.2.4 Debugging Issues 13.3 Design Patterns 13.3.1 Template Function Objects 13.3.2 Handle-body or Bridge 13.3.3 Composite 13.3.4 Strategy 13.3.5 Class Policies and Traits 13.3.6 Singleton 13.3.7 Proxy 13.3.8 Factory Method 13.3.9 Prototype 13.4 Object Lifetime and Memory Management 13.5 Image Processing Platforms 73.5.7 Image Processing Libraries 13.5.2 Writing Software for Different Platforms 13.6 Closure 13.6.1 Further Reading

429 429 430 430 430 431 431 436 437 438 438 440 441 441 442 445 447 448 450 450 451 452 453 455 455 455 456 456

14

457

Image Processing Library

References

459

Index

475