3D Object Retrieval: history and future challenges Benjamin Bustos CIWS / PRISMA Research Group Department of Computer Science University of Chile [email protected]

Motivation

Lots of practical applications for this problem!

2

Motivation


Main problems for solving 3D object retrieval 

Representation of the 3D model






How to measure similarity?




Objects at different scales / “resolution” / pose Triangle meshes / Point clouds Quality (holes, badly oriented triangles, etc.) Geometric or semantic?

What about partial similarity?


Whole-to-part / Part-to-whole

3

Outline





Global descriptors Local descriptors Evaluation: SHREC Tracks Future challenges

4

GLOBAL DESCRIPTORS

5

3D model similarity search


Geometric or semantic similarity?

6

3D model similarity search


General approach: feature vectors

7

3D model similarity search


Requirements for 3D descriptors 

Invariance with respect to rotations, translations, scaling and (in some cases) reflections

8

3D model similarity search


Requirements for 3D descriptors 

Robustness with respect to level-of-detail (different tessellations)

9

3D model similarity search


Requirements for 3D descriptors    

Robustness with respect to outliers Efficient feature extraction Multiresolution representation Discrimination of different geometric forms

10

Normalization of 3D objects


Goal 




Result 




Find a canonical coordinate system Invariance against translation, rotation, scaling and reflections

Tools*    

Center of mass (translation) Continuous PCA (rotation) Test based on moments (reflection) Scaling factor

* D. Vranic, D. Saupe, and J. Richter. Tools for 3D-object Retrieval: Karhunen-Loeve Transform and Spherical Harmonics. In Proc. IEEE Workshop Multimedia Signal Processing, pages 293—298, 2001

11

Normalization of 3D objects


Global transformation of the object with PCA

12

Normalization of 3D objects


Examples of pose normalization with PCA

13

3D descriptors


Extracting 3D feature vectors*

* B. Bustos, D. Keim, D. Saupe, T. Schreck, and D. Vranic. Feature-based similarity search in 3D object databases. ACM Computing Surveys, 37(4):345-387, 2005

14

Shape distribution with D2*


Category: Statistical   

Select two random points on the surface Compute Euclidean distance between the points Repeat the process and construct a histogram of distances

* R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin. Shape distributions. ACM Transactions on Graphics, 21(4):807—832, 2002

15

Extended gaussian image*


Category: surface geometry

* B. Horn. Extended Gaussian Image. Proceedings of the IEEE, 72(12):1671—1686, 1984 16

Ray-based*


Category: extension-based (dim=162)

* D. Vranic and D. Saupe. 3D Model Retrieval. In Proc. Spring Conference on Computer Graphics and its Applications (SCCG’00), pages 89—93, 2000

17

Volume-based FV*


Category: volume-based

* M. Heczko, D. Keim, D. Saupe, and D. Vranic. Methods for similarity search of 3D objects. Datenbank-Spektrum, 2(2):54—63, dpunkt.verlag, 2002

18

Depth-buffer FV*


Category: image-based




Apply 2D discrete Fourier transform to the six images

* M. Heczko, D. Keim, D. Saupe, and D. Vranic. Methods for similarity search of 3D objects. Datenbank-Spektrum, 2(2):54—63, dpunkt.verlag, 2002

19

Experimental evaluation


Experimental framework* 

1,838 3D objects collected from the web





 

472 classified objects in 55 classes (planes, cars, animals, swords, plants, human figures, etc.) Konstanz Dataset

16 implemented 3D descriptors Effectiveness evaluation measures



Precision vs. recall diagrams R-precision

* B. Bustos, D. Keim, D. Saupe, T. Schreck, and D. Vranic. An experimental effectiveness comparison of methods for 3D similarity search. International Journal on Digital Libraries, Special issue on Multimedia Contents and Management in Digital Libraries, 6(1):39-54, 2006

20

Experimental evaluation


Average precision vs. recall

21

Experimental evaluation


Dependency on dimensionality

22

Experimental evaluation


Dependency on query object

23

Experimental evaluation


Dependency on query object

24

Disadvantages of global descriptors


Many other global descriptors:  




But:  




Image-based: best in practice Combinations of descriptors (fixed, dynamic) Global descriptors suffer from pose sensitivity They are not well-suited for partial similarity

New approach: Local descriptors!  

Avoid the canonical pose problem Support non-rigid and partial matching 25

LOCAL DESCRIPTORS

26

Local descriptors for 3DOR


Main approach  




Each shape is represented as a set of descriptors The matching is performed by finding correspondences

Example: Shape Google *  

Heat kernel signatures Bag-of-features approach

* Ovsjanikov, M., Bronstein, A.M., Guibas, L.J., Bronstein, M.M.: Shape Google: a computer vision approach to invariant shape retrieval. In: Proc. Workshop on Non-Rigid Shape Anal. and Deform. Image Alignment, 2009

27

Key-points


Interest points on the 3D surface 

Harris 3D method *

* I. Sipiran and B. Bustos. Harris 3D: A robust extension of the Harris operator for interest point detection on 3D meshes. The Visual Computer, 27(11):963-976, 2011

28

Key-points


Harris 3D is a robust interest points detector

29

Key-components


Regions of interest on the 3D shape * 

Based on key-points

* I. Sipiran and B. Bustos. Key-components: Detection of salient regions on 3D meshes. The Visual Computer 29(12):1319-1332, 2013

30

Key-components


Main idea: clustering of key-points in geodesic space

31

Key-components


Key-components are robust

32

Finding correspondences *

* I. Sipiran and B. Bustos. A fully hierarchical approach for finding correspondences in non-rigid shapes. In Proc. 14th IEEE International Conference on Computer Vision (ICCV'13), pages 817-824. IEEE, 2013

33

Finding correspondences


Pre-processing  

 

Key-point detection: Harris 3D algorithm Compute geodesic distances between every pair of key-points Discard key-points with low density For each remaining key-point, compute descriptors



Internal nodes: Heat Kernel Signature [Sun et al. 2009] Leaves: Wave Kernel Signature [Aubry et al. 2011]

34

Finding correspondences


Decomposition tree

35

Finding correspondences


Decomposition tree: node creation 

  

Compute smallest sphere that encloses keypoints within the cluster Find the medoid of the cluster Extract mesh patch (key-component) Compute descriptor for the region




Average key-points descriptor

Continue decomposition if possible



Area patch is large It contains enough key-points 36

Finding correspondences


Hierarchical matching 

Given two shapes and their corresponding decomposition trees, matching is performed levelwise

37

Finding correspondences


SHREC'2010 correspondence dataset 



It evaluates the robustness of correspondences to transformations: holes, noise, topology changes, etc. Data:



Three shapes (null shapes) Set of transformed versions of the null shapes 




45 transformed shapes for each null shape

Measure: geodesic error

38

Experimental evaluation


Some results

39

Experimental evaluation


Strong perturbation of the data

40

EVALUATION: SHREC TRACKS

41

SHREC tracks






Objective: “evaluate the effectiveness of 3Dshape retrieval algorithms” * Started in 2006 It provide many resources for evaluating 3D retrieval methods:   

Reference collections Descriptions of the algorithms But no open source code (yet)

* http://www.projects.science.uu.nl/shrec/

42

SHREC tracks


Current typical tracks:   



Non-rigid shape matching 3D sketch-based retrieval Retrieval based on range scans or depth scanners Multimodal 3D object retrieval

43

SHREC tracks


Typical evaluation measures:     

Mean Average Precision Nearest Neighbor First Tier Second Tier Discounted Cumulative Gain

44

SHREC Track 2013


Large-Scale Partial Shape Retrieval Track Using Simulated Range Images  



Task: partial retrieval (part-to-whole) Queries were simulated as if they were taken using an RGB-D camera Dataset:



360 models (20 classes, 18 objects per class) Queries: 20 partial views per model, 7.200 queries

45

SHREC Track 2013


Object classes:

46

SHREC Track 2013


Query simulation:

47

SHREC Track 2013



Ten teams registered for the track… … but only two teams submitted results! 



Some teams had problems regarding the size of the dataset Other teams had problems processing the simulated range scans

48

SHREC Track 2013


Main results

49

SHREC Track 2013


Conclusions 



Overall performance was moderate: problem far from being solved Efficiency and robustness issues



How to deal with noisy or “imperfect” data? How to do it fast?

50

SHREC Track 2015




Scalability of 3D Shape Retrieval Task: non-rigid shape retrieval Dataset: 

229 non-rigid shapes, 9 classes [Summer]



Plus objects from other well known datasets Plus parts of objects using the simulated range scans (post-processed)

51

SHREC Track 2015



Our dataset: 96,487 models


Previous largest dataset: 8,987 models

52

SHREC Track 2015



Four teams participated in the track We wanted to evaluate efficiency 




First time that information about the hardware was requested to teams

For evaluation of effectiveness, we used the standard measures (NN, FT, ST, MAP)

53

SHREC Track 2015


Main results: effectiveness

54

SHREC Track 2015


Conclusions: 

 

Near-perfect effectiveness with one of the methods Efficiency vs. effectiveness trade-off Effective methods are not scalable (yet)

55

FUTURE CHALLENGES

56

Future challenges




Partial matching “Functional” matching Scalability issues   

Preprocessing steps Computation of the 3D descriptors Similarity search

57

Future challenges





Similarity search using point cloud data directly obtained from sensors Multimodal search

58

Thank you for your attention!

59