Annotation Propagation in Large Image Databases via Dense Image Correspondence Supplemental Material

Annotation Propagation in Large Image Databases via Dense Image Correspondence Supplemental Material 1 Additional Results Here we show more results...
Author: Cori Richard
1 downloads 3 Views 8MB Size
Annotation Propagation in Large Image Databases via Dense Image Correspondence Supplemental Material

1

Additional Results

Here we show more results of our system on the datasets we experimented with. Description of the datasets and the experiment setups are given in Section 5 in the paper. We recommend viewing the results electronically and zoom in for more details. Notice that some figures span more than one page. We indexed the figures in a grid to allow referencing particular results. We note that all the results are for images that are originally untagged and unlabeled in the database (the subset of images I \ It ). In Fig. 1 we show more results on LabelMe Outdoors (LMO) dataset [1]. In addition to the final result, we also show the MAP labels based on local evidence alone (the appearance model in Section 3, Eqn. 2), similar to Fig. 3(c) in the paper. Fig. 2 shows the estimated spatial prior of each word (Eqn. 7) in LMO’s vocabulary. It can be seen that the prior agrees with the true spatial prior, computed from the ground truth labels, for more frequent words. The estimated prior is somewhat blurrier than the ground truth, indicating some errors in classification, however the general layout is captured correctly. For example, sky is mostly at the top of the image, building is in the middle, and road and sea are at the bottom. Fig. 3 shows more results on SUN dataset [2], as well as comparison with the results by [3], similar to Fig. 8 in the paper. The results of [3] were produced using the authors’ original implementation (available online), modified by us to account for tagged images as described in Section 3 in their paper (termed “weak supervision”). Taken together with Fig. 1, these results show that the algorithm can handle large variety of both indoor and outdoor scenes. Notice that while SUN has a relatively large vocabulary (500+ words), the tags inferred by the algorithm tend to correspond to words with higher frequency in the dataset. That is because words that occur frequently, and co-occur frequently with other words, are considered more probable by the algorithm (Eqn. 3). Fig. 4 and 5 show more results on the ESP game dataset [4] and IAPR benchmark [5], where we used the same images and vocabulary as in [6] (available online). These two datasets are much noisier in terms of both image content and vocabulary, and so are more challenging for the algorithm. In particular, both datasets include more abstract words (e.g. smile, night) that are harder to model, as well as words that might not correspond to a particular image region (e.g. photo). Finally, Fig. 6 shows more failure cases on all datasets. Limitations of the system include incorrect classification under similar visual appearance or insufficient exemplars of particular words (e.g. row 1 columns 2-3, row 3 columns 1,3 in (a), row 1 column 1 in (b), row 2 columns 2-3 in (c)), and errors due to incorrect inter-image correspondence (e.g. row 1 column 1, row 5 columns 1,3 in (a), row 2 column 3 in (b), row 1 column 1 in (c)).

2

1

2 tree sky river plant mountain

tree sky river plant mountain

tree sky sea mountain building

tree sky sea mountain building

sky sea plant mountain building

sky sea plant mountain building

tree sky sea mountain building

tree sky sea building

sky sea sand person building

sky sea sand building

unlabeled tree sky river

tree sky river

tree sky mountain grass building

tree sky mountain grass building

tree sky person grass

tree sky person grass

unlabeled tree sky rock plant person

tree sky plant person

tree sky plant mountain grass

tree sky plant mountain grass

tree sky rock river plant

tree sky rock river plant

unlabeled tree sky river mountain

tree sky river mountain

tree sky river plant mountain

tree sky plant mountain

tree sky person mountain building

tree sky mountain building

tree sky river person mountain

tree sky river person mountain

unlabeled tree sky river plant

tree sky river plant

streetlight sky road crosswalk car building

sky road car building

tree streetlight sky sign road car

tree sky sign road car unlabeled

tree sky sign road car

tree sky sign road car

tree sky sign road car

tree sky sign road car

1

2

3

4

5

6

7

8

9

10 Source

Appearance model

Result

Source

Appearance model

Result

Title Suppressed Due to Excessive Length

1

3

2 tree sky sign road car

tree sky sign road car unlabeled

tree sky sign road car

tree sky sign road car

unlabeled tree sky road mountain car

tree sky road mountain car

unlabeled sky sidewalk road car building

sky sidewalk road building

window tree sky door building

window tree sky door building

unlabeled window sky grass building

window sky grass building

unlabeled window sidewalk road door building

window sidewalk road door building

unlabeled window sky road door building

window sky road door building

window sidewalk road door building

window sidewalk road door building

unlabeled tree sidewalk road car building

tree sidewalk road car building

tree sky road car building

tree sky road car building

sky sidewalk road car building

sky sidewalk road car building

unlabeled window tree car building awning

window tree car building awning

tree sky road mountain

tree sky road mountain

tree sky river mountain

tree sky river mountain

tree sky sidewalk road person car building bridge

tree sky sidewalk road person car building bridge

tree streetlight sky sidewalk road person mountain car building

tree sky sidewalk road mountain car building

unlabeled tree sky building

tree sky building

11

12

13

14

15

16

17

18

19 Source

Appearance model

Result

Source

Appearance model

Result

Fig. 1. More results on LMO. For each example, we show the source image on the left, the MAP labeling using the appearance model only in the middle (computed independently at each pixel; see Fig. 3 in the paper), and the final result of the annotation propagation algorithm (appearance model + spatial regularization + regularization via dense image correspondences) on the right. Note that the final result might not contain all tags from the appearance model result.

4

Estimated

Ground truth

sky

building

mountain

tree

road

sea

field

grass

river

plant

car

sand

rock

sidewalk

window

desert

door

bridge

person

fence

balcony

staircase

awning

crosswalk

sign

streetlight

boat

pole

bus

sun

cow

bird

moon

Fig. 2. The estimated spatial prior hsl (Eqn. 7) for the LMO vocabulary. Words are ordered from top left to bottom right according to their frequency in the dataset. For each word, the left image is the estimated prior and the right image is the true prior according to human labels. The colormap is the same as Fig. 1 above, with saturation corresponding to probability, from white (zero probability) to saturated (high probability).

Title Suppressed Due to Excessive Length

1

2 wall floor door ceiling lamp ceiling

exercise machine entrance dresser desk lamp curtain cuddly toy crosswalk counter chandelier chair ceiling lamp ceiling cabinet bulletin board box bottle boat bathtub bars balcony

shelves person floor curtain

extractor hood entrance door desk lamp cushion cubicle counter column chair ceiling lamp ceiling candle bulletin board bridge bowl bottle book balcony bag arcade

window wall floor ceiling lamp

person painting outlet mountain microwave machine large window floor deck chair curtain cupboard countertop cliff central reservation ceiling car building bleachers bench bed

window wall floor

handrail ground glass wall fountain fence faucet extractor hood door desk lamp cushion curtain cupboard cliff chest car bulletin board building bridge boat balustrade

shelves person floor curtain

court clock chair ceiling lamp ceiling car cabinet bulletin board building box bottle boat bleachers bench barrel bag awning arcade altarpiece airplane

wall floor door ceiling lamp ceiling

flowers floor flag deck chair curtain conveyor belt coffee maker cloud ceiling lamp ceiling car cabinet building bridge book boat bleachers awning armchair animal

window wall plant floor ceiling

cup cuddly toy counter column chair ceiling lamp ceiling can cabinet box book bleachers bench bed beam basket barrel armchair arcade apple

window wall floor ceiling bed

countertop counter cloud chimney chandelier chair ceiling lamp ceiling can cabinet bulletin board building bowl bottle book bell bed basket bag altar

window wall table floor ceiling lamp

countertop chest chandelier chair ceiling lamp ceiling can bulletin board bread box bowl bottle book billiard table bench bed bathtub basket bag awning

window wall floor

curtain curb column cloud chandelier chair ceiling lamp candle cabinet building brand name box bowl book bicycle bathtub basket balcony awning armchair

window wall floor ceiling

mirror microwave ground drawer door desk lamp curtain counter column chair ceiling lamp ceiling card building brand name bicycle balcony armchair apple alarm clock

window wall table

candle candies can cabinet building bucket bridge bread brand name box bowl book boat bell bed bars barrel balcony bag alarm clock

shelves person floor curtain

files field extractor hood elevator drawer curtain counter ceiling lamp ceiling car cabinet box bench bed bathtub basket balcony bag altarpiece air conditioning

water tree sky ground

elephant door desk lamp crosswalk counter clothes chair central reservation car candle bulletin board building box bottle bicycle bench bed basket awning airplane

tree sky plant grass

wall tree swimming pool sky shop window sheep sea pole plant pitch path mountain grass floe field deck chair

tree sky mountain building

door dome doll desk lamp curtain cupboard column coffee maker cloud clock ceiling caravan car cabinet building brand name bench basket awning air conditioning

window sky road mountain building

cloud central reservation ceiling card car cabinet building brand name box boat boards bench beam bathtub basket bars barbecue bag backpack altar

tree sky mountain building

person path mountain magazines land ground grass files field embankment elephant door dirt track crosswalk court chair building bucket barrel arcade

window tree sky mountain building

curtain cubicle crosswalk countertop counter city chair ceiling car cabinet building bucket box bottle boat boards bicycle bench awning armchair

tree sky grass building

floor fence elephant door dome curtain countertop counter column cloud chimney chair ceiling lamp car cabinet building brand name bottle blanket balcony

1

2

3

4

5

6

7

8

9

10 Source

AP (ours)

5

STF

Source

AP (ours)

STF

6

1

2 wall table floor ceiling

chandelier chair ceiling lamp ceiling car can bulletin board building bridge bread box bottle bookcase book boat bicycle basket bar bag armchair

wall floor door ceiling lamp ceiling

ground glass wall floor fence door cushion cross conveyor belt ceiling lamp ceiling car cabinet bulletin board building bridge box bench bars armchair air conditioning

tree sky road building

road plant path mountain land ground grass floor field fence door dirt track cupboard court car bulletin board building book boat barrel

tree sky road mountain

separation sea rug road river railing pole pipe person mountain machinery ledge ladder glass wall fountain door conveyor belt car cabinet bulletin board

window tree sidewalk road door building

column closet chair ceiling lamp ceiling car cabinet bulletin board building bucket bridge box boat bar ball balcony bag awning animal air conditioning

window tree sky grass building

pitch person path painting mountain grass flowers field fence faucet dummy door chair ceiling lamp car building bucket box book billboard

wall tray stove drawer ceiling

glass wall garage door frame floor file cabinet fence cutlery curtain counter cloud clock central reservation ceiling lamp ceiling car bread bicycle bench balcony bag

11

12

13

swimming pool sky sea river mountain fish

sky sea mountain

14 sky road mountain building

snowy ground snow sky sign separation sea sand road river refrigerator railing pole plant mountain glass wall field faucet embankment conveyor belt car

window wall plant floor

decoration curtain column closet chair ceiling lamp ceiling car can building box bottle book boat bench bed balcony bag awning armchair

window wall tree door building

dome desk lamp cutlery counter cloud chandelier chair ceiling lamp ceiling can cabinet bulletin board building bowl book bench bed basket bag awning

tree sky mountain building

field fence elephant dirt track decoration cow column chest car bulletin board bridge bookcase book boat bell bed bars balustrade armchair animal

tree sky grass

water tree sky sign sheep rock plant mountain land grass glass wall field embankment curtain conveyor belt ceiling building armchair

15

16

swimming pool sky sea river mountain floe beam

tree sky mountain

17 tree sky person grass building

tree toaster sky plant path leaf hay roll grass grapes deck chair cabinet boat boards bench animal

tree sky mountain building

gate flowers floor fence extractor hood cross clothes chest chair can cabinet bulletin board building bucket book boat bench bell bathtub bag

tree sky mountain building

toy tower teddy bear streetlight stage snow sky rock pillow piano person mountain floor floe embankment dirt track caravan candle building boat

sky mountain building

fence drawer door dome desk lamp cutlery cross column cloud central reservation ceiling lamp card car building boat bleachers bench bed bars animal

wall person floor

sky sink plant painting mirror jar handrail fluorescent tube floor field extractor hood decoration cow column ceiling lamp ceiling bulletin board building bridge balcony

sky person monitor grass ceiling

18

19

20 Source

AP (ours)

STF

Source

AP (ours)

tree sky machine grass ceiling

STF

Title Suppressed Due to Excessive Length

1

2 sky road car building

mirror ground flowers flag fence easel desk lamp deck chair curtain cupboard coffee maker chair ceiling lamp ceiling car building brand name box balcony awning

tree sky road building

sand road printer piano painting mirror mezzanine heater hat fence easel door dome cross car building brand name bleachers balcony armchair

tree sky road building

curtain cuddly toy clothes closet chest chair central reservation ceiling lamp ceiling car building brand name box bottle boat bleachers bed bag armchair animal

window wall floor ceiling

fluorescent tube floor flag fan dummy drawer dish curtain cup counter closet chair ceiling lamp ceiling bread box book beam basket balcony

wall plant floor door

floor file cabinet field fence door knob dirt track deck cubicle column chair ceiling lamp ceiling car bulletin board building bench bars barrel bar air conditioning

tree sky road person building

machine ice rink ground flag faucet extractor hood cup cross counter ceiling car building bridge brand name bleachers bench basket bag awning apple

tree sky grass building

tree streetlight steps sky sculpture screen rock pole plant person path hay roll ground grass grapes fence dirt track deck boat animal

tree sky mountain building

scaffolding rocking chair road projection screen poster piano person mountain mezzanine guitar garage door fish fence embankment dummy curtain building bread bottle bicycle

window tree sky building

chair ceiling lamp ceiling can cabinet bulletin board building bucket bridge box book bleachers bell bed bars bag awning animal altar air conditioning

tree sky ground building

tree sky plant person path mountain grass grandstand field fence elephant door cloud caravan building

window tree sky building

desk lamp curtain cupboard counter cloud chair ceiling lamp car cabinet building brand name box bowl boat basket balcony awning attic alarm clock air conditioning

window tree sky building

refrigerator projection screen pole plateau person mountain hat ground gate fountain floe fence door ceiling building boat boards blanket billiard table bed

sky sea mountain building

drawer door knob dish cupboard counter cloud chair ceiling lamp car cabinet bulletin board building box bowl book billiard table bench bed basket bag

tree sky road building

elephant drawer dome desk lamp cow chest ceiling lamp caravan car cabinet bulletin board building bread box boat bars bag awning armchair apple

21

22

23

24

25

26

27 Source

AP (ours)

7

STF

Source

AP (ours)

STF

Fig. 3. More results on SUN, and comparison with semantic texton forests (STF) [3]. For some images, STF assigned too many tags for a clear visualization, and so we limited the legends to 20 words (omitting any excess words; for visualization purposes only).

8

1

1

2 tree sky old church building

2

plant green flower

3

round red plate circle black

3 woman hair girl face eye

woman white square rectangle picture

woman man hair face black

4

silver money gray gold coin

sky man green grass flower

5

man graph chart blue

yellow white square round red

6

7

8

water sky sea ocean cloud

white tree light building

water tree lake grass flower

tower street sky mountain city

tree sun sky light

tree sky grass

tree house green grass

smile man hat face black

water tree sky mountain lake

woman water swim hair girl

tree sky picture people blue

music man hair eye black

white green drawing diagram

picture man hair black

Fig. 4. More results on ESP.

Title Suppressed Due to Excessive Length

1

2

9

3

1

tree sky range mountain house

water tree statue sky house

tree side road river mountain

2

tree people meadow helmet bush

tourist rock mountain front desert

sky sea road house bush

3

sweater sky man hill grass

window tree tower sky front

4

tourist garden front forest chair

5

tower street square night lamp

view sky house cloud city

6

sky people lake group front

sign road gravel front forest

water sky sea palm beach

sky road lookout landscape canyon

tree slope people mountain bush

sky rock people mountain desert

wall tourist stone front entrance

7

waterfall stage sky rock face

woman slope sky rock mountain

view slope mountain house building

8

tree sun sky horizon

tree sky mountain man front

tree sky rock man cloud

Fig. 5. More results on IAPR.

10

1

2

3

1

tree sky road mountain building

sky sea sand rock mountain

sky sea sand rock mountain

2

tree sky road mountain car

tree sky plant field building

tree sky river plant grass

3

tree sky river mountain building

tree sky road mountain building

tree sky mountain car building

4

sky road door building

window tree ground building

tree sky mountain grass

5

window sky road door building

tree sky sea mountain building

sky road building

(a) SUN (and LMO)

1

tree sky people man green

2

white sun red orange flower

smile girl eye ear black

round gold cross coin circle

tree sky rock green forest

tree star space sky night

(b) ESP

1

2

woman sky man house hat

tree sky hill cloud chair

sky sea sand dune cloud

tree sky plane building airport

sky people middle man desert

window wall table room lamp

(c) IAPR

Fig. 6. More failure cases on SUN, ESP and IAPR.

Title Suppressed Due to Excessive Length

11

References 1. Russell, B., Torralba, A., Murphy, K., Freeman, W.: Labelme: a database and web-based tool for image annotation. IJCV 77 (2008) 157–173 2. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR. (2010) 3485–3492 3. Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: CVPR. (2008) 4. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: SIGCHI. (2004) 5. Grubinger, M., Clough, P., M¨uller, H., Deselaers, T.: The iapr benchmark: A new evaluation resource for visual information systems. In: LREC. (2006) 13–23 6. Makadia, A., Pavlovic, V., Kumar, S.: Baselines for image annotation. IJCV 90 (2010)

Suggest Documents