Annotation Propagation in Large Image Databases via Dense Image Correspondence Supplemental Material
1
Additional Results
Here we show more results of our system on the datasets we experimented with. Description of the datasets and the experiment setups are given in Section 5 in the paper. We recommend viewing the results electronically and zoom in for more details. Notice that some figures span more than one page. We indexed the figures in a grid to allow referencing particular results. We note that all the results are for images that are originally untagged and unlabeled in the database (the subset of images I \ It ). In Fig. 1 we show more results on LabelMe Outdoors (LMO) dataset [1]. In addition to the final result, we also show the MAP labels based on local evidence alone (the appearance model in Section 3, Eqn. 2), similar to Fig. 3(c) in the paper. Fig. 2 shows the estimated spatial prior of each word (Eqn. 7) in LMO’s vocabulary. It can be seen that the prior agrees with the true spatial prior, computed from the ground truth labels, for more frequent words. The estimated prior is somewhat blurrier than the ground truth, indicating some errors in classification, however the general layout is captured correctly. For example, sky is mostly at the top of the image, building is in the middle, and road and sea are at the bottom. Fig. 3 shows more results on SUN dataset [2], as well as comparison with the results by [3], similar to Fig. 8 in the paper. The results of [3] were produced using the authors’ original implementation (available online), modified by us to account for tagged images as described in Section 3 in their paper (termed “weak supervision”). Taken together with Fig. 1, these results show that the algorithm can handle large variety of both indoor and outdoor scenes. Notice that while SUN has a relatively large vocabulary (500+ words), the tags inferred by the algorithm tend to correspond to words with higher frequency in the dataset. That is because words that occur frequently, and co-occur frequently with other words, are considered more probable by the algorithm (Eqn. 3). Fig. 4 and 5 show more results on the ESP game dataset [4] and IAPR benchmark [5], where we used the same images and vocabulary as in [6] (available online). These two datasets are much noisier in terms of both image content and vocabulary, and so are more challenging for the algorithm. In particular, both datasets include more abstract words (e.g. smile, night) that are harder to model, as well as words that might not correspond to a particular image region (e.g. photo). Finally, Fig. 6 shows more failure cases on all datasets. Limitations of the system include incorrect classification under similar visual appearance or insufficient exemplars of particular words (e.g. row 1 columns 2-3, row 3 columns 1,3 in (a), row 1 column 1 in (b), row 2 columns 2-3 in (c)), and errors due to incorrect inter-image correspondence (e.g. row 1 column 1, row 5 columns 1,3 in (a), row 2 column 3 in (b), row 1 column 1 in (c)).
2
1
2 tree sky river plant mountain
tree sky river plant mountain
tree sky sea mountain building
tree sky sea mountain building
sky sea plant mountain building
sky sea plant mountain building
tree sky sea mountain building
tree sky sea building
sky sea sand person building
sky sea sand building
unlabeled tree sky river
tree sky river
tree sky mountain grass building
tree sky mountain grass building
tree sky person grass
tree sky person grass
unlabeled tree sky rock plant person
tree sky plant person
tree sky plant mountain grass
tree sky plant mountain grass
tree sky rock river plant
tree sky rock river plant
unlabeled tree sky river mountain
tree sky river mountain
tree sky river plant mountain
tree sky plant mountain
tree sky person mountain building
tree sky mountain building
tree sky river person mountain
tree sky river person mountain
unlabeled tree sky river plant
tree sky river plant
streetlight sky road crosswalk car building
sky road car building
tree streetlight sky sign road car
tree sky sign road car unlabeled
tree sky sign road car
tree sky sign road car
tree sky sign road car
tree sky sign road car
1
2
3
4
5
6
7
8
9
10 Source
Appearance model
Result
Source
Appearance model
Result
Title Suppressed Due to Excessive Length
1
3
2 tree sky sign road car
tree sky sign road car unlabeled
tree sky sign road car
tree sky sign road car
unlabeled tree sky road mountain car
tree sky road mountain car
unlabeled sky sidewalk road car building
sky sidewalk road building
window tree sky door building
window tree sky door building
unlabeled window sky grass building
window sky grass building
unlabeled window sidewalk road door building
window sidewalk road door building
unlabeled window sky road door building
window sky road door building
window sidewalk road door building
window sidewalk road door building
unlabeled tree sidewalk road car building
tree sidewalk road car building
tree sky road car building
tree sky road car building
sky sidewalk road car building
sky sidewalk road car building
unlabeled window tree car building awning
window tree car building awning
tree sky road mountain
tree sky road mountain
tree sky river mountain
tree sky river mountain
tree sky sidewalk road person car building bridge
tree sky sidewalk road person car building bridge
tree streetlight sky sidewalk road person mountain car building
tree sky sidewalk road mountain car building
unlabeled tree sky building
tree sky building
11
12
13
14
15
16
17
18
19 Source
Appearance model
Result
Source
Appearance model
Result
Fig. 1. More results on LMO. For each example, we show the source image on the left, the MAP labeling using the appearance model only in the middle (computed independently at each pixel; see Fig. 3 in the paper), and the final result of the annotation propagation algorithm (appearance model + spatial regularization + regularization via dense image correspondences) on the right. Note that the final result might not contain all tags from the appearance model result.
4
Estimated
Ground truth
sky
building
mountain
tree
road
sea
field
grass
river
plant
car
sand
rock
sidewalk
window
desert
door
bridge
person
fence
balcony
staircase
awning
crosswalk
sign
streetlight
boat
pole
bus
sun
cow
bird
moon
Fig. 2. The estimated spatial prior hsl (Eqn. 7) for the LMO vocabulary. Words are ordered from top left to bottom right according to their frequency in the dataset. For each word, the left image is the estimated prior and the right image is the true prior according to human labels. The colormap is the same as Fig. 1 above, with saturation corresponding to probability, from white (zero probability) to saturated (high probability).
Title Suppressed Due to Excessive Length
1
2 wall floor door ceiling lamp ceiling
exercise machine entrance dresser desk lamp curtain cuddly toy crosswalk counter chandelier chair ceiling lamp ceiling cabinet bulletin board box bottle boat bathtub bars balcony
shelves person floor curtain
extractor hood entrance door desk lamp cushion cubicle counter column chair ceiling lamp ceiling candle bulletin board bridge bowl bottle book balcony bag arcade
window wall floor ceiling lamp
person painting outlet mountain microwave machine large window floor deck chair curtain cupboard countertop cliff central reservation ceiling car building bleachers bench bed
window wall floor
handrail ground glass wall fountain fence faucet extractor hood door desk lamp cushion curtain cupboard cliff chest car bulletin board building bridge boat balustrade
shelves person floor curtain
court clock chair ceiling lamp ceiling car cabinet bulletin board building box bottle boat bleachers bench barrel bag awning arcade altarpiece airplane
wall floor door ceiling lamp ceiling
flowers floor flag deck chair curtain conveyor belt coffee maker cloud ceiling lamp ceiling car cabinet building bridge book boat bleachers awning armchair animal
window wall plant floor ceiling
cup cuddly toy counter column chair ceiling lamp ceiling can cabinet box book bleachers bench bed beam basket barrel armchair arcade apple
window wall floor ceiling bed
countertop counter cloud chimney chandelier chair ceiling lamp ceiling can cabinet bulletin board building bowl bottle book bell bed basket bag altar
window wall table floor ceiling lamp
countertop chest chandelier chair ceiling lamp ceiling can bulletin board bread box bowl bottle book billiard table bench bed bathtub basket bag awning
window wall floor
curtain curb column cloud chandelier chair ceiling lamp candle cabinet building brand name box bowl book bicycle bathtub basket balcony awning armchair
window wall floor ceiling
mirror microwave ground drawer door desk lamp curtain counter column chair ceiling lamp ceiling card building brand name bicycle balcony armchair apple alarm clock
window wall table
candle candies can cabinet building bucket bridge bread brand name box bowl book boat bell bed bars barrel balcony bag alarm clock
shelves person floor curtain
files field extractor hood elevator drawer curtain counter ceiling lamp ceiling car cabinet box bench bed bathtub basket balcony bag altarpiece air conditioning
water tree sky ground
elephant door desk lamp crosswalk counter clothes chair central reservation car candle bulletin board building box bottle bicycle bench bed basket awning airplane
tree sky plant grass
wall tree swimming pool sky shop window sheep sea pole plant pitch path mountain grass floe field deck chair
tree sky mountain building
door dome doll desk lamp curtain cupboard column coffee maker cloud clock ceiling caravan car cabinet building brand name bench basket awning air conditioning
window sky road mountain building
cloud central reservation ceiling card car cabinet building brand name box boat boards bench beam bathtub basket bars barbecue bag backpack altar
tree sky mountain building
person path mountain magazines land ground grass files field embankment elephant door dirt track crosswalk court chair building bucket barrel arcade
window tree sky mountain building
curtain cubicle crosswalk countertop counter city chair ceiling car cabinet building bucket box bottle boat boards bicycle bench awning armchair
tree sky grass building
floor fence elephant door dome curtain countertop counter column cloud chimney chair ceiling lamp car cabinet building brand name bottle blanket balcony
1
2
3
4
5
6
7
8
9
10 Source
AP (ours)
5
STF
Source
AP (ours)
STF
6
1
2 wall table floor ceiling
chandelier chair ceiling lamp ceiling car can bulletin board building bridge bread box bottle bookcase book boat bicycle basket bar bag armchair
wall floor door ceiling lamp ceiling
ground glass wall floor fence door cushion cross conveyor belt ceiling lamp ceiling car cabinet bulletin board building bridge box bench bars armchair air conditioning
tree sky road building
road plant path mountain land ground grass floor field fence door dirt track cupboard court car bulletin board building book boat barrel
tree sky road mountain
separation sea rug road river railing pole pipe person mountain machinery ledge ladder glass wall fountain door conveyor belt car cabinet bulletin board
window tree sidewalk road door building
column closet chair ceiling lamp ceiling car cabinet bulletin board building bucket bridge box boat bar ball balcony bag awning animal air conditioning
window tree sky grass building
pitch person path painting mountain grass flowers field fence faucet dummy door chair ceiling lamp car building bucket box book billboard
wall tray stove drawer ceiling
glass wall garage door frame floor file cabinet fence cutlery curtain counter cloud clock central reservation ceiling lamp ceiling car bread bicycle bench balcony bag
11
12
13
swimming pool sky sea river mountain fish
sky sea mountain
14 sky road mountain building
snowy ground snow sky sign separation sea sand road river refrigerator railing pole plant mountain glass wall field faucet embankment conveyor belt car
window wall plant floor
decoration curtain column closet chair ceiling lamp ceiling car can building box bottle book boat bench bed balcony bag awning armchair
window wall tree door building
dome desk lamp cutlery counter cloud chandelier chair ceiling lamp ceiling can cabinet bulletin board building bowl book bench bed basket bag awning
tree sky mountain building
field fence elephant dirt track decoration cow column chest car bulletin board bridge bookcase book boat bell bed bars balustrade armchair animal
tree sky grass
water tree sky sign sheep rock plant mountain land grass glass wall field embankment curtain conveyor belt ceiling building armchair
15
16
swimming pool sky sea river mountain floe beam
tree sky mountain
17 tree sky person grass building
tree toaster sky plant path leaf hay roll grass grapes deck chair cabinet boat boards bench animal
tree sky mountain building
gate flowers floor fence extractor hood cross clothes chest chair can cabinet bulletin board building bucket book boat bench bell bathtub bag
tree sky mountain building
toy tower teddy bear streetlight stage snow sky rock pillow piano person mountain floor floe embankment dirt track caravan candle building boat
sky mountain building
fence drawer door dome desk lamp cutlery cross column cloud central reservation ceiling lamp card car building boat bleachers bench bed bars animal
wall person floor
sky sink plant painting mirror jar handrail fluorescent tube floor field extractor hood decoration cow column ceiling lamp ceiling bulletin board building bridge balcony
sky person monitor grass ceiling
18
19
20 Source
AP (ours)
STF
Source
AP (ours)
tree sky machine grass ceiling
STF
Title Suppressed Due to Excessive Length
1
2 sky road car building
mirror ground flowers flag fence easel desk lamp deck chair curtain cupboard coffee maker chair ceiling lamp ceiling car building brand name box balcony awning
tree sky road building
sand road printer piano painting mirror mezzanine heater hat fence easel door dome cross car building brand name bleachers balcony armchair
tree sky road building
curtain cuddly toy clothes closet chest chair central reservation ceiling lamp ceiling car building brand name box bottle boat bleachers bed bag armchair animal
window wall floor ceiling
fluorescent tube floor flag fan dummy drawer dish curtain cup counter closet chair ceiling lamp ceiling bread box book beam basket balcony
wall plant floor door
floor file cabinet field fence door knob dirt track deck cubicle column chair ceiling lamp ceiling car bulletin board building bench bars barrel bar air conditioning
tree sky road person building
machine ice rink ground flag faucet extractor hood cup cross counter ceiling car building bridge brand name bleachers bench basket bag awning apple
tree sky grass building
tree streetlight steps sky sculpture screen rock pole plant person path hay roll ground grass grapes fence dirt track deck boat animal
tree sky mountain building
scaffolding rocking chair road projection screen poster piano person mountain mezzanine guitar garage door fish fence embankment dummy curtain building bread bottle bicycle
window tree sky building
chair ceiling lamp ceiling can cabinet bulletin board building bucket bridge box book bleachers bell bed bars bag awning animal altar air conditioning
tree sky ground building
tree sky plant person path mountain grass grandstand field fence elephant door cloud caravan building
window tree sky building
desk lamp curtain cupboard counter cloud chair ceiling lamp car cabinet building brand name box bowl boat basket balcony awning attic alarm clock air conditioning
window tree sky building
refrigerator projection screen pole plateau person mountain hat ground gate fountain floe fence door ceiling building boat boards blanket billiard table bed
sky sea mountain building
drawer door knob dish cupboard counter cloud chair ceiling lamp car cabinet bulletin board building box bowl book billiard table bench bed basket bag
tree sky road building
elephant drawer dome desk lamp cow chest ceiling lamp caravan car cabinet bulletin board building bread box boat bars bag awning armchair apple
21
22
23
24
25
26
27 Source
AP (ours)
7
STF
Source
AP (ours)
STF
Fig. 3. More results on SUN, and comparison with semantic texton forests (STF) [3]. For some images, STF assigned too many tags for a clear visualization, and so we limited the legends to 20 words (omitting any excess words; for visualization purposes only).
8
1
1
2 tree sky old church building
2
plant green flower
3
round red plate circle black
3 woman hair girl face eye
woman white square rectangle picture
woman man hair face black
4
silver money gray gold coin
sky man green grass flower
5
man graph chart blue
yellow white square round red
6
7
8
water sky sea ocean cloud
white tree light building
water tree lake grass flower
tower street sky mountain city
tree sun sky light
tree sky grass
tree house green grass
smile man hat face black
water tree sky mountain lake
woman water swim hair girl
tree sky picture people blue
music man hair eye black
white green drawing diagram
picture man hair black
Fig. 4. More results on ESP.
Title Suppressed Due to Excessive Length
1
2
9
3
1
tree sky range mountain house
water tree statue sky house
tree side road river mountain
2
tree people meadow helmet bush
tourist rock mountain front desert
sky sea road house bush
3
sweater sky man hill grass
window tree tower sky front
4
tourist garden front forest chair
5
tower street square night lamp
view sky house cloud city
6
sky people lake group front
sign road gravel front forest
water sky sea palm beach
sky road lookout landscape canyon
tree slope people mountain bush
sky rock people mountain desert
wall tourist stone front entrance
7
waterfall stage sky rock face
woman slope sky rock mountain
view slope mountain house building
8
tree sun sky horizon
tree sky mountain man front
tree sky rock man cloud
Fig. 5. More results on IAPR.
10
1
2
3
1
tree sky road mountain building
sky sea sand rock mountain
sky sea sand rock mountain
2
tree sky road mountain car
tree sky plant field building
tree sky river plant grass
3
tree sky river mountain building
tree sky road mountain building
tree sky mountain car building
4
sky road door building
window tree ground building
tree sky mountain grass
5
window sky road door building
tree sky sea mountain building
sky road building
(a) SUN (and LMO)
1
tree sky people man green
2
white sun red orange flower
smile girl eye ear black
round gold cross coin circle
tree sky rock green forest
tree star space sky night
(b) ESP
1
2
woman sky man house hat
tree sky hill cloud chair
sky sea sand dune cloud
tree sky plane building airport
sky people middle man desert
window wall table room lamp
(c) IAPR
Fig. 6. More failure cases on SUN, ESP and IAPR.
Title Suppressed Due to Excessive Length
11
References 1. Russell, B., Torralba, A., Murphy, K., Freeman, W.: Labelme: a database and web-based tool for image annotation. IJCV 77 (2008) 157–173 2. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR. (2010) 3485–3492 3. Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: CVPR. (2008) 4. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: SIGCHI. (2004) 5. Grubinger, M., Clough, P., M¨uller, H., Deselaers, T.: The iapr benchmark: A new evaluation resource for visual information systems. In: LREC. (2006) 13–23 6. Makadia, A., Pavlovic, V., Kumar, S.: Baselines for image annotation. IJCV 90 (2010)