Visible to the public Reasoning About Fine-Grained Attribute Phrases Using Reference Games

TitleReasoning About Fine-Grained Attribute Phrases Using Reference Games
Publication TypeConference Paper
Year of Publication2017
AuthorsSu, J. C., Wu, C., Jiang, H., Maji, S.
Conference Name2017 IEEE International Conference on Computer Vision (ICCV)
Keywordscompositionality, Computer vision, fine-grained attribute phrases, fine-grained visual differences, Games, Image color analysis, image representation, image retrieval, inference mechanisms, instances, learning (artificial intelligence), listener, Nose, Pragmatics, pubcrawl, reasoning, reference games, semantic attributes, Semantics, speaker, visual descriptions, visualization

We present a framework for learning to describe finegrained visual differences between instances using attribute phrases. Attribute phrases capture distinguishing aspects of an object (e.g., "propeller on the nose" or "door near the wing" for airplanes) in a compositional manner. Instances within a category can be described by a set of these phrases and collectively they span the space of semantic attributes for a category. We collect a large dataset of such phrases by asking annotators to describe several visual differences between a pair of instances within a category. We then learn to describe and ground these phrases to images in the context of a reference game between a speaker and a listener. The goal of a speaker is to describe attributes of an image that allows the listener to correctly identify it within a pair. Data collected in a pairwise manner improves the ability of the speaker to generate, and the ability of the listener to interpret visual descriptions. Moreover, due to the compositionality of attribute phrases, the trained listeners can interpret descriptions not seen during training for image retrieval, and the speakers can generate attribute-based explanations for differences between previously unseen categories. We also show that embedding an image into the semantic space of attribute phrases derived from listeners offers 20% improvement in accuracy over existing attributebased representations on the FGVC-aircraft dataset.

Citation Keysu_reasoning_2017