Visible to the public Cross-Modal Knowledge Transfer: Improving the Word Embedding of Apple by Looking at Oranges

TitleCross-Modal Knowledge Transfer: Improving the Word Embedding of Apple by Looking at Oranges
Publication TypeConference Paper
Year of Publication2017
AuthorsBoth, Fabian, Thoma, Steffen, Rettinger, Achim
Conference NameProceedings of the Knowledge Capture Conference
Conference LocationNew York, NY, USA
ISBN Number978-1-4503-5553-7
Keywordsanalogical transfer, analogies, Human Behavior, human factors, knowledge transfer, Multi-Modality, pubcrawl, Transference, Word Similarity

Capturing knowledge via learned latent vector representations of words, images and knowledge graph (KG) entities has shown state-of-the-art performance in computer vision, computational linguistics and KG tasks. Recent results demonstrate that the learning of such representations across modalities can be beneficial, since each modality captures complementary information. However, those approaches are limited to concepts with cross-modal alignments in the training data which are only available for just a few concepts. Especially for visual objects exist far fewer embeddings than for words or KG entities. We investigate whether a word embedding (e.g., for "apple") can still capture information from other modalities even if there is no matching concept within the other modalities (i.e., no images or KG entities of apples but of oranges as pictured in the title analogy). The empirical results of our knowledge transfer approach demonstrate that word embeddings do benefit from extrapolating information across modalities even for concepts that are not represented in the other modalities. Interestingly, this applies most to concrete concepts (e.g., dragonfly) while abstract concepts (e.g., animal) benefit most if aligned concepts are available in the other modalities.

Citation Keyboth_cross-modal_2017