Visual Entity Linking: A Preliminary Study

Summary, in English

In this paper, we describe a system that jointly extracts

entities appearing in images and mentioned in their ac-

companying captions. As input, the entity linking pro-

gram takes a segmented image together with its cap-

tion. It consists of a sequence of processing steps: part-

of-speech tagging, dependency parsing, and coreference

resolution that enables us to identify the entities as well

as possible textual relations from the captions. The pro-

gram uses the image regions labelled with a set of pre-

defined categories and computes WordNet similarities

between these labels and the entity names. Finally, the

program links the entities it detected across the text and

the images. We applied our system on the Segmented

and Annotated IAPR TC-12 dataset that we enriched

with entity annotations and we obtained a correct as-

signment rate of 55.48%

• Författare: Rebecka Weegar; Linus Hammarlund; Agnes Tegen; Magnus Oskarsson; Karl Åström; Pierre Nugues
• Avdelning/ar: Matematik LTH; Matematikcentrum; Institutionen för datavetenskap; ELLIIT: the Linköping-Lund initiative on IT and mobile communication; eSSENCE: The e-Science Collaboration; Robotik och Semantiska System
• Publiceringsår: 2014
• Språk: Engelska
• Sidor: 46-49
• Publikation/Tidskrift/Serie: Cognitive computing for augmented human intelligence, papers presented at the Twenty-Eighth AAAI Conference on Artificial Intelligence
• Fulltext: - 782 kB; Download statistics
• Dokumenttyp: Konferensbidrag
• Förlag: AAAI
• Ämne: Computer Science; Mathematics
• Conference name: AAAI 2014 Workshop on Cognitive Computing for Augmented Human Intelligence
• Conference date: 2014-07-27
• Status: Published
• ISBN/ISSN/Övrigt: ISBN: 978-1-57735-664-6