The Library
In defense of scene graphs for image captioning
Tools
Tools
Tools
Nguyen, Kien, Tripathi, Subarna, Du, Bang, Guha, Tanaya and Truong, Dinh Quang (2022) In defense of scene graphs for image captioning. In: International Conference on Computer Vision (ICCV), 2021, Virtual conference, 11-17 Oct 2021. Published in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) ISBN 9781665428125. doi:10.1109/ICCV48922.2021.00144
|
PDF
WRAP-In-defense-scene-graphs-image-captioning-2021.pdf - Accepted Version - Requires a PDF viewer. Download (4Mb) | Preview |
Official URL: https://doi.org/10.1109/ICCV48922.2021.00144
Abstract
The mainstream image captioning models rely on Convolutional Neural Network (CNN) image features to generate captions via recurrent models. Recently, image scene graphs have been used to augment captioning models so as to leverage their structural semantics, such as object entities, relationships and attributes. Several studies have noted that the naive use of scene graphs from a black-box scene graph generator harms image captioning performance and that scene graph-based captioning models have to incur the overhead of explicit use of image features to generate decent captions. Addressing these challenges, we propose SG2Caps, a framework that utilizes only the scene graph labels for competitive image captioning performance. The basic idea is to close the semantic gap between the two scene graphs - one derived from the input image and the other from its caption. In order to achieve this, we leverage the spatial location of objects and the Human-Object-Interaction (HOI) labels as an additional HOI graph. SG2Caps outperforms existing scene graph-only captioning models by a large margin, indicating scene graphs as a promising representation for image captioning. Direct utilization of scene graph labels avoids expensive graph convolutions over high-dimensional CNN features resulting in 49% fewer trainable parameters. Our code is available at: https://github.com/Kien085/SG2Caps
| Item Type: | Conference Item (Paper) | ||||||
|---|---|---|---|---|---|---|---|
| Subjects: | T Technology > TA Engineering (General). Civil engineering (General) | ||||||
| Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||
| Library of Congress Subject Headings (LCSH): | Computer vision, Pattern recognition systems, Graph theory -- Data processing | ||||||
| Journal or Publication Title: | 2021 IEEE/CVF International Conference on Computer Vision (ICCV) | ||||||
| Publisher: | IEEE Computer Society | ||||||
| ISBN: | 9781665428125 | ||||||
| Official Date: | 28 February 2022 | ||||||
| Dates: |
|
||||||
| DOI: | 10.1109/ICCV48922.2021.00144 | ||||||
| Status: | Peer Reviewed | ||||||
| Publication Status: | Published | ||||||
| Reuse Statement (publisher, data, author rights): | © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. | ||||||
| Access rights to Published version: | Open Access (Creative Commons) | ||||||
| Date of first compliant deposit: | 23 August 2021 | ||||||
| Date of first compliant Open Access: | 13 December 2021 | ||||||
| Conference Paper Type: | Paper | ||||||
| Title of Event: | International Conference on Computer Vision (ICCV), 2021 | ||||||
| Type of Event: | Conference | ||||||
| Location of Event: | Virtual conference | ||||||
| Date(s) of Event: | 11-17 Oct 2021 | ||||||
| Related URLs: | |||||||
| Open Access Version: |
Request changes or add full text files to a record
Repository staff actions (login required)
 |
View Item |
Downloads
Downloads per month over past year
