Lao, M.; Guo, Y.; Wang, H.; Zhang, X. Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering. Preprints2018, 2018040313. https://doi.org/10.20944/preprints201804.0313.v1
APA Style
Lao, M., Guo, Y., Wang, H., & Zhang, X. (2018). Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering. Preprints. https://doi.org/10.20944/preprints201804.0313.v1
Chicago/Turabian Style
Lao, M., Hui Wang and Xin Zhang. 2018 "Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering" Preprints. https://doi.org/10.20944/preprints201804.0313.v1
Abstract
Visual question answering (VQA) is receiving increasing attention from researchers in both the computer vision and natural language processing fields. There are two key components in the VQA task: feature extraction and multi-modal fusion. For feature extraction, we introduce a novel co-attention scheme by combining Sentence-guide Word Attention (SWA) and Question-guide Image Attention (QIA) in a unified framework. To be specific, the textual attention SWA relies on the semantics of the whole question sentence to calculate contributions of different question words for text representation. For the multi-modal fusion, we propose a “Cross-modal Multistep Fusion (CMF)” network to generate multistep features and achieve multiple interactions for two modalities, rather than focusing on modeling complex interactions between two modals like most current feature fusion methods. To avoid the linear increase of the computational cost, we share the parameters for each step in the CMF. Extensive experiments demonstrate that the proposed method can achieve competitive or better performance than the state-of-the-art.
Computer Science and Mathematics, Computer Networks and Communications
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
