Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval
원문 초록
Cross-modal retrieval between visual data and natural language description remains a long-standing challenge in multimedia. While recent image-text retrieval methods offer great promise by learning deep representations aligned across modalities, most of these methods are plagued by the issue of training with small-scale datasets covering a limited number of images with ground-truth sentences. Moreover, it is extremely expensive to create a larger dataset by annotating millions of images with sentences and may lead to a biased model. Inspired by the recent success of webly supervised learning in deep neural networks, we capitalize on readily-available web images with noisy annotations to learn robust image-text joint representation. Specifically, our main idea is to leverage web images and corresponding tags, along with fully annotated datasets, in training for learning the visual-semantic joint embedding. We propose a two-stage approach for the task that can augment a typical supervised pair-wise ranking loss based formulation with weakly-annotated web images to learn a more robust visual-semantic embedding. Experiments on two standard benchmark datasets demonstrate that our method achieves a significant performance gain in image-text retrieval compared to state-of-the-art approaches.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval
Global–Local Information Soft-Alignment for Cross-Modal Remote-Sensing Image–Text Retrieval
Dynamic Modality Interaction Modeling for Image-Text Retrieval
Adding simple structure at inference improves Vision-Language Compositionality
Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation
ESA: External Space Attention Aggregation for Image-Text Retrieval
토론
이 논문과 연결된 포럼 글이 아직 없습니다.