Revising Image-Text Retrieval via Multi-Modal Entailment
원문 초록
An outstanding image-text retrieval model depends on high-quality labeled data. While the builders of existing image-text retrieval datasets strive to ensure that the caption matches the linked image, they cannot prevent a caption from fitting other images. We observe that such a many-to-many matching phenomenon is quite common in the widely-used retrieval datasets, where one caption can describe up to 178 images. These large matching-lost data not only confuse the model in training but also weaken the evaluation accuracy. Inspired by visual and textual entailment tasks, we propose a multi-modal entailment classifier to determine whether a sentence is entailed by an image plus its linked captions. Subsequently, we revise the image-text retrieval datasets by adding these entailed captions as additional weak labels of an image and develop a universal variable learning rate strategy to teach a retrieval model to distinguish the entailed captions from other negative samples. In experiments, we manually annotate an entailment-corrected image-text retrieval dataset for evaluation. The results demonstrate that the proposed entailment classifier achieves about 78% accuracy and consistently improves the performance of image-text retrieval baselines.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Adding simple structure at inference improves Vision-Language Compositionality
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval
Global–Local Information Soft-Alignment for Cross-Modal Remote-Sensing Image–Text Retrieval
ESA: External Space Attention Aggregation for Image-Text Retrieval
GAFNet: A Global Fourier Self Attention Based Novel Network for multi-modal downstream tasks
Dynamic Modality Interaction Modeling for Image-Text Retrieval
토론
이 논문과 연결된 포럼 글이 아직 없습니다.