LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval
원문 초록
Dual encoders and cross encoders have been widely used for image-text retrieval. Between the two, the dual encoder encodes the image and text independently followed by a dot product, while the cross encoder jointly feeds image and text as the input and performs dense multi-modal fusion. These two architectures are typically modeled separately without interaction. In this work, we propose LoopITR, which combines them in the same network for joint learning. Specifically, we let the dual encoder provide hard negatives to the cross encoder, and use the more discriminative cross encoder to distill its predictions back to the dual encoder. Both steps are efficiently performed together in the same model. Our work centers on empirical analyses of this combined architecture, putting the main focus on the design of the distillation objective. Our experimental results highlight the benefits of training the two encoders in the same network, and demonstrate that distillation can be quite effective with just a few hard negative examples. Experiments on two standard datasets (Flickr30K and COCO) show our approach achieves state-of-the-art dual encoder performance when compared with approaches using a similar amount of data.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Adding simple structure at inference improves Vision-Language Compositionality
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval
Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation
Global–Local Information Soft-Alignment for Cross-Modal Remote-Sensing Image–Text Retrieval
ESA: External Space Attention Aggregation for Image-Text Retrieval
GAFNet: A Global Fourier Self Attention Based Novel Network for multi-modal downstream tasks
토론
이 논문과 연결된 포럼 글이 아직 없습니다.