Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning
원문 초록
The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only methods. Our work studies this question through a carefully controlled comparison of two approaches in terms of their ability to learn representations that generalize to downstream classification tasks. We find that when the pre-training dataset meets certain criteria -- it is sufficiently large and contains descriptive captions with low variability -- image-only methods do not match CLIP's transfer performance, even when they are trained with more image data. However, contrary to what one might expect, there are practical settings in which these criteria are not met, wherein added supervision through captions is actually detrimental. Motivated by our findings, we devise simple prescriptions to enable CLIP to better leverage the language information present in existing pre-training datasets.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
On Designing Diffusion Autoencoders for Efficient Generation and Representation Learning
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
Describe Anything Model for Visual Question Answering on Text-rich Images
Assay2Mol: large language model-based drug design using BioAssay context
Dual Dimensions Geometric Representation Learning Based Document Dewarping
Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion
토론
이 논문과 연결된 포럼 글이 아직 없습니다.