Chain of Thought Prompt Tuning in Vision Language Models
원문 초록
Language-Image Pre-training has demonstrated promising results on zero-shot and few-shot downstream tasks by prompting visual models with natural language prompts. However, most recent studies only use a single prompt for tuning, neglecting the inherent step-to-step cognitive reasoning process that humans conduct in complex task settings, for example, when processing images from unfamiliar domains. Chain of Thought is a simple and effective approximation to human reasoning process and has been proven useful for natural language processing (NLP) tasks. Based on this cognitive intuition, we believe that conducting effective reasoning is also an important problem in visual tasks, and a chain of thought could be a solution to this problem. In this work, we propose a novel chain of thought prompt tuning for vision-language modeling. Extensive experiments show that our method not only generalizes better in image classification tasks, has greater transferability beyond a single dataset, and has stronger domain generalization performance, but also performs much better in imagetext retrieval and visual question answering, which require more reasoning capabilities. We are the first to successfully adapt chain-of-thought prompting that combines visual and textual embeddings. We will release our codes
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing
Describe Anything Model for Visual Question Answering on Text-rich Images
Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility
OpenTable-R1: A Reinforcement Learning Augmented Tool Agent for Open-Domain Table Question Answering
Large Language Model Agent for Modular Task Execution in Drug Discovery
Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions: A Case Study on Jordanian Laws
토론
이 논문과 연결된 포럼 글이 아직 없습니다.