PyTorchKR Papers

“Visual Question Answering”

총 2,804편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2403.20213공식 코드 · Papers with Code

VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis

This paper develops a Versatile and Honest vision language Model (VHM) for remote sensing image analysis. VHM is built on a large-scale remote sensing image-text dataset with rich-content captions (VersaD), and an honest instruction dataset comprising both factual and deceptive questions (HnstD). Unlike prevailing remote sensing image-text datasets, in which image captions focus on a few prominent objects and their relationships, VersaD captions provide detailed information about image properties, object attributes, and the overall scene. This comprehensive captioning enables VHM to thoroughly understand remote sensing images and perform diverse remote sensing tasks. Moreover, different from existing remote sensing instruction datasets that only include factual questions, HnstD contains additional deceptive questions stemming from the non-existence of objects. This feature prevents VHM from producing affirmative answers to nonsense queries, thereby ensuring its honesty. In our experiments, VHM significantly outperforms various vision language models on common tasks of scene classification, visual question answering, and visual grounding. Additionally, VHM achieves competent performance on several unexplored tasks, such as building vectorizing, multi-label classification and honest question answering. We will release the code, data and model weights at https://github.com/opendatalab/VHM .

arXiv:2610.01637v1cs.CV

베트남어 시각적 질의응답을 위한 다층 융합 트랜스포머 기반 시각 및 텍스트 표현 융합

Fusing Visual and Textual Representations via Multi-layer Fusing Transformers for Vietnamese Visual Question Answering

최근 수십 년간 인공지능은 이미지를 이해하고 상호작용하는 데 있어 상당한 진전을 이루었습니다. 이러한 기술의 중요한 응용 분야 중 하나는 컴퓨터가 자연스러운 방식으로 이미지에 대한 질문을 이해하고 답하도록 요구하는 연구 분야인 시각적 질의응답(Visual Question Answering, VQA)입니다. 영어 대상의 VQA에 대한 광범위한 연구 및 개발에도 불구하고, 다른 언어, 특히 베트남어에 대해서는 유사한 노력이 거의 이루어지지 않았습니다. 이러한 격차는 베트남어 환경에서 VQA 기술의 발전을 위한 중대한 도전 과제이자 기회를 제공합니다. 이 격차를 해소함으로써 베트남어 VQA 분야는 인공지능 연구의 다양성을 풍부하게 할 뿐만 아니라 전 세계 베트남어 사용자를 대상으로 교육, 의료, 엔터테인먼트 등 다양한 영역에서 실용적인 응용을 가능하게 합니다. 따라서 베트남어 VQA 시스템의 탐험(exploration)과 개발은 컴퓨터 비전과 자연어 처리의 교차점에서 연구와 실용적 응용 모두를 발전시킬 수 있는 막대한 잠재력을 지니고 있습니다. 본 논문에서는 서로 다른 층(layer)의 이미지 및 텍스트가 가진 다중 모달리티 특징을 하나의 통합된 표현으로 결합하기 위해 교차 어텐션(cross attention) 모듈을 활용하는 다층 융합 트랜스포머(Multi-layer Fusing Transformer) 모델을 제안합니다. 제안하는 아키텍처를 통해 저수준부터 고수준까지의 정보를 추출할 수 있습니다. 상세한 실험과 제거 연구(ablation study)를 통해 본 모델은 베트남어 ViVQA 데이터셋에서 경쟁력 있는 베이스라인 대비 유망한 결과를 달성합니다.

arXiv:2609.34798v2cs.CL

InfiMed2: 문맥적 근거와 안정성 인식 감독을 통한 범용 의료 멀티모달 기반 모델

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

최근 의료 멀티모달 모델(multimodal model)은 더 큰 말뭉치, 더 넓은 모달리티 적용 범위, 강화된 추론 지향 학습(training)을 통해 발전해 왔으나, 지속 사전학습(continued pretraining, CPT) 및 사후 학습(post-training) 전반에 걸친 효과적인 데이터 설계는 여전히 어려운 과제로 남아 있습니다. 의료 데이터 소스는 구조, 세분성, 정보 밀도가 크게 다르며, 광범위한 지식 습득에서 후기 단계의 통합으로 학습이 진행됨에 따라 그 유용성도 변화합니다. 한편 사후 학습은 단답형 시각적 질문 답변(VQA)이 지배적인 경우가 많아 정보성이 높고 답변과 일치하는 설명을 위한 감독을 충분히 제공하지 못합니다. 본 논문에서는 단계 인식형 데이터 설계를 중심으로 구축된 4B 및 27B 규모의 범용 의료 멀티모달 기반 모델(foundation model) 패밀리인 InfiMed2를 소개합니다. 소스 맞춤형 처리를 통해 광범위한 임상 지식과 문맥이 풍부한 생물의학 시각적 근거를 결합한 556.8억(55.68B) 토큰(token) 규모의 말뭉치를 구축했습니다. 본 CPT 파이프라인은 먼저 비전 인코더를 적응시킨 다음 광범위한 의료 지식을 구축하고, 최종적으로 학습률 감쇠 단계에서 근거 중심의 데이터 혼합으로 전환합니다. 지도 미세조정(SFT)을 위해 답변 안정성, 답변 마스킹 재구성, 정확도 제약 선별을 활용하여 시각적 질문 답변 반응을 재생성함으로써 더 유익하고 답변 일관성을 갖춘 감독 신호를 생성합니다. 4B 모델은 검증 가능한 보상을 통한 강화학습(RLVR)으로 추가 최적화되었습니다. 5개 의료 멀티모달 벤치마크(benchmark) 전반에서 InfiMed2-4B는 RLVR 적용 후 66.73%의 평균 정확도를 달성하여 더 큰 규모의 Qwen3.5-9B를 능가했으며, InfiMed2-27B는 73.72%를 기록하여 평가된 오픈 가중치(open-weight) 모델 중 가장 높은 성능을 달성했습니다.

arXiv:2608.01660v1cs.CV

그라운드, 커버, 리파인: 긴 비디오 질의응답을 위한 증거 중심 프레임 선택

Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering

긴 비디오 질의응답은 제한된 시각 토큰(visual-token) 예산 하에서 수천 개의 프레임이 포함된 비디오로부터 희소하지만 중요한 증거를 식별하는 것을 요구합니다. 기존 방식들은 단일 패스로 질의 인지적 프레임을 선택하거나 타임스탬프가 지정된 텍스트를 검색 유도용으로만 의존하여 두 가지 주요 한계를 보입니다. 첫째, 선택된 프레임들이 국소적 관련성 피크 주변에 군집하는 경향이 있어 예산이 소진되면 누락된 증거를 복구할 수 없습니다. 둘째, 텍스트 증거와 시각적 증거의 정렬이 여전히 취약합니다. 본 논문에서는 고정 예산 프레임 선택을 공동 증거 큐레이션 문제로 다루는 학습(training)이 필요 없는 프레임워크인 GCR을 제안합니다. Ground(그라운드) 단계는 타임스탬프 텍스트를 시간적 이벤트로 변환하고 질의와 관련된 실제 프레임 앵커를 선택하며, 각 이벤트 텍스트를 시간적으로 정렬된 프레임 위에 렌더링합니다. Cover(커버) 단계는 상호보완적인 시각적 증거를 제공하기 위해 그라운딩된 이벤트에 직접적인 시각적 앵커를 보충하고, 다양한 맥락을 보존하고자 전역 최대 한계 관련성(maximal marginal relevance)을 적용합니다. Refine(리파인) 단계는 누락된 시간 영역을 다시 탐색하여 더 높은 증거 가치를 제공하는 경우에 한해 가장 취약한 수정 가능 맥락 프레임을 실제 프레임 메도이드(medoid)로 대체합니다. GCR은 시간 순서대로 정렬된 고정 개수의 프레임을 유지하며, VLM 학습이나 구조 변경을 요구하지 않습니다. 3개의 7B 백본과 8, 32, 64개의 프레임 예산에 대해 LongVideoBench 및 Video-MME에서 수행한 실험은 긴 비디오 질의응답 전반에서 일관된 성능 향상을 입증합니다. 7B LLaVA-OV 백본 및 32개 프레임 설정에서 GCR은 두 벤치마크에서 각각 64.25%와 62.15%를 달성하여 가장 강력한 재현 기준 모델들을 각각 2.54%p 및 1.93%p 능가합니다.

arXiv:2405.07163

Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI

Educational scholars have analyzed various image data acquired from teaching and learning situations, such as photos that shows classroom dynamics, students' drawings with regard to the learning content, textbook illustrations, etc. Unquestioningly, most qualitative analysis of and explanation on image data have been conducted by human researchers, without machine-based automation. It was partially because most image processing artificial intelligence models were not accessible to general educational scholars or explainable due to their complex deep neural network architecture. However, the recent development of Visual Question Answering (VQA) techniques is accomplishing usable visual language models, which receive from the user a question about the given image and returns an answer, both in natural language. Particularly, GPT-4V released by OpenAI, has wide opened the state-of-the-art visual langauge model service so that VQA could be used for a variety of purposes. However, VQA and GPT-4V have not yet been applied to educational studies much. In this position paper, we suggest that GPT-4V contributes to realizing VQA for education. By 'realizing' VQA, we denote two meanings: (1) GPT-4V realizes the utilization of VQA techniques by any educational scholars without technical/accessibility barrier, and (2) GPT-4V makes educational scholars realize the usefulness of VQA to educational research. Given these, this paper aims to introduce VQA for educational studies so that it provides a milestone for educational research methodology. In this paper, chapter II reviews the development of VQA techniques, which primes with the release of GPT-4V. Chapter III reviews the use of image analysis in educational studies. Chapter IV demonstrates how GPT-4V can be used for each research usage reviewed in Chapter III, with operating prompts provided. Finally, chapter V discusses the future implications.

arXiv:1801.09041

Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions

Visual Question Answering (VQA) has attracted attention from both computer vision and natural language processing communities. Most existing approaches adopt the pipeline of representing an image via pre-trained CNNs, and then using the uninterpretable CNN features in conjunction with the question to predict the answer. Although such end-to-end models might report promising performance, they rarely provide any insight, apart from the answer, into the VQA process. In this work, we propose to break up the end-to-end VQA into two steps: explaining and reasoning, in an attempt towards a more explainable VQA by shedding light on the intermediate results between these two steps. To that end, we first extract attributes and generate descriptions as explanations for an image using pre-trained attribute detectors and image captioning models, respectively. Next, a reasoning module utilizes these explanations in place of the image to infer an answer to the question. The advantages of such a breakdown include: (1) the attributes and captions can reflect what the system extracts from the image, thus can provide some explanations for the predicted answer; (2) these intermediate results can help us identify the inabilities of both the image understanding part and the answer inference part when the predicted answer is wrong. We conduct extensive experiments on a popular VQA dataset and dissect all results according to several measurements of the explanation quality. Our system achieves comparable performance with the state-of-the-art, yet with added benefits of explainability and the inherent ability to further improve with higher quality explanations.

arXiv:2311.06607공식 코드 · Papers with Code

Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabilities. Firstly, Monkey processes input images by dividing them into uniform patches, each matching the size (e.g., 448x448) used in the original training of the well-trained vision encoder. Equipped with individual adapter for each patch, Monkey can handle higher resolutions up to 1344x896 pixels, enabling the detailed capture of complex visual information. Secondly, it employs a multi-level description generation method, enriching the context for scene-object associations. This two-part strategy ensures more effective learning from generated data: the higher resolution allows for a more detailed capture of visuals, which in turn enhances the effectiveness of comprehensive descriptions. Extensive ablative results validate the effectiveness of our designs. Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats. Specially, in qualitative tests focused on dense text question answering, Monkey has exhibited encouraging results compared with GPT4V. Code is available at https://github.com/Yuliang-Liu/Monkey.

arXiv:2608.21796v1cs.CV

SAFE-G: 지식 기반 시각적 질의응답을 위한 구조 인식 충실 증거 유도 생성

SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering

지식 기반 시각적 질의응답(KB-VQA)은 시각 콘텐츠를 넘어 외부 지식 원천에 대한 추론이 필요한 질의에 답하는 것을 목표로 한다. 일반적으로 기존 방법들은 다중모달 특징을 융합하여 외부 정보를 검색한 뒤, 검색된 증거로부터 답을 도출하기 위해 다중모달 대규모 언어 모델(MLLM)을 활용한다. 그러나 이러한 방법들은 복잡한 맥락 내 구조적 연관성을 포착하여 노이즈를 효과적으로 걸러내는 데 어려움을 겪는 경우가 많다. 또한 추론 과정이 검색된 증거에 엄격하게 충실하게 유지되도록 보장하지 못하는 경우가 빈번하다. 이러한 과제를 해결하기 위해, 정밀한 증거 국소화와 신뢰할 수 있는 추론을 가능하게 하는 구조 인식 충실 증거 유도 생성(Structure-Aware Faithful Evidence-guided Generation) 프레임워크인 SAFE-G를 제안한다. 구체적으로, 먼저 시각 및 텍스트 모달리티를 융합한 거친 수준의 하이브리드 검색을 통해 후보 문서를 회수하고, 이어 구조적 의존성을 포착하여 노이즈를 걸러내고 정확한 증거를 특정하는 구조 인식의 세밀한 그래프 검색을 수행한다. 또한 증거에 기반한 보상을 갖는 강화학습(RL) 전략을 도입하여, 선택된 증거가 올바를 때만 정답에 공을 할당한다. 이 엄격한 정렬(alignment) 제약은 모델이 검색된 맥락에 응답을 고정하도록 강제하며, 다중모달 특징을 통해 증거를 찾아내고 충실한 추론을 수행하는 능력을 효과적으로 향상시킨다. Encyclopedic-VQA 및 InfoSeek 벤치마크에 대한 광범위한 실험에서 SAFE-G가 기존 방법들을 각각 8.9% 및 3.5% 마진만큼 능가함을 보여주며, 전반적인 추론 정확도를 크게 향상시킨다. 소스 코드는 https://github.com/MINE-USTC/SAFE-G 에서 공개되어 있다.

arXiv:2307.03130

VisKoP: Visual Knowledge oriented Programming for Interactive Knowledge Base Question Answering

We present Visual Knowledge oriented Programming platform (VisKoP), a knowledge base question answering (KBQA) system that integrates human into the loop to edit and debug the knowledge base (KB) queries. VisKoP not only provides a neural program induction module, which converts natural language questions into knowledge oriented program language (KoPL), but also maps KoPL programs into graphical elements. KoPL programs can be edited with simple graphical operators, such as dragging to add knowledge operators and slot filling to designate operator arguments. Moreover, VisKoP provides auto-completion for its knowledge base schema and users can easily debug the KoPL program by checking its intermediate results. To facilitate the practical KBQA on a million-entity-level KB, we design a highly efficient KoPL execution engine for the back-end. Experiment results show that VisKoP is highly efficient and user interaction can fix a large portion of wrong KoPL programs to acquire the correct answer. The VisKoP online demo https://demoviskop.xlore.cn (Stable release of this paper) and https://viskop.xlore.cn (Beta release with new features), highly efficient KoPL engine https://pypi.org/project/kopl-engine, and screencast video https://youtu.be/zAbJtxFPTXo are now publicly available.

arXiv:2307.11986공식 코드 · Papers with Code

Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering

To contribute to automating the medical vision-language model, we propose a novel Chest-Xray Difference Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions on both diseases and, more importantly, the differences between them. This is consistent with the radiologist's diagnosis practice that compares the current image with the reference before concluding the report. We collect a new dataset, namely MIMIC-Diff-VQA, including 700,703 QA pairs from 164,324 pairs of main and reference images. Compared to existing medical VQA datasets, our questions are tailored to the Assessment-Diagnosis-Intervention-Evaluation treatment procedure used by clinical professionals. Meanwhile, we also propose a novel expert knowledge-aware graph representation learning model to address this task. The proposed baseline model leverages expert knowledge such as anatomical structure prior, semantic, and spatial knowledge to construct a multi-relationship graph, representing the image differences between two images for the image difference VQA task. The dataset and code can be found at https://github.com/Holipori/MIMIC-Diff-VQA. We believe this work would further push forward the medical vision language model.

arXiv:2108.08965

Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling

As an important task in multimodal context understanding, Text-VQA (Visual Question Answering) aims at question answering through reading text information in images. It differentiates from the original VQA task as Text-VQA requires large amounts of scene-text relationship understanding, in addition to the cross-modal grounding capability. In this paper, we propose Localize, Group, and Select (LOGOS), a novel model which attempts to tackle this problem from multiple aspects. LOGOS leverages two grounding tasks to better localize the key information of the image, utilizes scene text clustering to group individual OCR tokens, and learns to select the best answer from different sources of OCR (Optical Character Recognition) texts. Experiments show that LOGOS outperforms previous state-of-the-art methods on two Text-VQA benchmarks without using additional OCR annotation data. Ablation studies and analysis demonstrate the capability of LOGOS to bridge different modalities and better understand scene text.

arXiv:2109.10246공식 코드 · Papers with Code

Does Vision-and-Language Pretraining Improve Lexical Grounding?

Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pretraining for two recent VL models using a suite of analyses (clustering, probing, and performance on a commonsense question answering task) in a language-only setting. We find that the multimodal models fail to significantly outperform the text-only variants, suggesting that future work is required if multimodal pretraining is to be pursued as a means of improving NLP in general.

arXiv:2412.08564

Can We Generate Visual Programs Without Prompting LLMs?

Visual programming prompts LLMs (large language mod-els) to generate executable code for visual tasks like visual question answering (VQA). Prompt-based methods are difficult to improve while also being unreliable and costly in both time and money. Our goal is to develop an efficient visual programming system without 1) using prompt-based LLMs at inference time and 2) a large set of program and answer annotations. We develop a synthetic data augmentation approach and alternative program generation method based on decoupling programs into higher-level skills called templates and the corresponding arguments. Our results show that with data augmentation, prompt-free smaller LLMs ($\approx$ 1B parameters) are competitive with state-of-the art models with the added benefit of much faster inference

arXiv:1901.01492

What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning

Long-term planning poses a major difficulty to many reinforcement learning algorithms. This problem becomes even more pronounced in dynamic visual environments. In this work we propose Hierarchical Planning and Reinforcement Learning (HIP-RL), a method for merging the benefits and capabilities of Symbolic Planning with the learning abilities of Deep Reinforcement Learning. We apply HIPRL to the complex visual tasks of interactive question answering and visual semantic planning and achieve state-of-the-art results on three challenging datasets all while taking fewer steps at test time and training in fewer iterations. Sample results can be found at youtu.be/0TtWJ_0mPfI

arXiv:2301.09045공식 코드 · Papers with Code

Champion Solution for the WSDM2023 Toloka VQA Challenge

In this report, we present our champion solution to the WSDM2023 Toloka Visual Question Answering (VQA) Challenge. Different from the common VQA and visual grounding (VG) tasks, this challenge involves a more complex scenario, i.e. inferring and locating the object implicitly specified by the given interrogative question. For this task, we leverage ViT-Adapter, a pre-training-free adapter network, to adapt multi-modal pre-trained Uni-Perceiver for better cross-modal localization. Our method ranks first on the leaderboard, achieving 77.5 and 76.347 IoU on public and private test sets, respectively. It shows that ViT-Adapter is also an effective paradigm for adapting the unified perception model to vision-language downstream tasks. Code and models will be released at https://github.com/czczup/ViT-Adapter/tree/main/wsdm2023.

arXiv:2603.13800v2cs.CV

의료 진단을 넘어서: 의료 다중모달 대규모 언어 모델(MLLM)은 공간에서 어떻게 사고하는가

Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

시각적 공간 지능은 의료 영상 해석에 핵심적이지만, 3D 영상을 위한 다중모달 대규모 언어 모델(MLLM)에서는 여전히 거의 탐구되지 않았다. 이 격차는 기본 라벨을 넘어선 구조화된 3D 공간 주석을 포함하는 데이터셋의 체계적 부재로 인해 지속된다. 본 연구에서는 체적 추정 및 바운딩 박스(bounding box) 추출과 같은 계산 도구를 다중 에이전트 협업과 전문 방사선과 의사 검증과 함께 조율하여 공간적 시각 질의응답(VQA) 데이터를 자율적으로 합성하는 에이전트 파이프라인을 제안한다. 또한 의료 MLLM의 3D 공간 지능을 평가하기 위한 최초의 포괄적 벤치마크인 SpatialMed를 제시하며, 이는 여러 장기와 종양 유형에 걸쳐 31,253개의 질의-응답 쌍으로 구성된다. 24개의 최신 MLLM에 대한 평가와 광범위한 분석을 통해, 현재 모델들은 의료 영상에 대한 견고한 공간 추론 능력이 부족함을 밝힌다.

arXiv:2212.01639공식 코드 · Papers with Code

Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests

Different types of mental rotation tests have been used extensively in psychology to understand human visual reasoning and perception. Understanding what an object or visual scene would look like from another viewpoint is a challenging problem that is made even harder if it must be performed from a single image. We explore a controlled setting whereby questions are posed about the properties of a scene if that scene was observed from another viewpoint. To do this we have created a new version of the CLEVR dataset that we call CLEVR Mental Rotation Tests (CLEVR-MRT). Using CLEVR-MRT we examine standard methods, show how they fall short, then explore novel neural architectures that involve inferring volumetric representations of a scene. These volumes can be manipulated via camera-conditioned transformations to answer the question. We examine the efficacy of different model variants through rigorous ablations and demonstrate the efficacy of volumetric representations.

arXiv:2406.19170공식 코드 · Papers with Code

The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systems

We examine how users perceive the limitations of an AI system when it encounters a task that it cannot perform perfectly and whether providing explanations alongside its answers aids users in constructing an appropriate mental model of the system's capabilities and limitations. We employ a visual question answer and explanation task where we control the AI system's limitations by manipulating the visual inputs: during inference, the system either processes full-color or grayscale images. Our goal is to determine whether participants can perceive the limitations of the system. We hypothesize that explanations will make limited AI capabilities more transparent to users. However, our results show that explanations do not have this effect. Instead of allowing users to more accurately assess the limitations of the AI system, explanations generally increase users' perceptions of the system's competence - regardless of its actual performance.

arXiv:2209.10320공식 코드 · Papers with Code

Continual VQA for Disaster Response Systems

Visual Question Answering (VQA) is a multi-modal task that involves answering questions from an input image, semantically understanding the contents of the image and answering it in natural language. Using VQA for disaster management is an important line of research due to the scope of problems that are answered by the VQA system. However, the main challenge is the delay caused by the generation of labels in the assessment of the affected areas. To tackle this, we deployed pre-trained CLIP model, which is trained on visual-image pairs. however, we empirically see that the model has poor zero-shot performance. Thus, we instead use pre-trained embeddings of text and image from this model for our supervised training and surpass previous state-of-the-art results on the FloodNet dataset. We expand this to a continual setting, which is a more real-life scenario. We tackle the problem of catastrophic forgetting using various experience replay methods. Our training runs are available at: https://wandb.ai/compyle/continual_vqa_final. Our code is available at https://github.com/AdityaKane2001/continual_vqa.

arXiv:1811.10582공식 코드 · Papers with Code

Visual Entailment Task for Visually-Grounded Language Learning

We introduce a new inference task - Visual Entailment (VE) - which differs from traditional Textual Entailment (TE) tasks whereby a premise is defined by an image, rather than a natural language sentence as in TE tasks. A novel dataset SNLI-VE (publicly available at https://github.com/necla-ml/SNLI-VE) is proposed for VE tasks based on the Stanford Natural Language Inference corpus and Flickr30k. We introduce a differentiable architecture called the Explainable Visual Entailment model (EVE) to tackle the VE problem. EVE and several other state-of-the-art visual question answering (VQA) based models are evaluated on the SNLI-VE dataset, facilitating grounded language understanding and providing insights on how modern VQA based models perform.