PyTorchKR Papers

“Visual Question Answering”

총 2,807편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2504.02227공식 코드 · Papers with Code

VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence

Social Intelligence Queries (Social-IQ) serve as the primary multimodal benchmark for evaluating a model's social intelligence level. While impressive multiple-choice question(MCQ) accuracy is achieved by current solutions, increasing evidence shows that they are largely, and in some cases entirely, dependent on language modality, overlooking visual context. Additionally, the closed-set nature further prevents the exploration of whether and to what extent the reasoning path behind selection is correct. To address these limitations, we propose the Visually Explainable and Grounded Artificial Social Intelligence (VEGAS) model. As a generative multimodal model, VEGAS leverages open-ended answering to provide explainable responses, which enhances the clarity and evaluation of reasoning paths. To enable visually grounded answering, we propose a novel sampling strategy to provide the model with more relevant visual frames. We then enhance the model's interpretation of these frames through Generalist Instruction Fine-Tuning (GIFT), which aims to: i) learn multimodal-language transformations for fundamental emotional social traits, and ii) establish multimodal joint reasoning capabilities. Extensive experiments, comprising modality ablation, open-ended assessments, and supervised MCQ evaluations, consistently show that VEGAS effectively utilizes visual information in reasoning to produce correct and also credible answers. We expect this work to of fer a new perspective on Social-IQ and advance the development of human-like social AI.

arXiv:2608.02833v1cs.CV

CURV: 커리큘럼 시각적 그라운딩 추론을 통한 차트 이해 능력 향상

CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning

차트 질의응답(CQA)은 멀티모달 대규모 언어 모델(MLLM)이 시각적 이해와 논리적 추론을 통합할 것을 요구하지만, 현재 모델들은 정확한 시각적 그라운딩(visual grounding)과 일관된 추론 체인을 형성하는 데 어려움을 겪고 있습니다. 외재적 사고의 연쇄(Chain-of-Thought) 프롬프트와 시각적 단서가 성능을 크게 향상시키기는 하지만, 현재 MLLM은 내재적인 시각적 그라운딩 추론 능력이 부족하여 부정확한 인식과 시각적 증거와 단절된 추론을 초래합니다. 이러한 한계를 해결하기 위해, 본 논문에서는 CQA를 각 단계마다 공간적 어텐션(attention) 집중을 통해 논리적 추론과 동적 시각적 그라운딩을 조율하는 다단계 시각적 그라운딩 추론으로 재구성함으로써 내재적 시각 추론 능력을 개발하는 커리큘럼 학습(curriculum learning) 프레임워크인 CURV를 제안합니다. 모델(model) 학습을 돕기 위해, 다양한 차트 유형과 추론 패턴에 걸쳐 확장 가능한 합성 생성을 지원하는 3단계 커리큘럼 데이터셋(dataset)인 CCQA를 추가로 소개합니다. 본 커리큘럼은 기본적인 단일 연산 추론부터 복잡한 다중 차트 구성 작업까지 체계적으로 발전합니다. 실험을 통해 CURV가 베이스라인 대비 최대 $\uparrow20.50\%$의 성능 향상을 달성하고, 실제 벤치마크(최대 $\uparrow12.30\%$) 및 도메인 외(out-of-domain) 멀티모달 추론 작업(최대 $\uparrow10.20\%$)으로의 일반화가 가능함을 보여주며, 차트 이해 능력 향상을 위해 동적 그라운딩을 갖춘 시각 추론을 내재화하는 것의 유효성을 입증합니다. 코드는 https://xhguo7.github.io/CURV/ 에서 확인할 수 있습니다.

arXiv:2608.08612v1cs.CV

REVEAL: 장시간 비디오 질의응답을 위한 루브릭 기반 명시적 증거 충분성 검증 에이전트

REVEAL: A Rubric-Guided Agent for Explicit Evidence Sufficiency Verificationin Long-Video Question Answering

최근 검색-증강(retrieval-augmented) 및 메모리-증강(memory-augmented) 방법이 장시간 비디오 질의응답(long-video question answering)을 위한 두 가지 유망한 패러다임으로 부상했다. 그러나 기존 방법은 대개 고정된 길이의 시간 청킹(예: 10초)과 정적 오프라인 메모리 뱅크에 의존하여, 일관된 연속 사건을 단편화할 뿐 아니라 실시간 추론 중 적응에도 실패한다. 더 나아가 다중 스케일 요약이든 멀티모달 지식 그래프든, 현재 접근법은 검색 관련성을 우선시하면서 증거 충분성(evidence sufficiency)은 간과하는 경우가 많아, 핵심적인 시간적·인과적·세밀한 행동 증거가 아직 부족한데도 의미적으로 관련된 단서만 검색되면 답변을 멈춘다. 이러한 과제를 해결하기 위해 우리는 루브릭 기반 에이전트 프레임워크인 REVEAL을 제안한다. 기반으로서, 시각적으로 일관된 인접 프레임을 자연스러운 사건 단위로 묶는 적응형 시각 유사도 기반 전처리 파이프라인을 도입하여, 오프라인-온라인 비디오 메모리를 구성한다—오프라인에서는 전역 비디오 맥락을 포착하고 온라인에서는 질문 조건화(question-conditioned) 메모리를 동적으로 유지한다. 이 구조화된 메모리 위에서 REVEAL은 자동 구축된 루브릭 라이브러리를 사용해 검색된 증거가 충분성 기준을 만족하는지 명시적으로 검증하고, 검증 실패 시 누락된 단서를 특정하며, 보완 정보를 위한 표적 재검색(targeted re-retrieval)을 지시한다. 추가 학습 없이 REVEAL은 광범위한 실험에서 폐쇄형·오픈소스 최신(state-of-the-art) 방법 모두를 일관되게 능가한다. 이 결과는 의미적 관련성에서 멈추기보다 증거 충분성을 명시적으로 검증하는 것이 이전 방법이 놓친 결정적 단서를 검색하고 더 신뢰할 수 있는 장시간 비디오 추론을 가능하게 함을 보여준다.

arXiv:2406.12384공식 코드 · Papers with Code

VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

We introduce a new benchmark designed to advance the development of general-purpose, large-scale vision-language models for remote sensing images. Although several vision-language datasets in remote sensing have been proposed to pursue this goal, existing datasets are typically tailored to single tasks, lack detailed object information, or suffer from inadequate quality control. Exploring these improvement opportunities, we present a Versatile vision-language Benchmark for Remote Sensing image understanding, termed VRSBench. This benchmark comprises 29,614 images, with 29,614 human-verified detailed captions, 52,472 object references, and 123,221 question-answer pairs. It facilitates the training and evaluation of vision-language models across a broad spectrum of remote sensing image understanding tasks. We further evaluated state-of-the-art models on this benchmark for three vision-language tasks: image captioning, visual grounding, and visual question answering. Our work aims to significantly contribute to the development of advanced vision-language models in the field of remote sensing. The data and code can be accessed at https://github.com/lx709/VRSBench.

arXiv:2607.27637v1cs.CV

MMOOC: 멀티모달 대규모 언어 모델의 문맥 이탈 평가를 위한 포괄적 벤치마크

MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models

멀티모달 대규모 언어 모델(MLLM)은 광범위한 비전-언어(vision-language) 태스크에서 뛰어난 성능을 달성했지만, 불완전하거나 변화된 문맥에서는 종종 실패합니다. 신뢰할 수 있는 MLLM은 주체 수준의 문맥 변화가 있는 진정한 문맥 이탈(out-of-context, OOC) 질문은 거절해야 하며, 비주체 문맥 변화가 있는 변화된 문맥 내(Shifted in-context, Shifted IC) 질문에는 여전히 답변할 수 있어야 합니다. 기존 벤치마크는 주로 OOC 또는 시각적으로 답변 불가능한 질문을 대상으로 하지만, 답변 가능한 Shifted IC 사례를 간과하고 제한적인 OOC 변화만을 다룹니다. 이러한 공백을 메우기 위해, 본 연구에서는 MLLM의 거절 및 강건한 답변 능력을 평가하기 위한 대규모 벤치마크인 MMOOC를 제시합니다. MMOOC는 MLLM 기반 필터링과 인간 검증을 통해 데이터 품질을 보장하며 세 가지 질문 형식, 여덟 가지 변화 유형 및 여섯 가지 시각적 시나리오에 걸쳐 답변 가능한 Shifted IC 사례와 답변 불가능한 OOC 사례를 포함하는 4만 1천 개 이상의 이미지-질문 쌍으로 구성됩니다. 모델 응답은 정확도(Accuracy)와 거절률(Refusal Rate)을 통해 평가하며, 모델 추론의 정확성을 평가하기 위해 LLM-as-a-judge 지표를 추가로 도입합니다. 다양한 MLLM을 대상으로 한 실험 결과, 현재 모델들은 변화된 문맥에서 답변 가능성과 거절 간의 균형을 유지하는 데 여전히 어려움을 겪고 있음을 보여줍니다. 나아가 주요 실패 패턴을 분석하고 사후 학습(post-training)을 통해 강건성을 향상시킬 수 있음을 입증합니다. MMOOC는 공개적으로 제공될 예정입니다.

arXiv:2306.09265공식 코드 · Papers with Code

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building a LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of $8$ representative LVLMs such as InstructBLIP and MiniGPT-4, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates $6$ categories of multimodal capabilities of LVLMs such as visual question answering and embodied artificial intelligence on $47$ standard text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study reveals several innovative findings. First, instruction-tuned LVLM with massive in-domain data such as InstructBLIP heavily overfits many existing tasks, generalizing poorly in the open-world scenario. Second, instruction-tuned LVLM with moderate instruction-following data may result in object hallucination issues (i.e., generate objects that are inconsistent with target images in the descriptions). It either makes the current evaluation metric such as CIDEr for image captioning ineffective or generates wrong answers. Third, employing a multi-turn reasoning evaluation framework can mitigate the issue of object hallucination, shedding light on developing an effective pipeline for LVLM evaluation. The findings provide a foundational framework for the conception and assessment of innovative strategies aimed at enhancing zero-shot multimodal techniques. Our LVLM-eHub will be available at https://github.com/OpenGVLab/Multi-Modality-Arena

arXiv:2312.03587공식 코드 · Papers with Code

Language-Informed Visual Concept Learning

Our understanding of the visual world is centered around various concept axes, characterizing different aspects of visual entities. While different concept axes can be easily specified by language, e.g. color, the exact visual nuances along each axis often exceed the limitations of linguistic articulations, e.g. a particular style of painting. In this work, our goal is to learn a language-informed visual concept representation, by simply distilling large pre-trained vision-language models. Specifically, we train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes, with an objective of reproducing the input image through a pre-trained Text-to-Image (T2I) model. To encourage better disentanglement of different concept encoders, we anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts. With a lightweight test-time finetuning procedure, it can also generalize to novel concepts unseen at training.

arXiv:2406.20092공식 코드 · Papers with Code

Efficient Large Multi-modal Models via Visual Context Compression

While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

arXiv:1808.04446공식 코드 · Papers with Code

Visual Reasoning with Multi-hop Feature Modulation

Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach is to condition image-based convolutional network computation on language via Feature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and shifting. We propose to generate the parameters of FiLM layers going up the hierarchy of a convolutional network in a multi-hop fashion rather than all at once, as in prior work. By alternating between attending to the language input and generating FiLM layer parameters, this approach is better able to scale to settings with longer input sequences such as dialogue. We demonstrate that multi-hop FiLM generation achieves state-of-the-art for the short input sequence task ReferIt --- on-par with single-hop FiLM generation --- while also significantly outperforming prior state-of-the-art and single-hop FiLM generation on the GuessWhat?! visual dialogue task.

arXiv:2211.11559

Visual Programming: Compositional visual reasoning without training

We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VISPROG avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing routines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VISPROG on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like VISPROG are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.

arXiv:2411.11706공식 코드 · Papers with Code

MC-LLaVA: Multi-Concept Personalized Vision-Language Model

Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies investigate VLM personalization to understand user-provided concepts. However, they mainly focus on single-concept personalization, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes the first multi-concept personalization paradigm, MC-LLaVA. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the costs related to joint training, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location confidence maps for enhanced recognition and grounding capabilities. To advance multi-concept personalization research, we further contribute a high-quality instruction tuning dataset. We carefully collect images with multiple characters and objects from movies and manually generate question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses, paving the way for VLMs to become better user-specific assistants. The code and dataset will be publicly available at https://github.com/arctanxarc/MC-LLaVA.

arXiv:2402.16664공식 코드 · Papers with Code

LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery

Visual question answering (VQA) is crucial for promoting surgical education. In practice, the needs of trainees are constantly evolving, such as learning more surgical types, adapting to different robots, and learning new surgical instruments and techniques for various surgeries. However, patient data privacy often restricts the availability of old data when updating the model, necessitating an exemplar-free continual learning (CL) setup. Prior CL studies overlooked two vital problems in the surgical domain: 1) large domain shifts from diverse surgical operations collected from multiple sources, and 2) severe data imbalance arising from the uneven presence of surgical instruments or activities. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology. We first develop a new multi-teacher CL framework that leverages a multimodal LLM as the additional teacher. The strong generalization ability of the LLM can bridge the knowledge gap when domain shifts and data imbalances occur. We then put forth a novel data processing method that transforms complex LLM embeddings into logits compatible with our CL framework. We further design an adaptive weight assignment approach that balances the generalization ability of the LLM and the domain expertise of the old CL model. Finally, to comprehensively test the effectiveness of our proposed method, we have also constructed two new surgical VQA datasets that are largely different from existing ones and could be valuable resources for future research. Extensive experimental results on the tested datasets demonstrate the superiority of our method to other advanced CL schemes.

arXiv:1902.00313

VrR-VG: Refocusing Visually-Relevant Relationships

Relationships encode the interactions among individual instances, and play a critical role in deep visual scene understanding. Suffering from the high predictability with non-visual information, existing methods tend to fit the statistical bias rather than ``learning'' to ``infer'' the relationships from images. To encourage further development in visual relationships, we propose a novel method to automatically mine more valuable relationships by pruning visually-irrelevant ones. We construct a new scene-graph dataset named Visually-Relevant Relationships Dataset (VrR-VG) based on Visual Genome. Compared with existing datasets, the performance gap between learnable and statistical method is more significant in VrR-VG, and frequency-based analysis does not work anymore. Moreover, we propose to learn a relationship-aware representation by jointly considering instances, attributes and relationships. By applying the representation-aware feature learned on VrR-VG, the performances of image captioning and visual question answering are systematically improved with a large margin, which demonstrates the gain of our dataset and the features embedding schema. VrR-VG is available via http://vrr-vg.com/.

arXiv:2110.08797공식 코드 · Papers with Code

Towards Language-guided Visual Recognition via Dynamic Convolutions

In this paper, we are committed to establishing an unified and end-to-end multi-modal network via exploring the language-guided visual recognition. To approach this target, we first propose a novel multi-modal convolution module called Language-dependent Convolution (LaConv). Its convolution kernels are dynamically generated based on natural language information, which can help extract differentiated visual features for different multi-modal examples. Based on the LaConv module, we further build the first fully language-driven convolution network, termed as LaConvNet, which can unify the visual recognition and multi-modal reasoning in one forward structure. To validate LaConv and LaConvNet, we conduct extensive experiments on four benchmark datasets of two vision-and-language tasks, i.e., visual question answering (VQA) and referring expression comprehension (REC). The experimental results not only shows the performance gains of LaConv compared to the existing multi-modal modules, but also witness the merits of LaConvNet as an unified network, including compact network, high generalization ability and excellent performance, e.g., +4.7% on RefCOCO+.

arXiv:2505.15755

Exploring The Visual Feature Space for Multimodal Neural Decoding

The intrication of brain signals drives research that leverages multimodal AI to align brain modalities with visual and textual data for explainable descriptions. However, most existing studies are limited to coarse interpretations, lacking essential details on object descriptions, locations, attributes, and their relationships. This leads to imprecise and ambiguous reconstructions when using such cues for visual decoding. To address this, we analyze different choices of vision feature spaces from pre-trained visual components within Multimodal Large Language Models (MLLMs) and introduce a zero-shot multimodal brain decoding method that interacts with these models to decode across multiple levels of granularities. % To assess a model's ability to decode fine details from brain signals, we propose the Multi-Granularity Brain Detail Understanding Benchmark (MG-BrainDub). This benchmark includes two key tasks: detailed descriptions and salient question-answering, with metrics highlighting key visual elements like objects, attributes, and relationships. Our approach enhances neural decoding precision and supports more accurate neuro-decoding applications. Code will be available at https://github.com/weihaox/VINDEX.

arXiv:2507.07297

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC, a comprehensive benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense, and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning.

arXiv:2206.08358공식 코드 · Papers with Code

MixGen: A New Multi-Modal Data Augmentation

Data augmentation is a necessity to enhance data efficiency in deep learning. For vision-language pre-training, data is only augmented either for images or for text in previous works. In this paper, we present MixGen: a joint data augmentation for vision-language representation learning to further improve data efficiency. It generates new image-text pairs with semantic relationships preserved by interpolating images and concatenating text. It's simple, and can be plug-and-played into existing pipelines. We evaluate MixGen on four architectures, including CLIP, ViLT, ALBEF and TCL, across five downstream vision-language tasks to show its versatility and effectiveness. For example, adding MixGen in ALBEF pre-training leads to absolute performance improvements on downstream tasks: image-text retrieval (+6.2% on COCO fine-tuned and +5.3% on Flicker30K zero-shot), visual grounding (+0.9% on RefCOCO+), visual reasoning (+$0.9% on NLVR2), visual question answering (+0.3% on VQA2.0), and visual entailment (+0.4% on SNLI-VE).

arXiv:2410.16267

xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs

We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens. This enables BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs. 4608 tokens). We explore different types of temporal encoders, including learnable spatio-temporal pooling as well as sequential models like Token Turing Machines. We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using fewer visual tokens. The project website is at https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html

arXiv:2507.12795

City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning

Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf{\underline{SVM-City}}, deriving from multi\textbf{\underline{S}}cale scenarios with multi\textbf{\underline{V}}iew and multi\textbf{\underline{M}}odal instruction tuning data. It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf{\underline{City-VLM}}. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves $18.14 \%$ performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes.

arXiv:1906.09635

Investigating Biases in Textual Entailment Datasets

The ability to understand logical relationships between sentences is an important task in language understanding. To aid in progress for this task, researchers have collected datasets for machine learning and evaluation of current systems. However, like in the crowdsourced Visual Question Answering (VQA) task, some biases in the data inevitably occur. In our experiments, we find that performing classification on just the hypotheses on the SNLI dataset yields an accuracy of 64%. We analyze the bias extent in the SNLI and the MultiNLI dataset, discuss its implication, and propose a simple method to reduce the biases in the datasets.