Multimodal Large Language Models have advanced AI in applications like text-to-video generation and visual question answering. These models rely on visual encoders to convert non-text data into vectors, but current encoders either lack semantic alignment or overlook non-salient objects. We propose the Guiding Visual Encoder to Perceive Overlooked Information (GiVE) approach. GiVE enhances visual representation with an Attention-Guided Adapter (AG-Adapter) module and an Object-focused Visual Semantic Learning module. These incorporate three novel loss terms: Object-focused Image-Text Contrast (OITC) loss, Object-focused Image-Image Contrast (OIIC) loss, and Object-focused Image Discrimination (OID) loss, improving object consideration, retrieval accuracy, and comprehensiveness. Our contributions include dynamic visual focus adjustment, novel loss functions to enhance object retrieval, and the Multi-Object Instruction (MOInst) dataset. Experiments show our approach achieves state-of-the-art performance.
대규모 비전-언어 모델(LVLM)은 이미지 캡셔닝 및 시각적 질의응답과 같은 시각 이해 태스크에서 괄목할 만한 발전을 이루었습니다. 그러나 LVLM은 실제 시각 입력과 일치하지 않는 내용을 생성하는 환각(hallucination)에 여전히 취약합니다. 기존 방법들은 주로 디코딩 단계에서 개입하는 반면, 디코딩 과정을 오도하는 무관하거나 노이즈가 섞인 시각 토큰이라는 환각의 핵심적인 원인을 간과합니다. 이 문제를 해결하기 위해, 우리는 전통적인 머신러닝의 특징(feature) 엔지니어링 개념을 LVLM에 도입하는 학습(training) 불필요 프레임워크인 SeeMe를 제안합니다. SeeMe는 3단계 토큰(token) 엔지니어링 프로세스를 통해 시각 토큰을 재구성함으로써 환각 원인을 억제하는 동시에 정보가 풍부한 시각적 근거를 보존합니다. 4개의 LVLM에서 MME, POPE, AMBER 벤치마크에 대한 실험 결과, SeeMe가 일관되게 환각을 감소시키고 출력 일관성을 향상시킴을 보여주며, LVLM의 환각 완화를 위한 새로운 관점을 제시합니다.
Recently, there has been an increasing interest in building question answering (QA) models that reason across multiple modalities, such as text and images. However, QA using images is often limited to just picking the answer from a pre-defined set of options. In addition, images in the real world, especially in news, have objects that are co-referential to the text, with complementary information from both modalities. In this paper, we present a new QA evaluation benchmark with 1,384 questions over news articles that require cross-media grounding of objects in images onto text. Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question. In addition, we introduce a novel multimedia data augmentation framework, based on cross-media knowledge extraction and synthetic question-answer generation, to automatically augment data that can provide weak supervision for this task. We evaluate both pipeline-based and end-to-end pretraining-based multimedia QA models on our benchmark, and show that they achieve promising performance, while considerably lagging behind human performance hence leaving large room for future work on this challenging new task.
The success of deep learning has enabled advances in multimodal tasks that require non-trivial fusion of multiple input domains. Although multimodal models have shown potential in many problems, their increased complexity makes them more vulnerable to attacks. A Backdoor (or Trojan) attack is a class of security vulnerability wherein an attacker embeds a malicious secret behavior into a network (e.g. targeted misclassification) that is activated when an attacker-specified trigger is added to an input. In this work, we show that multimodal networks are vulnerable to a novel type of attack that we refer to as Dual-Key Multimodal Backdoors. This attack exploits the complex fusion mechanisms used by state-of-the-art networks to embed backdoors that are both effective and stealthy. Instead of using a single trigger, the proposed attack embeds a trigger in each of the input modalities and activates the malicious behavior only when both the triggers are present. We present an extensive study of multimodal backdoors on the Visual Question Answering (VQA) task with multiple architectures and visual feature backbones. A major challenge in embedding backdoors in VQA models is that most models use visual features extracted from a fixed pretrained object detector. This is challenging for the attacker as the detector can distort or ignore the visual trigger entirely, which leads to models where backdoors are over-reliant on the language trigger. We tackle this problem by proposing a visual trigger optimization strategy designed for pretrained object detectors. Through this method, we create Dual-Key Backdoors with over a 98% attack success rate while only poisoning 1% of the training data. Finally, we release TrojVQA, a large collection of clean and trojan VQA models to enable research in defending against multimodal backdoors.
경량 사용자 기기에서 시청각 질의응답을 하려면 속도, 메모리 사용량, 개인정보 보호, 통신 비용 사이에서 균형을 잡아야 한다. 한 가지 방법은 서버에서 시청각 인코더를 실행하고 사용자 기기에서 경량 언어 디코더를 실행하는 것이다. 미디어 인코딩을 서버에 맡기면서 질문은 기기 안에 둘 수 있다. 그러나 전체 멀티모달 표현을 전송하고 처리하는 비용은 여전히 크다. 전송 전에 재사용 가능한 작은 메모리를 만드는 토큰 가지치기 프레임워크를 제안한다. 학습이 필요 없고 쿼리에 의존하지 않는다. 비디오의 시공간 블록과 그에 맞춰 정렬한 오디오 윈도를 참여자로 정의하고 몬테카를로 섀플리 값으로 중요도를 추정한다. 임베딩 포괄 범위를 평가하는 목적함수는 쿼리 독립적인 효용과 모달리티 간 중복 보정을 결합한다. 포괄 범위의 한계 증가량으로 개별 모달리티 안의 다양성에도 보상한다. 제한된 토큰 예산 안에서 점수가 높은 참여자의 원래 토큰 임베딩을 유지한다. 이렇게 만든 메모리를 한 번만 전송하고 이후 기기의 여러 질문에서 재사용한다. video-SALMONN 2+(30억 매개변수)와 video-SALMONN 2 Pro(40억 매개변수)를 사용해 Video-MME, Daily-Omni, Video-Holmes에서 실험한 결과, 평가한 모든 설정에서 비교 압축 방법 가운데 질의응답 정확도가 가장 높거나 공동으로 가장 높았다. Daily-Omni에서 video-SALMONN 2+(30억 매개변수)의 토큰을 80% 가지치기했을 때 비교 방법보다 정확도가 12.86~14.62%포인트 높았다. 기기에서 대규모 언어 모델의 프리필과 디코딩 지연 시간이 가장 낮았으며, 비슷한 임베딩 메모리 사용량에서 디코딩 속도가 최대 $1.92\times$ 빨라졌다.
최근 수어(Sign Language, SL) 이해(SLU)의 발전은 연속 수어 인식 및 수어 번역과 같은 과제에서 괄목할 만한 진전을 이끌어냈습니다. 그러나 이러한 과제들은 사전에 정의된 목표로 설계되어, 모델이 수어 영상에서 글로스(gloss)나 음성 언어 문장으로의 고정된 매핑을 학습하도록 요구합니다. 결과적으로 이는 모델이 수어 영상의 의미론적 내용을 진정으로 이해하고 있는지에 대해 제한된 평가만을 제공합니다. 이러한 한계를 해결하기 위해, \textbf{저희는 모델이 수어 영상에 대한 임의의 자연어 질문에 답하도록 요구함으로써 수어 이해를 평가하는 새로운 과제인 수어 질의응답(Sign Language Question Answering, SLQA)을 처음으로 제안합니다}. 이전의 SLU 과제들과 달리, SLQA는 인식 및 번역을 넘어 다양한 추론 능력을 평가하는 더욱 유연하고 포괄적인 평가 프레임워크를 제공합니다. 이 과제를 촉진하기 위해, \textbf{저희는 정교하게 설계된 템플릿을 사용하여 기존의 글로스 및 문장 주석으로부터 질문-답변 쌍을 자동 생성함으로써 PHOENIX14T 및 CSL-Daily를 기반으로 한 두 개의 SignQA 벤치마크를 추가로 구축합니다}. 그 결과로 도출된 데이터셋은 위치 추론, 구조적 추론, 시각적 탐색, 글로스 인식, 번역 이해를 포함한 상호 보완적인 5가지 질문 범주를 다룹니다. \textbf{마지막으로, 질문 조건부 변조 시간 다운샘플링(Question-Conditioned Modulated Temporal Downsampling) 모듈과 도메인 내 지식 전이 전략을 갖춘 단순하면서도 효과적인 기준선 모델을 제안합니다}. 이를 통해 질문 인식 시간 특징 모델링을 강화하는 동시에 기존 SLU 과제로부터 효과적인 지식 전이를 가능하게 합니다. 광범위한 실험을 통해 저희의 기준선 모델이 모든 질문 범주에 걸쳐 대표적인 비전-언어 모델들을 지속적으로 능가함을 입증하며, 향후 SLQA 연구를 위한 강력한 벤치마크를 정립합니다. 데이터셋은 https://huggingface.co/datasets/hulala/SignQA-2026 에서 이용할 수 있습니다.
The collaborative reasoning for understanding each image-question pair is very critical but under-explored for an interpretable Visual Question Answering (VQA) system. Although very recent works also tried the explicit compositional processes to assemble multiple sub-tasks embedded in the questions, their models heavily rely on the annotations or hand-crafted rules to obtain valid reasoning layout, leading to either heavy labor or poor performance on composition reasoning. In this paper, to enable global context reasoning for better aligning image and language domains in diverse and unrestricted cases, we propose a novel reasoning network called Adversarial Composition Modular Network (ACMN). This network comprises of two collaborative modules: i) an adversarial attention module to exploit the local visual evidence for each word parsed from the question; ii) a residual composition module to compose the previously mined evidence. Given a dependency parse tree for each question, the adversarial attention module progressively discovers salient regions of one word by densely combining regions of child word nodes in an adversarial manner. Then residual composition module merges the hidden representations of an arbitrary number of children through sum pooling and residual connection. Our ACMN is thus capable of building an interpretable VQA system that gradually dives the image cues following a question-driven reasoning route and makes global reasoning by incorporating the learned knowledge of all attention modules in a principled manner. Experiments on relational datasets demonstrate the superiority of our ACMN and visualization results show the explainable capability of our reasoning system.
This paper focuses on open-ended video question answering, which aims to find the correct answers from a large answer set in response to a video-related question. This is essentially a multi-label classification task, since a question may have multiple answers. However, due to annotation costs, the labels in existing benchmarks are always extremely insufficient, typically one answer per question. As a result, existing works tend to directly treat all the unlabeled answers as negative labels, leading to limited ability for generalization. In this work, we introduce a simple yet effective ranking distillation framework (RADI) to mitigate this problem without additional manual annotation. RADI employs a teacher model trained with incomplete labels to generate rankings for potential answers, which contain rich knowledge about label priority as well as label-associated visual cues, thereby enriching the insufficient labeling information. To avoid overconfidence in the imperfect teacher model, we further present two robust and parameter-free ranking distillation approaches: a pairwise approach which introduces adaptive soft margins to dynamically refine the optimization constraints on various pairwise rankings, and a listwise approach which adopts sampling-based partial listwise learning to resist the bias in teacher ranking. Extensive experiments on five popular benchmarks consistently show that both our pairwise and listwise RADIs outperform state-of-the-art methods. Further analysis demonstrates the effectiveness of our methods on the insufficient labeling problem.
Recent advancements in Computer Assisted Diagnosis have shown promising performance in medical imaging tasks, particularly in chest X-ray analysis. However, the interaction between these models and radiologists has been primarily limited to input images. This work proposes a novel approach to enhance human-computer interaction in chest X-ray analysis using Vision-Language Models (VLMs) enhanced with radiologists' attention by incorporating eye gaze data alongside textual prompts. Our approach leverages heatmaps generated from eye gaze data, overlaying them onto medical images to highlight areas of intense radiologist's focus during chest X-ray evaluation. We evaluate this methodology in tasks such as visual question answering, chest X-ray report automation, error detection, and differential diagnosis. Our results demonstrate the inclusion of eye gaze information significantly enhances the accuracy of chest X-ray analysis. Also, the impact of eye gaze on fine-tuning was confirmed as it outperformed other medical VLMs in all tasks except visual question answering. This work marks the potential of leveraging both the VLM's capabilities and the radiologist's domain knowledge to improve the capabilities of AI models in medical imaging, paving a novel way for Computer Assisted Diagnosis with a human-centred AI.
Cross-modal alignment Learning integrates information from different modalities like text, image, audio and video to create unified models. This approach develops shared representations and learns correlations between modalities, enabling applications such as visual question answering and audiovisual content analysis. Current techniques rely on large modality-specific encoders, necessitating fine-tuning or training from scratch on vast aligned datasets (e.g., text-image, text-audio, image-audio). This approach has limitations: (i) it is very expensive due to the need for training large encoders on extensive datasets, (ii) acquiring aligned large paired datasets is challenging, and (iii) adding new modalities requires retraining the entire framework to incorporate these modalities. To address these issues, we propose OneEncoder, a lightweight framework that progressively represents and aligns four modalities (image, text, audio, video). Initially, we train a lightweight Universal Projection module (UP) to align image and text modalities. Then, we freeze the pretrained UP and progressively align future modalities to those already aligned. OneEncoder operates efficiently and cost-effectively, even in scenarios where vast aligned datasets are unavailable, due to its lightweight design. Trained on small paired datasets, it shows strong performance in tasks like classification, querying, and visual question answering, surpassing methods that rely on large datasets and specialized encoders.
Visual Question Answering (VQA) is a challenging problem that requires to process multimodal input. Answer-Set Programming (ASP) has shown great potential in this regard to add interpretability and explainability to modular VQA architectures. In this work, we address the problem of how to integrate ASP with modules for vision and natural language processing to solve a new and demanding VQA variant that is concerned with images of graphs (not graphs in symbolic form). Images containing graph-based structures are an ubiquitous and popular form of visualisation. Here, we deal with the particular problem of graphs inspired by transit networks, and we introduce a novel dataset that amends an existing one by adding images of graphs that resemble metro lines. Our modular neuro-symbolic approach combines optical graph recognition for graph parsing, a pretrained optical character recognition neural network for parsing labels, Large Language Models (LLMs) for language processing, and ASP for reasoning. This method serves as a first baseline and achieves an overall average accuracy of 73% on the dataset. Our evaluation provides further evidence of the potential of modular neuro-symbolic systems, in particular with pretrained models that do not involve any further training and logic programming for reasoning, to solve complex VQA tasks.
Visual question answering and visual dialogue tasks have been increasingly studied in the multimodal field towards more practical real-world scenarios. A more challenging task, audio visual scene-aware dialogue (AVSD), is proposed to further advance the technologies that connect audio, vision, and language, which introduces temporal video information and dialogue interactions between a questioner and an answerer. This paper proposes an intuitive mechanism that fuses features and attention in multiple stages in order to well integrate multimodal features, and the results demonstrate its capability in the experiments. Also, we apply several state-of-the-art models in other tasks to the AVSD task, and further analyze their generalization across different tasks.
With the new generation of satellite technologies, the archives of remote sensing (RS) images are growing very fast. To make the intrinsic information of each RS image easily accessible, visual question answering (VQA) has been introduced in RS. VQA allows a user to formulate a free-form question concerning the content of RS images to extract generic information. It has been shown that the fusion of the input modalities (i.e., image and text) is crucial for the performance of VQA systems. Most of the current fusion approaches use modality-specific representations in their fusion modules instead of joint representation learning. However, to discover the underlying relation between both the image and question modality, the model is required to learn the joint representation instead of simply combining (e.g., concatenating, adding, or multiplying) the modality-specific representations. We propose a multi-modal transformer-based architecture to overcome this issue. Our proposed architecture consists of three main modules: i) the feature extraction module for extracting the modality-specific features; ii) the fusion module, which leverages a user-defined number of multi-modal transformer layers of the VisualBERT model (VB); and iii) the classification module to obtain the answer. Experimental results obtained on the RSVQAxBEN and RSVQA-LR datasets (which are made up of RGB bands of Sentinel-2 images) demonstrate the effectiveness of VBFusion for VQA tasks in RS. To analyze the importance of using other spectral bands for the description of the complex content of RS images in the framework of VQA, we extend the RSVQAxBEN dataset to include all the spectral bands of Sentinel-2 images with 10m and 20m spatial resolution.
지식 그래프 질의응답(KGQA)은 구조화된 그래프 추론을 통해 자연어 질문에 답할 수 있게 하며, 여러 벤치마크와 응용 분야에서 상당한 진전을 이뤘다. 최근 멀티모달 KGQA(MM-KGQA)는 많은 질문이 멀티모달 입력과 KG 증거를 함께 사용해야 하기 때문에 주목을 받고 있다. 그러나 기존 MM-KGQA 방법들은 일반적으로 멀티모달 정보를 시작 엔티티 그라운딩 또는 증거 검색에만 사용한 뒤, 다중 홉(multi-hop) 추론이 텍스트 전용 그래프 탐색으로 퇴화한다. 그 결과 중간 홉에서 중요해지는 멀티모달 단서를 활용하지 못한다. 이 한계를 해결하기 위해 MM-KGQA를 위한 시각적 의도 기반 경로 추론 프레임워크인 VISPATH를 제안한다. VISPATH는 먼저 멀티모달 그라운딩과 그래프 구조적 단서를 결합해 신뢰할 수 있는 시작 엔티티를 식별한다. 이어 입력, 질문, 현재 부분 경로로부터 홉별 멀티모달 의도를 재계산하는 의도 기반 경로 탐색을 수행하여, 각 확장이 현재 추론 상태에 의해 안내되도록 한다. 발견된 경로는 추론 연쇄 가지치기(reasoning-chain pruning)를 통해 추가로 정제되며, 이는 후보 경로를 질문, 추론 개요(reasoning sketch), 홉별 의도와의 일관성에 따라 완전한 증거 연쇄로 평가한다. 마지막으로 VISPATH는 선택된 증거가 답 생성에 충분한지 확인한다. 또한 KG 경로에서 2~4홉을 요구하는 질문을 포함하여, KG에 대한 멀티모달 다중 홉 추론을 평가하는 벤치마크인 VISPATH-Bench를 구축한다. VISPATH-Bench와 추가 멀티모달 QA 벤치마크 3개에 대한 광범위한 실험에서 VISPATH는 강력한 기준선들을 일관되게 능가한다. 특히 GPT-4o를 백본으로 할 때 VISPATH는 VISPATH-Bench에서 GPT-5.4를 넘어서며, 평균 정확도에서 10.6% 상대 향상과 2홉 추론에서 13.1% 향상을 달성한다.
This paper introduces a novel and significant challenge for Vision Language Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the VLM's ability to withhold answers when faced with unsolvable problems in the context of Visual Question Answering (VQA) tasks. UPD encompasses three distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply investigate the UPD problem, extensive experiments indicate that most VLMs, including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying extents, highlighting significant room for the improvements. To address UPD, we explore both training-free and training-based solutions, offering new insights into their effectiveness and limitations. We hope our insights, together with future efforts within the proposed UPD settings, will enhance the broader understanding and development of more practical and reliable VLMs.
지식 기반 시각적 질문 답변(KB-VQA)은 시각적으로 그라운딩된(visually grounded) 질문에 답하기 위해 외부 소스로부터 엔티티 지식을 검색해야 합니다. 기존의 검색-증강(retrieval-augmented) 시스템은 두 가지 중요한 한계점을 지닙니다. 첫째, 단일 검색 모달리티에 의존하면 상호 보완적인 소스를 통해서만 접근할 수 있는 정답(ground-truth) 엔티티를 놓치는 단일 소스 검색 병목 현상(Single-Source Retrieval Bottleneck)이 발생합니다. 둘째, 듀얼 타워 포인트와이즈 재순위화기(reranker)는 검색 출처와 후보 수준의 검색 사전 정보(retrieval priors)를 간과하여 불필요한 모달리티 의존성을 유발하는 검색 소스 블라인드 재순위화(Retrieval-Source-Blind Reranking) 문제를 겪습니다. 이러한 문제를 해결하기 위해, 본 논문에서는 이종 소스 엔티티 검색 및 재순위화를 위한 통합 경량 프레임워크인 UniHEAR를 제안합니다. UniHEAR는 각 후보 엔티티에 대해 대략적 검색 기술자(Coarse Retrieval Descriptor)를 구축하고, 대략적 검색 사전 정보의 엔트로피 가중 소스 융합(Entropy-Weighted Source Fusion)으로 보완되는 검색 유도 어텐션 모달리티 게이팅(Retrieval-Guided Attentive Modality Gating)을 도입하여 이 기술자에 맞춰 모달리티 어텐션 가중치를 조건화합니다. 대조 학습(contrastive learning)과 보조 모달리티 보존 손실을 결합한 하이브리드 학습 전략은 단일 모델 내에서 엔티티 수준 및 섹션 수준의 검색을 통합합니다. E-VQA 및 InfoSeek에 대한 광범위한 실험 결과, UniHEAR가 경량 재순위화 아키텍처를 유지하면서도 가장 강력한 베이스라인 대비 Recall@1을 각각 6.7포인트 및 1.2포인트 향상시켜 최고 수준의 검색 및 VQA 성능을 달성함을 보여줍니다. 코드와 모델은 https://github.com/iven-luo/UniHEAR 에서 확인할 수 있습니다.
Document Visual Question Answering (DocVQA) has quickly grown into a central task of document understanding. But despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees. In this work, we explore privacy in the domain of DocVQA for the first time, highlighting privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions. Specifically, we focus on invoice processing as a realistic document understanding scenario, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the data of the invoice provider is the sensitive information to be protected. We demonstrate that non-private models tend to memorise, a behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through either or both of the two input modalities: vision (document image) or language (OCR tokens). Finally, we design attacks exploiting the memorisation effect of the model, and demonstrate their effectiveness in probing a representative DocVQA models.
Knowledge-based visual question answering (VQA) involves answering questions that require external knowledge not present in the image. Existing methods first retrieve knowledge from external resources, then reason over the selected knowledge, the input image, and question for answer prediction. However, this two-step approach could lead to mismatches that potentially limit the VQA performance. For example, the retrieved knowledge might be noisy and irrelevant to the question, and the re-embedded knowledge features during reasoning might deviate from their original meanings in the knowledge base (KB). To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by GPT-3's power in knowledge retrieval and question answering, instead of using structured KBs as in previous work, we treat GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge. Specifically, we first convert the image into captions (or tags) that GPT-3 can understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just providing a few in-context VQA examples. We further boost performance by carefully investigating: (i) what text formats best describe the image content, and (ii) how in-context examples can be better selected and used. PICa unlocks the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa surpasses the supervised state of the art by an absolute +8.6 points on the OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent few-shot performance.
Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert
Recently, large language models have shown remarkable reasoning capabilities through long-chain reasoning before responding. However, how to extend this capability to visual reasoning tasks remains an open challenge. Existing multimodal reasoning approaches transfer such visual reasoning task into textual reasoning task via several image-to-text conversions, which often lose critical structural and semantic information embedded in visualizations, especially for tasks like chart question answering that require a large amount of visual details. To bridge this gap, we propose ChartReasoner, a code-driven novel two-stage framework designed to enable precise, interpretable reasoning over charts. We first train a high-fidelity model to convert diverse chart images into structured ECharts codes, preserving both layout and data semantics as lossless as possible. Then, we design a general chart reasoning data synthesis pipeline, which leverages this pretrained transport model to automatically and scalably generate chart reasoning trajectories and utilizes a code validator to filter out low-quality samples. Finally, we train the final multimodal model using a combination of supervised fine-tuning and reinforcement learning on our synthesized chart reasoning dataset and experimental results on four public benchmarks clearly demonstrate the effectiveness of our proposed ChartReasoner. It can preserve the original details of the charts as much as possible and perform comparably with state-of-the-art open-source models while using fewer parameters, approaching the performance of proprietary systems like GPT-4o in out-of-domain settings.