PyTorchKR Papers

“Visual Question Answering”

총 2,807편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2406.09454공식 코드 · Papers with Code

Advancing High Resolution Vision-Language Models in Biomedicine

Multi-modal learning has significantly advanced generative AI, especially in vision-language modeling. Innovations like GPT-4V and open-source projects such as LLaVA have enabled robust conversational agents capable of zero-shot task completions. However, applying these technologies in the biomedical field presents unique challenges. Recent initiatives like LLaVA-Med have started to adapt instruction-tuning for biomedical contexts using large datasets such as PMC-15M. Our research offers three key contributions: (i) we present a new instruct dataset enriched with medical image-text pairs from Claude3-Opus and LLaMA3 70B, (ii) we propose a novel image encoding strategy using hierarchical representations to improve fine-grained biomedical visual comprehension, and (iii) we develop the Llama3-Med model, which achieves state-of-the-art zero-shot performance on biomedical visual question answering benchmarks, with an average performance improvement of over 10% compared to previous methods. These advancements provide more accurate and reliable tools for medical professionals, bridging gaps in current multi-modal conversational assistants and promoting further innovations in medical AI.

arXiv:2504.10465공식 코드 · Papers with Code

Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding

Multimodal Large Language Models (MLLMs) achieve remarkable performance for fine-grained pixel-level understanding tasks. However, all the works rely heavily on extra components, such as vision encoder (CLIP), segmentation experts, leading to high system complexity and limiting model scaling. In this work, our goal is to explore a highly simplified MLLM without introducing extra components. Our work is motivated by the recent works on Single trAnsformer as a unified vIsion-Language Model (SAIL) design, where these works jointly learn vision tokens and text tokens in transformers. We present Pixel-SAIL, a single transformer for pixel-wise MLLM tasks. In particular, we present three technical improvements on the plain baseline. First, we design a learnable upsampling module to refine visual token features. Secondly, we propose a novel visual prompt injection strategy to enable the single transformer to understand visual prompt inputs and benefit from the early fusion of visual prompt embeddings and vision tokens. Thirdly, we introduce a vision expert distillation strategy to efficiently enhance the single transformer's fine-grained feature extraction capability. In addition, we have collected a comprehensive pixel understanding benchmark (PerBench), using a manual check. It includes three tasks: detailed object description, visual prompt-based question answering, and visual-text referring segmentation. Extensive experiments on four referring segmentation benchmarks, one visual prompt benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or even better results with a much simpler pipeline. Code and model will be released at https://github.com/magic-research/Sa2VA.

arXiv:2606.12590v2cs.CV

의료 대규모 비전-언어 모델에서의 세분화된 선호 최적화 분석 및 개선

Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs

선호 최적화(preference optimization)는 의료 대규모 비전-언어 모델(LVLM)의 사후학습(post-training)에 점점 더 널리 사용되고 있으나, 임상적 정확성을 정의하는 단위보다 훨씬 거친 단위(granularity)에서 동작합니다. 한 응답이 다른 응답보다 임상적으로 더 나은지 여부는 통상적으로 해부학적 좌우 구분(laterality)이나 병변 속성과 같은 소수의 결정적인 어구, 그리고 각 어구가 질문이 다루는 이미지 영역에 의해 뒷받침되는지 여부에 달려 있습니다. 반면 직접 선호 최적화(DPO) 및 그 변형들은 비교를 단일 응답 수준의 스칼라로 축소하므로, 어떤 토큰(token)이 임상적 의미를 담고 있는지를 구조적으로 표현할 수 없는 목적 함수입니다. 본 연구에서는 이러한 불일치가 치명적임을 보입니다. 비교 대상 목적 함수 중 응답 수준 DPO는 임상적 정확성을 결정짓는 어구에 가장 낮은 보상(reward)을 부여하며, 선호 응답을 지도학습 기준(supervised reference)으로 대체할 경우 모델(model)이 보상 해킹 지름길로 악용하는 문체적 격차가 발생하여 선호 정확도는 높아지지만 임상적 정확도는 향상되지 않습니다. 임상적 비교에서 전체 피드백을 포착하기 위해, 본 논문에서는 세분화된 정규화 의료 선호 최적화(FiRe-MPO)를 제안합니다. 선호 쌍은 모델 자체 생성물을 최소한으로 편집하여 선호 응답과 거부 응답이 임상적으로 결정적인 구간(span)에서만 차이 나도록 구성되며, 각 응답은 뒷받침하는 시각적 근거를 감춘 병변 손상 이미지와 짝지어집니다. 구간 단위로 국소화된 보상은 희소(sparse)하므로, 양방향 토큰 단위 KL 정규화기를 통해 최적화를 안정화합니다. 의료 시각적 질문 답변(VQA) 및 판독문 생성 전반에 걸쳐, FiRe-MPO는 의료 특화 및 범용으로 널리 사용되는 두 가지 LVLM에서 DPO, RRPO 및 경쟁 세분화 목적 함수들의 성능을 능가하는 동시에, 시각적 그라운딩(grounding)을 강화하고 대안 대비 의료 어구에 더 많은 보상을 부여합니다.

arXiv:2312.06352공식 코드 · Papers with Code

NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations

Visual Question Answering (VQA) is one of the most important tasks in autonomous driving, which requires accurate recognition and complex situation evaluations. However, datasets annotated in a QA format, which guarantees precise language generation and scene recognition from driving scenes, have not been established yet. In this work, we introduce Markup-QA, a novel dataset annotation technique in which QAs are enclosed within markups. This approach facilitates the simultaneous evaluation of a model's capabilities in sentence generation and VQA. Moreover, using this annotation methodology, we designed the NuScenes-MQA dataset. This dataset empowers the development of vision language models, especially for autonomous driving tasks, by focusing on both descriptive capabilities and precise QA. The dataset is available at https://github.com/turingmotors/NuScenes-MQA.

arXiv:2402.15096

Multimodal Transformer With a Low-Computational-Cost Guarantee

Transformer-based models have significantly improved performance across a range of multimodal understanding tasks, such as visual question answering and action recognition. However, multimodal Transformers significantly suffer from a quadratic complexity of the multi-head attention with the input sequence length, especially as the number of modalities increases. To address this, we introduce Low-Cost Multimodal Transformer (LoCoMT), a novel multimodal attention mechanism that aims to reduce computational cost during training and inference with minimal performance loss. Specifically, by assigning different multimodal attention patterns to each attention head, LoCoMT can flexibly control multimodal signals and theoretically ensures a reduced computational cost compared to existing multimodal Transformer variants. Experimental results on two multimodal datasets, namely Audioset and MedVidCL demonstrate that LoCoMT not only reduces GFLOPs but also matches or even outperforms established models.

arXiv:2411.05056

Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models

Multi-Modal Language Models (MLLMs) have transformed artificial intelligence by combining visual and text data, making applications like image captioning, visual question answering, and multi-modal content creation possible. This ability to understand and work with complex information has made MLLMs useful in areas such as healthcare, autonomous systems, and digital content. However, integrating multiple types of data also creates security risks. Attackers can manipulate either the visual or text inputs, or both, to make the model produce unintended or even harmful responses. This paper reviews how visual inputs in MLLMs can be exploited by various attack strategies. We break down these attacks into categories: simple visual tweaks and cross-modal manipulations, as well as advanced strategies like VLATTACK, HADES, and Collaborative Multimodal Adversarial Attack (Co-Attack). These attacks can mislead even the most robust models while looking nearly identical to the original visuals, making them hard to detect. We also discuss the broader security risks, including threats to privacy and safety in important applications. To counter these risks, we review current defense methods like the SmoothVLM framework, pixel-wise randomization, and MirrorCheck, looking at their strengths and limitations. We also discuss new methods to make MLLMs more secure, including adaptive defenses, better evaluation tools, and security approaches that protect both visual and text data. By bringing together recent developments and identifying key areas for improvement, this review aims to support the creation of more secure and reliable multi-modal AI systems for real-world use.

arXiv:2409.03272

OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving

The rise of multi-modal large language models(MLLMs) has spurred their applications in autonomous driving. Recent MLLM-based methods perform action by learning a direct mapping from perception to action, neglecting the dynamics of the world and the relations between action and world dynamics. In contrast, human beings possess world model that enables them to simulate the future states based on 3D internal visual representation and plan actions accordingly. To this end, we propose OccLLaMA, an occupancy-language-action generative world model, which uses semantic occupancy as a general visual representation and unifies vision-language-action(VLA) modalities through an autoregressive model. Specifically, we introduce a novel VQVAE-like scene tokenizer to efficiently discretize and reconstruct semantic occupancy scenes, considering its sparsity and classes imbalance. Then, we build a unified multi-modal vocabulary for vision, language and action. Furthermore, we enhance LLM, specifically LLaMA, to perform the next token/scene prediction on the unified vocabulary to complete multiple tasks in autonomous driving. Extensive experiments demonstrate that OccLLaMA achieves competitive performance across multiple tasks, including 4D occupancy forecasting, motion planning, and visual question answering, showcasing its potential as a foundation model in autonomous driving.

arXiv:2308.08414공식 코드 · Papers with Code

Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer

Video-language pre-trained models have shown remarkable success in guiding video question-answering (VideoQA) tasks. However, due to the length of video sequences, training large-scale video-based models incurs considerably higher costs than training image-based ones. This motivates us to leverage the knowledge from image-based pretraining, despite the obvious gaps between image and video domains. To bridge these gaps, in this paper, we propose Tem-Adapter, which enables the learning of temporal dynamics and complex semantics by a visual Temporal Aligner and a textual Semantic Aligner. Unlike conventional pretrained knowledge adaptation methods that only concentrate on the downstream task objective, the Temporal Aligner introduces an extra language-guided autoregressive task aimed at facilitating the learning of temporal dependencies, with the objective of predicting future states based on historical clues and language guidance that describes event progression. Besides, to reduce the semantic gap and adapt the textual representation for better event description, we introduce a Semantic Aligner that first designs a template to fuse question and answer pairs as event descriptions and then learns a Transformer decoder with the whole video sequence as guidance for refinement. We evaluate Tem-Adapter and different pre-train transferring methods on two VideoQA benchmarks, and the significant performance improvement demonstrates the effectiveness of our method.

arXiv:2506.20330

Semantic-enhanced Modality-asymmetric Retrieval for Online E-commerce Search

Semantic retrieval, which retrieves semantically matched items given a textual query, has been an essential component to enhance system effectiveness in e-commerce search. In this paper, we study the multimodal retrieval problem, where the visual information (e.g, image) of item is leveraged as supplementary of textual information to enrich item representation and further improve retrieval performance. Though learning from cross-modality data has been studied extensively in tasks such as visual question answering or media summarization, multimodal retrieval remains a non-trivial and unsolved problem especially in the asymmetric scenario where the query is unimodal while the item is multimodal. In this paper, we propose a novel model named SMAR, which stands for Semantic-enhanced Modality-Asymmetric Retrieval, to tackle the problem of modality fusion and alignment in this kind of asymmetric scenario. Extensive experimental results on an industrial dataset show that the proposed model outperforms baseline models significantly in retrieval accuracy. We have open sourced our industrial dataset for the sake of reproducibility and future research works.

arXiv:2606.17412v3cs.CV

교차 스케일 추론을 통한 병리학적 비전-언어 모델 강화

Enhancing Pathological VLMs with Cross-scale Reasoning

병리학적 이미지는 본질적으로 다중 스케일을 가지며, 병리학자는 정확한 진단을 위해 낮은 배율에서의 전체 조직 구조부터 높은 배율에서의 세포 형태학까지 증거를 통합해야 한다. 기존의 비전-언어 모델(VLM)을 위한 병리학적 데이터셋은 다양한 스케일을 포함하지만, 명시적인 교차 스케일 추론 목표가 부족한 경우가 많다. 이러한 한계로 인해 VLM은 필수적인 교차 스케일 표현을 포착하고 증거 기반 추론을 학습하는 데 어려움을 겪는다. 이 격차를 해소하기 위해, 우리는 병리학적 해석을 다중 배율 추론으로 공식화하는 최초의 교차 스케일 학습 및 평가 패러다임을 소개한다. 그러나 이러한 태스크를 구성하는 과정에서 중요한 문제가 드러난다: 다중 이미지 시각적 질의응답(VQA)은 텍스트 전용 단축키에 취약하여, 모델이 시각적 증거가 아닌 배율 의존적 아티팩트를 이용해 답을 추측할 수 있다. 이를 해결하기 위해, 우리는 적대적 텍스트 전용 스크리닝과 제약 조건 기반 질문 설계를 결합한 누출 인식(leakage-aware) 큐레이션 파이프라인을 제안한다. 이 파이프라인을 활용하여, 다양한 배율 수준의 2,537개 병리학 이미지를 기반으로 한 4,685개의 다지선다형 질문으로 구성된 고품질 벤치마크 Scale-VQA를 구축한다. 마지막으로, 교차 스케일 VQA 태스크의 성능을 최적화하기 위해 강화학습을 통해 학습된 모델인 ScaleReasoner-R1을 제시한다. ScaleReasoner-R1은 교차 스케일 추론 벤치마크에서 최첨단(state-of-the-art) 성능을 달성하며, 기존의 단일 스케일 벤치마크에서도 최첨단 성능으로 일반화된다. 연구 결과는 제한된 교차 스케일 지도학습만으로도 병리학적 이해를 크게 향상시킬 수 있음을 시사한다. 코드는 https://github.com/iMVR-PL/ScaleReasoner-R1 에서 확인할 수 있다.

arXiv:2406.11334

Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

Large language and multimodal models have shown remarkable successes on various benchmarks focused on specific skills such as general-purpose programming, natural language understanding, math word problem-solving, and visual question answering. However, it is unclear how well these models perform on tasks that require a combination of these skills. In this paper, we curate a novel program synthesis benchmark based on the XLogoOnline visual programming environment. The benchmark comprises 85 real-world tasks from the Mini-level of the XLogoOnline environment, each requiring a combination of different skills such as spatial planning, basic programming, and logical reasoning. Our evaluation shows that current state-of-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only 20% and 2.35% success rates. Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over 80000 tasks. Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution. We showcase that a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models, and provide an in-depth analysis of the models' expertise across different skill dimensions. We will publicly release the benchmark for future research on program synthesis in visual programming.

arXiv:2609.00232v1cs.CV

맹목적 순응을 넘어서: OCR 추론에서의 작업 검증 벤치마킹

Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning

멀티모달 대규모 언어 모델(MLLM)은 OCR 중심의 문서 이해 및 텍스트가 풍부한 시각적 추론 벤치마크(benchmark)에서 뛰어난 성능을 달성했습니다. 그러나 기존의 평가(evaluation)는 모든 작업이 유효하고 답변 가능하다는 가정을 크게 전제로 합니다. 실제 OCR 시나리오에서는 이러한 가정이 무너지는 경우가 많습니다. 즉, 질문이 판독 불가능한 텍스트, 가려진 증거, 존재하지 않는 시각적 대상, 모순되는 전제, 또는 누락된 변수에 의존할 수 있습니다. 본 연구에서는 이러한 신뢰성 격차(reliability gap)를 OCR 그라운딩(grounding) 기반 작업 검증(Task Verification) 관점에서 연구합니다. 즉, 모델(model)은 답변하기 전에 이미지 전제(IP), 텍스트 전제(TP), 질문(Q)이 실행 가능한 작업을 공동으로 정의하는지 여부를 판단해야 합니다. 이를 위해 8개의 OCR 관련 데이터셋(dataset)에서 추출되고 8개의 실제 이미지 도메인에 걸친 소스 이미지와 통제된 이미지 기반 진단 작업으로 구축된 1,800개 샘플 규모의 인간 검증 벤치마크인 VeriOCRBench를 소개합니다. 이 벤치마크는 시각적(Visual), 맥락적(Contextual), 사실적(Factual), 논리적(Logical)이라는 4가지 검증 차원과 8가지 함정 유형에 걸친 1,600개의 함정 주입 무효 작업에 더해, 과도한 거부(over-refusal)를 측정하기 위한 200개의 무함정 대조군을 포함합니다. 시각적 원자 사실(Visual Atomic Fact, VAF) 고정 파이프라인과 전수 인간 감사를 통해 구축된 VeriOCRBench는 작업 검증, 근본 원인 진단 및 과도한 거부에 대한 분리된 평가를 가능하게 합니다. 15개의 주요 MLLM을 평가한 결과, 지속적인 맹목적 순응, 진단 실패, 프롬프트(prompt)로 유발된 과도한 거부가 드러나며 현재 OCR 추론 시스템의 심각한 신뢰성 격차가 드러났습니다. 코드는 https://github.com/zy001122/Beyond-Blind-Compliance 에서 확인할 수 있습니다.

arXiv:2503.06003

Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Models

Situational awareness applications rely heavily on real-time processing of visual and textual data to provide actionable insights. Vision language models (VLMs) have become essential tools for interpreting complex environments by connecting visual inputs with natural language descriptions. However, these models often face computational challenges, especially when required to perform efficiently in real environments. This research presents a novel vision language model (VLM) framework that leverages frequency domain transformations and low-rank adaptation (LoRA) to enhance feature extraction, scalability, and efficiency. Unlike traditional VLMs, which rely solely on spatial-domain representations, our approach incorporates Discrete Fourier Transform (DFT) based low-rank features while retaining pretrained spatial weights, enabling robust performance in noisy or low visibility scenarios. We evaluated the proposed model on caption generation and Visual Question Answering (VQA) tasks using benchmark datasets with varying levels of Gaussian noise. Quantitative results demonstrate that our model achieves evaluation metrics comparable to state-of-the-art VLMs, such as CLIP ViT-L/14 and SigLIP. Qualitative analysis further reveals that our model provides more detailed and contextually relevant responses, particularly for real-world images captured by a RealSense camera mounted on an Unmanned Ground Vehicle (UGV).

arXiv:1506.01144

What value do explicit high level concepts have in vision to language problems?

Much of the recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. We propose here a method of incorporating high-level concepts into the very successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art performance in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. In doing so we provide an analysis of the value of high level semantic information in V2L problems.

arXiv:2412.13835공식 코드 · Papers with Code

RACQUET: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs

Ambiguity resolution is key to effective communication. While humans effortlessly address ambiguity through conversational grounding strategies, the extent to which current language models can emulate these strategies remains unclear. In this work, we examine referential ambiguity in image-based question answering by introducing RACQUET, a carefully curated dataset targeting distinct aspects of ambiguity. Through a series of evaluations, we reveal significant limitations and problems of overconfidence of state-of-the-art large multimodal language models in addressing ambiguity in their responses. The overconfidence issue becomes particularly relevant for RACQUET-BIAS, a subset designed to analyze a critical yet underexplored problem: failing to address ambiguity leads to stereotypical, socially biased responses. Our results underscore the urgency of equipping models with robust strategies to deal with uncertainty without resorting to undesirable stereotypes.

arXiv:2607.08136v1cs.AI

활력을 불어넣은 답집합 프로그래밍! ASP와 에너지 기반 모델을 활용한 엔드-투-엔드 신경기호 추론 및 학습

Answer Set Programming Energised! End-to-End Neurosymbolic Reasoning and Learning with ASP and Energy Based Models

답집합 프로그래밍(ASP)과 에너지 기반 모델(EBM)을 모듈식으로 통합하는 신경기호(neurosymbolic) 추론·학습 프레임워크 ASPEn을 제안하며, ASP의 비단조 추론과 제약 조건이 연속 잠재 공간의 에너지 최적화를 직접 구조화하는 엔드-투-엔드 학습을 지원한다. MNIST 기반 덧셈, 시각적 질문-답변 벤치마크 CLEVR, 다중 객체 추적 벤치마크 MOT에서 평가하여 소음·부분 관측 등 실세계 동적 도메인에서의 강건성을 입증한다.

arXiv:2607.23794v1cs.CV

PathScale-R1: 병리학적 이미지 분석을 위한 교차 스케일 추론

PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis

병리학적 진단은 본질적으로 다중 스케일(multi-scale)이며, 저배율에서의 전역 조직 구조와 고배율에서의 세포 형태학을 통합할 것을 요구한다. 그러나 기존 병리학 벤치마크와 비전-언어 모델(VLM)은 여전히 대부분 단일 스케일 설정 하에서 개발되어, 임상적으로 의미 있는 다중 배율 추론을 학습하는 능력을 제한한다. 또한, 단순하게 구성된 시각적 질의응답(VQA) 과제는 텍스트 전용 또는 피상적인 시각적 지름길(shortcut)에 취약할 수 있어, 시각적 이해에 대한 신뢰할 수 없는 평가로 이어진다. 이러한 한계를 해결하기 위해, 우리는 지름길에 강건한 교차 스케일 병리학 추론을 위한 벤치마크와 학습 프레임워크를 제안한다. 우리는 의미적 추론 질문을 위한 적대적 텍스트 전용 선별(Adversarial Text-only Screening) 전략과 시각적 그라운딩 질문을 위한 구조 제어형 방해 선택지 샘플링(Structure-controlled Distractor Sampling) 전략을 설계하여, 모델이 교차 스케일 시각적 증거에 의존하도록 유도한다. 이 파이프라인을 기반으로, 우리는 여러 배율 수준에 걸친 1,368개의 진단 경로에 기반한 10,373개의 객관식 질문을 포함하는 고품질 교차 스케일 병리학 VQA 벤치마크 PathScale-VQA를 구축한다. 의미적 추론 세트를 기반으로, PathScale-R1은 난이도 기반 추론 증류(Difficulty-driven Reasoning Distillation) 지도 미세조정과, 배율 간 증거 사용을 장려하는 스케일 인식 추론 구조(Scale-aware Reasoning Structure) 보상을 사용한 강화학습을 통해 최적화된다. 광범위한 실험은 교차 스케일 추론 과제에서 PathScale-R1의 최신(state-of-the-art) 성능과 기존 단일 스케일 병리학 VQA로의 효과적인 전이를 보여준다. 코드는 https://github.com/iMVR-PL/PathScale-R1에서 이용할 수 있다.

arXiv:2506.01850v2cs.CV

MoDA: 지시 기반 MLLM의 세밀한 비주얼 그라운딩을 위한 Modulation Adapter

MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs

멀티모달 대규모 언어 모델(MLLM)은 사전학습된 시각 인코더와 대규모 언어 모델(LLM)을 통합함으로써 지시 따르기(instruction-following) 과제에서 뛰어난 성과를 거두었다. 그러나 기존 접근법은 시각 패치 표현에서의 의미적 얽힘(semantic entanglement)으로 인해 세밀한 비주얼 그라운딩에 어려움을 겪는 경우가 많다. 개별 패치가 여러 개의 서로 다른 시각 요소를 혼합하여, 모델이 지시와 관련된 세부 사항에 집중하기 어렵게 만든다. 이 문제를 해결하기 위해, 우리는 지시 기반 채널별 변조(channel-wise modulation)를 통해 비주얼 그라운딩을 향상시키는 경량 모듈인 MoDA(Modulation Adapter)를 제안한다. Q-Former와 같이 가산적 특징 선택(additive feature selection)을 수행하는 토큰 수준 방법과 달리, MoDA는 이미 정렬된 특징에 대한 곱셈적 변조(multiplicative modulation)를 통해 채널 수준에서 동작하여, 각 지시에 대해 어떤 임베딩 차원이 관련 있는지를 세밀하게 제어할 수 있다. 표준 LLaVA 학습 프로토콜에 따라 MoDA는 언어 지시와 사전 정렬된 시각 특징 사이에 교차 어텐션(cross-attention)을 적용하여, 아키텍처 수정이나 추가 지도(supervision) 없이 동적 변조 마스크를 생성한다. 우리는 시각 질의응답(visual question answering), 비전 중심 추론(vision-centric reasoning), 환각(hallucination) 탐지에 걸친 12개 벤치마크(최근 2024 벤치마크 MMVP, CV-Bench, MMStar, RealWorldQA 포함)에서 MoDA를 평가하였으며, LLaVA-1.5, LLaVA-MoRE(2025), Qwen3-VL(2025)의 세 가지 서로 다른 MLLM 아키텍처에 대해 검증하였다. MoDA는 세 가지 계열 모두에서 일관된 향상을 보이며, LLaVA-1.5 계열에서는 MMVP에서 +12.0점, LLaVA-MoRE 계열에서는 ScienceQA에서 +4.8점, Qwen3-VL에서는 ScienceQA +4.9, RealWorldQA +4.1, GQA +3.8의 향상을 달성하여, CLIP 기반 인코더를 넘어 일반화되며 최소한의 오버헤드(<1% FLOPs)로 효과가 확인되었다. 코드는 https://github.com/waybarrios/MoDA 에서 제공된다.

arXiv:2002.11894

Unshuffling Data for Improved Generalization

Generalization beyond the training distribution is a core challenge in machine learning. The common practice of mixing and shuffling examples when training neural networks may not be optimal in this regard. We show that partitioning the data into well-chosen, non-i.i.d. subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization. We describe a training procedure to capture the patterns that are stable across environments while discarding spurious ones. The method makes a step beyond correlation-based learning: the choice of the partitioning allows injecting information about the task that cannot be otherwise recovered from the joint distribution of the training data. We demonstrate multiple use cases with the task of visual question answering, which is notorious for dataset biases. We obtain significant improvements on VQA-CP, using environments built from prior knowledge, existing meta data, or unsupervised clustering. We also get improvements on GQA using annotations of "equivalent questions", and on multi-dataset training (VQA v2 / Visual Genome) by treating them as distinct environments.

arXiv:2111.14725공식 코드 · Papers with Code

Searching the Search Space of Vision Transformer

Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures. In this paper, we propose to use neural architecture search to automate this process, by searching not only the architecture but also the search space. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using a weight-sharing supernet. Moreover, we provide design guidelines of general vision transformers with extensive analysis according to the space searching process, which could promote the understanding of vision transformer. Remarkably, the searched models, named S3 (short for Searching the Search Space), from the searched space achieve superior performance to recently proposed models, such as Swin, DeiT and ViT, when evaluated on ImageNet. The effectiveness of S3 is also illustrated on object detection, semantic segmentation and visual question answering, demonstrating its generality to downstream vision and vision-language tasks. Code and models will be available at https://github.com/microsoft/Cream.