PyTorchKR Papers

“Visual Question Answering”

총 2,804편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2203.12814공식 코드 · Papers with Code

Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering

Recent advances in Transformer architectures [1] have brought remarkable improvements to visual question answering (VQA). Nevertheless, Transformer-based VQA models are usually deep and wide to guarantee good performance, so they can only run on powerful GPU servers and cannot run on capacity-restricted platforms such as mobile phones. Therefore, it is desirable to learn an elastic VQA model that supports adaptive pruning at runtime to meet the efficiency constraints of different platforms. To this end, we present the bilaterally slimmable Transformer (BST), a general framework that can be seamlessly integrated into arbitrary Transformer-based VQA models to train a single model once and obtain various slimmed submodels of different widths and depths. To verify the effectiveness and generality of this method, we integrate the proposed BST framework with three typical Transformer-based VQA approaches, namely MCAN [2], UNITER [3], and CLIP-ViL [4], and conduct extensive experiments on two commonly-used benchmark datasets. In particular, one slimmed MCAN-BST submodel achieves comparable accuracy on VQA-v2, while being 0.38x smaller in model size and having 0.27x fewer FLOPs than the reference MCAN model. The smallest MCAN-BST submodel only has 9M parameters and 0.16G FLOPs during inference, making it possible to deploy it on a mobile device with less than 60 ms latency.

arXiv:2201.12926공식 코드 · Papers with Code

Compositionality as Lexical Symmetry

In tasks like semantic parsing, instruction following, and question answering, standard deep networks fail to generalize compositionally from small datasets. Many existing approaches overcome this limitation with model architectures that enforce a compositional process of sentence interpretation. In this paper, we present a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. Informally, we prove that whenever a task can be solved by a compositional model, there is a corresponding data augmentation scheme -- a procedure for transforming examples into other well formed examples -- that imparts compositional inductive bias on any model trained to solve the same task. We describe a procedure called LEXSYM that discovers these transformations automatically, then applies them to training data for ordinary neural sequence models. Unlike existing compositional data augmentation procedures, LEXSYM can be deployed agnostically across text, structured data, and even images. It matches or surpasses state-of-the-art, task-specific models on COGS semantic parsing, SCAN and ALCHEMY instruction following, and CLEVR-COGENT visual question answering datasets.

arXiv:2506.11684

MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in interpreting visual layouts and text. However, a significant challenge remains in their ability to interpret robustly and reason over multi-tabular data presented as images, a common occurrence in real-world scenarios like web pages and digital documents. Existing benchmarks typically address single tables or non-visual data (text/structured). This leaves a critical gap: they don't assess the ability to parse diverse table images, correlate information across them, and perform multi-hop reasoning on the combined visual data. We introduce MTabVQA, a novel benchmark specifically designed for multi-tabular visual question answering to bridge that gap. MTabVQA comprises 3,745 complex question-answer pairs that necessitate multi-hop reasoning across several visually rendered table images. We provide extensive benchmark results for state-of-the-art VLMs on MTabVQA, revealing significant performance limitations. We further investigate post-training techniques to enhance these reasoning abilities and release MTabVQA-Instruct, a large-scale instruction-tuning dataset. Our experiments show that fine-tuning VLMs with MTabVQA-Instruct substantially improves their performance on visual multi-tabular reasoning. Code and dataset (https://huggingface.co/datasets/mtabvqa/MTabVQA-Eval) are available online (https://anonymous.4open.science/r/MTabVQA-EMNLP-B16E).

arXiv:2608.21357v2cs.AI

VIALS: 생명과학 분야 시각적 아티팩트 해석 벤치마크

VIALS: A Benchmark for Visual Interpretation of Artifacts in the Life Sciences

전문적인 생명과학 워크플로에서 과학자들은 연구 의사결정에 활용하기 위해 시각적 아티팩트(젤 블롯, 현미경 이미지, 플라스미드 맵, 유세포 분석 플롯, 분자 구조 등)를 일상적으로 해석한다. 우리는 VIALS를 소개한다. VIALS는 161개의 이러한 해석 과제로 구성된 시각적 질의응답(VQA) 벤치마크이며, 출판물·교과서의 다듬어진 그림이 아니라 바이오텍 산업의 실험 워크플로 전반에서 검토되는 아티팩트 유형을 아우른다. 프론티어 비전-언어(vision-language) 모델은 이제 자연 이미지를 유창하게 기술할 수 있지만, 우리는 이들이 이러한 과학 이미지를 정확히 해석하지 못함을 확인했으며, 이는 도메인 지식과 도메인 특화 시각적 추론 능력의 한계를 반영한다. 반면, 관련 도메인 전문성을 가진 과학자들은 이러한 시각적 해석 과제를 간단하게 수행한다. 유사하게 이러한 이미지를 해석하지 못하는 AI는, 과학자들이 추론·소통·의사결정하는 방식의 중심에 이러한 아티팩트가 있는 전문 생명과학 워크플로에서 활용도가 제한적일 것이다.

arXiv:2002.05556공식 코드 · Papers with Code

Sparse and Structured Visual Attention

Visual attention mechanisms are widely used in multimodal tasks, as visual question answering (VQA). One drawback of softmax-based attention mechanisms is that they assign some probability mass to all image regions, regardless of their adjacency structure and of their relevance to the text. In this paper, to better link the image structure with the text, we replace the traditional softmax attention mechanism with two alternative sparsity-promoting transformations: sparsemax, which is able to select only the relevant regions (assigning zero weight to the rest), and a newly proposed Total-Variation Sparse Attention (TVmax), which further encourages the joint selection of adjacent spatial locations. Experiments in VQA show gains in accuracy as well as higher similarity to human attention, which suggests better interpretability.

arXiv:2208.10353공식 코드 · Papers with Code

Neuro-Symbolic Visual Dialog

We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog -a relative improvement of more than 10% over the state of the art while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.

arXiv:2011.11735공식 코드 · Papers with Code

Large Scale Multimodal Classification Using an Ensemble of Transformer Models and Co-Attention

Accurate and efficient product classification is significant for E-commerce applications, as it enables various downstream tasks such as recommendation, retrieval, and pricing. Items often contain textual and visual information, and utilizing both modalities usually outperforms classification utilizing either mode alone. In this paper we describe our methodology and results for the SIGIR eCom Rakuten Data Challenge. We employ a dual attention technique to model image-text relationships using pretrained language and image embeddings. While dual attention has been widely used for Visual Question Answering(VQA) tasks, ours is the first attempt to apply the concept for multimodal classification.

arXiv:2505.21755공식 코드 · Papers with Code

FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering

Visual question answering (VQA) systems face significant challenges when adapting to real-world data shifts, especially in multi-modal contexts. While robust fine-tuning strategies are essential for maintaining performance across in-distribution (ID) and out-of-distribution (OOD) scenarios, current evaluation settings are primarily unimodal or particular to some types of OOD, offering limited insight into the complexities of multi-modal contexts. In this work, we propose a new benchmark FRAMES-VQA (Fine-Tuning Robustness across Multi-Modal Shifts in VQA) for evaluating robust fine-tuning for VQA tasks. We utilize ten existing VQA benchmarks, including VQAv2, IV-VQA, VQA-CP, OK-VQA and others, and categorize them into ID, near and far OOD datasets covering uni-modal, multi-modal and adversarial distribution shifts. We first conduct a comprehensive comparison of existing robust fine-tuning methods. We then quantify the distribution shifts by calculating the Mahalanobis distance using uni-modal and multi-modal embeddings extracted from various models. Further, we perform an extensive analysis to explore the interactions between uni- and multi-modal shifts as well as modality importance for ID and OOD samples. These analyses offer valuable guidance on developing more robust fine-tuning methods to handle multi-modal distribution shifts. The code is available at https://github.com/chengyuehuang511/FRAMES-VQA .

arXiv:2211.07441공식 코드 · Papers with Code

Multi-VQG: Generating Engaging Questions for Multiple Images

Generating engaging content has drawn much recent attention in the NLP community. Asking questions is a natural way to respond to photos and promote awareness. However, most answers to questions in traditional question-answering (QA) datasets are factoids, which reduce individuals' willingness to answer. Furthermore, traditional visual question generation (VQG) confines the source data for question generation to single images, resulting in a limited ability to comprehend time-series information of the underlying event. In this paper, we propose generating engaging questions from multiple images. We present MVQG, a new dataset, and establish a series of baselines, including both end-to-end and dual-stage architectures. Results show that building stories behind the image sequence enables models to generate engaging questions, which confirms our assumption that people typically construct a picture of the event in their minds before asking questions. These results open up an exciting challenge for visual-and-language models to implicitly construct a story behind a series of photos to allow for creativity and experience sharing and hence draw attention to downstream applications.

arXiv:2608.06972v2cs.CV

생성형 임베딩 벤치마크: 조밀한 임베딩에는 얼마나 많은 정보가 보존되는가?

Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

임베딩(embedding)은 파운데이션 모델과 다운스트림 시스템을 연결하는 표준 표현 인터페이스로 자리 잡았습니다. 대부분의 임베딩 벤치마크(benchmark)는 판별 작업이나 임베딩 공간의 분리도(separability)를 중심으로 한 기하학적 기준을 통해 표현을 평가(evaluation)합니다. 그러나 이러한 평가에서 우수한 성능을 보인다고 해서 임베딩으로 압축된 콘텐츠가 다운스트림 생성기에서 여전히 접근 가능한 상태로 남아 있는지는 입증되지 않습니다. 이러한 격차를 해결하기 위해 본 논문에서는 원본 이미지나 중간 시각적 특징에 대한 접근 없이 고정된 임베딩과 질문 텍스트만을 사용하여 디코더가 질문에 답하는 생성형 임베딩 벤치마크(Generative Embedding Benchmark, GEB)를 소개합니다. 이 판독(readout) 방식 하에서의 답변 품질은 생성 정보(generative information), 즉 임베딩에서 복구 가능한 답변 관련 콘텐츠를 측정합니다. GEB에는 자연 이미지, 장면 텍스트, 시각적 문서를 다루는 1,800개 항목의 개발 분할과 900개 항목의 별도 테스트 분할로 구성된 큐레이션된 시각적 질문 답변(VQA) 데이터셋(dataset)이 포함되어 있습니다. 공통 디코더와 학습(training) 레시피를 사용하여 시각 전용 모드 및 비전-언어(vision-language) 결합 모드에서 7개의 공개 임베딩 모델을 평가합니다. 테스트 세트에서 시각 전용 점수는 28.25에서 33.21 사이였으며, 이미지-질문 결합 인코딩을 사용할 경우 5개의 VLM 기반 임베딩 모델 모두 더 높은 점수를 기록했고 최고 점수는 65.56에 달했습니다. 일치하는 임베딩은 텍스트 전용 입력, 제로 임베딩, 셔플된 임베딩보다 더 나은 성능을 보였습니다. 자연 이미지 정보는 장면 텍스트나 시각적 문서 정보보다 훨씬 복구하기 쉬웠던 반면, 원본 이미지에 접근할 수 있는 Qwen3-VL-2B 참조 모델은 84.30에 도달했습니다. 종합적으로 이러한 결과는 생성형 판독 방식이 분리도 기반 평가에서는 포착하지 못하는 정보 병목 현상을 드러냄을 보여줍니다.

arXiv:2408.12763

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

Multimodal large language models (MLLMs) can simultaneously process visual, textual, and auditory data, capturing insights that complement human analysis. However, existing video question-answering (VidQA) benchmarks and datasets often exhibit a bias toward a single modality, despite the goal of requiring advanced reasoning skills that integrate diverse modalities to answer the queries. In this work, we introduce the modality importance score (MIS) to identify such bias. It is designed to assess which modality embeds the necessary information to answer the question. Additionally, we propose an innovative method using state-of-the-art MLLMs to estimate the modality importance, which can serve as a proxy for human judgments of modality perception. With this MIS, we demonstrate the presence of unimodal bias and the scarcity of genuinely multimodal questions in existing datasets. We further validate the modality importance score with multiple ablation studies to evaluate the performance of MLLMs on permuted feature sets. Our results indicate that current models do not effectively integrate information due to modality imbalance in existing datasets. Our proposed MLLM-derived MIS can guide the curation of modality-balanced datasets that advance multimodal learning and enhance MLLMs' capabilities to understand and utilize synergistic relations across modalities.

arXiv:2207.08739공식 코드 · Papers with Code

Rethinking Data Augmentation for Robust Visual Question Answering

Data Augmentation (DA) -- generating extra training samples beyond original training set -- has been widely-used in today's unbiased VQA models to mitigate the language biases. Current mainstream DA strategies are synthetic-based methods, which synthesize new samples by either editing some visual regions/words, or re-generating them from scratch. However, these synthetic samples are always unnatural and error-prone. To avoid this issue, a recent DA work composes new augmented samples by randomly pairing pristine images and other human-written questions. Unfortunately, to guarantee augmented samples have reasonable ground-truth answers, they manually design a set of heuristic rules for several question types, which extremely limits its generalization abilities. To this end, we propose a new Knowledge Distillation based Data Augmentation for VQA, dubbed KDDAug. Specifically, we first relax the requirements of reasonable image-question pairs, which can be easily applied to any question types. Then, we design a knowledge distillation (KD) based answer assignment to generate pseudo answers for all composed image-question pairs, which are robust to both in-domain and out-of-distribution settings. Since KDDAug is a model-agnostic DA strategy, it can be seamlessly incorporated into any VQA architectures. Extensive ablation studies on multiple backbones and benchmarks have demonstrated the effectiveness and generalization abilities of KDDAug.

arXiv:2310.16527

Enhancing Document Information Analysis with Multi-Task Pre-training: A Robust Approach for Information Extraction in Visually-Rich Documents

This paper introduces a deep learning model tailored for document information analysis, emphasizing document classification, entity relation extraction, and document visual question answering. The proposed model leverages transformer-based models to encode all the information present in a document image, including textual, visual, and layout information. The model is pre-trained and subsequently fine-tuned for various document image analysis tasks. The proposed model incorporates three additional tasks during the pre-training phase, including reading order identification of different layout segments in a document image, layout segments categorization as per PubLayNet, and generation of the text sequence within a given layout segment (text block). The model also incorporates a collective pre-training scheme where losses of all the tasks under consideration, including pre-training and fine-tuning tasks with all datasets, are considered. Additional encoder and decoder blocks are added to the RoBERTa network to generate results for all tasks. The proposed model achieved impressive results across all tasks, with an accuracy of 95.87% on the RVL-CDIP dataset for document classification, F1 scores of 0.9306, 0.9804, 0.9794, and 0.8742 on the FUNSD, CORD, SROIE, and Kleister-NDA datasets respectively for entity relation extraction, and an ANLS score of 0.8468 on the DocVQA dataset for visual question answering. The results highlight the effectiveness of the proposed model in understanding and interpreting complex document layouts and content, making it a promising tool for document analysis tasks.

arXiv:2611.22329v1cs.CV

M$^3$-Test: 시각 질의응답의 복합 교란 테스트가 놓치는 사례

M$^3$-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering

시각 질의응답(VQA) 모델은 이미지와 질문을 각각 바꾸는 테스트를 모두 통과해도 두 변경을 동시에 적용하면 실패할 수 있다. 표준 강건성 보고는 여러 조건의 평균을 내므로 테스트를 통과한 요청 중 어떤 요청이 여전히 취약한지 알려 주지 못한다. 테스트 범위를 점검하는 M3-Test를 소개한다. 교란 없는 각 입력을 단일 모달리티 변형 및 5×5 변형 조합과 4×4 강도 조합을 빠짐없이 포함한 복합 교란 라이브러리에 연결한다. 따라서 축소한 테스트 모음도 동일한 요청에서 재실행할 수 있다. 비전-언어 모델 네 개의 응답 260만 건을 분석했다. Qwen에서는 단일 모달리티 테스트 40개를 모두 통과한 질문 중 26~29%에서 복합 교란으로 실패가 발생했다. 개별 복합 교란 기록의 실패율은 0.6% 미만이었다. 최대 강도 조합 25개를 통과해도 격차는 남았다. 이 조건을 통과한 질문 중 11~14%는 라이브러리의 다른 조합에서 실패했다. 보관한 실패 사례를 다시 실행하면 약 90%에서 실패가 재현됐다. 조건을 맞춘 통과 대조군에서 결과가 뒤집힌 비율은 0.3~0.6%였다. 결과가 가장 쉽게 달라지는 실패 기록을 재검사했을 때도 80~89%에서 실패가 뒷받침됐다. 조사한 근거에 따르면 실행 잡음과 채점 오류만으로는 이 격차를 모두 설명할 수 없다. 정렬된 격자를 사용하면 이전 평가를 재사용할 수 있다. 따로 보관한 이미지에서 다른 모델로부터 학습한 위험도 순서로 쿼리를 추가하면 25회 이내에 영향을 받은 Qwen 질문의 약 80%를 찾았다. 최대 강도 우선은 65~68%, 쌍별 커버링 배열은 53~57%였다. M3-Test는 강건성 테스트를 통과한 요청마다 여전히 평가하지 못한 조건을 보여 줘, 수락 여부 판단, 회귀 검토, 테스트 배분을 지원한다.

arXiv:2607.00159v1cs.CL

Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting

기존 KB-VQA 벤치마크(InfoSeek, E-VQA)가 답변 도출 불가, 불명확한 질문, 단일 개체 중심의 시각적 단순화 등 체계적 결함을 내포하고 있음을 감사(audit)를 통해 밝히고, 이를 해결하기 위한 정제 프로토콜과 다중 개체 증강 파이프라인을 제안하여 재평가 시 기존과 상이한 모델 성능 순위 및 추론 능력 측정치를 도출했다.

arXiv:2009.10054

Regularizing Attention Networks for Anomaly Detection in Visual Question Answering

For stability and reliability of real-world applications, the robustness of DNNs in unimodal tasks has been evaluated. However, few studies consider abnormal situations that a visual question answering (VQA) model might encounter at test time after deployment in the real-world. In this study, we evaluate the robustness of state-of-the-art VQA models to five different anomalies, including worst-case scenarios, the most frequent scenarios, and the current limitation of VQA models. Different from the results in unimodal tasks, the maximum confidence of answers in VQA models cannot detect anomalous inputs, and post-training of the outputs, such as outlier exposure, is ineffective for VQA models. Thus, we propose an attention-based method, which uses confidence of reasoning between input images and questions and shows much more promising results than the previous methods in unimodal tasks. In addition, we show that a maximum entropy regularization of attention networks can significantly improve the attention-based anomaly detection of the VQA models. Thanks to the simplicity, attention-based anomaly detection and the regularization are model-agnostic methods, which can be used for various cross-modal attentions in the state-of-the-art VQA models. The results imply that cross-modal attention in VQA is important to improve not only VQA accuracy, but also the robustness to various anomalies.

arXiv:2307.12045공식 코드 · Papers with Code

Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery

The visual-question localized-answering (VQLA) system can serve as a knowledgeable assistant in surgical education. Except for providing text-based answers, the VQLA system can highlight the interested region for better surgical scene understanding. However, deep neural networks (DNNs) suffer from catastrophic forgetting when learning new knowledge. Specifically, when DNNs learn on incremental classes or tasks, their performance on old tasks drops dramatically. Furthermore, due to medical data privacy and licensing issues, it is often difficult to access old data when updating continual learning (CL) models. Therefore, we develop a non-exemplar continual surgical VQLA framework, to explore and balance the rigidity-plasticity trade-off of DNNs in a sequential learning paradigm. We revisit the distillation loss in CL tasks, and propose rigidity-plasticity-aware distillation (RP-Dist) and self-calibrated heterogeneous distillation (SH-Dist) to preserve the old knowledge. The weight aligning (WA) technique is also integrated to adjust the weight bias between old and new tasks. We further establish a CL framework on three public surgical datasets in the context of surgical settings that consist of overlapping classes between old and new surgical VQLA tasks. With extensive experiments, we demonstrate that our proposed method excellently reconciles learning and forgetting on the continual surgical VQLA over conventional CL methods. Our code is publicly accessible.

arXiv:2309.08798공식 코드 · Papers with Code

D3: Data Diversity Design for Systematic Generalization in Visual Question Answering

Systematic generalization is a crucial aspect of intelligence, which refers to the ability to generalize to novel tasks by combining known subtasks and concepts. One critical factor that has been shown to influence systematic generalization is the diversity of training data. However, diversity can be defined in various ways, as data have many factors of variation. A more granular understanding of how different aspects of data diversity affect systematic generalization is lacking. We present new evidence in the problem of Visual Question Answering (VQA) that reveals that the diversity of simple tasks (i.e. tasks formed by a few subtasks and concepts) plays a key role in achieving systematic generalization. This implies that it may not be essential to gather a large and varied number of complex tasks, which could be costly to obtain. We demonstrate that this result is independent of the similarity between the training and testing data and applies to well-known families of neural network architectures for VQA (i.e. monolithic architectures and neural module networks). Additionally, we observe that neural module networks leverage all forms of data diversity we evaluated, while monolithic architectures require more extensive amounts of data to do so. These findings provide a first step towards understanding the interactions between data diversity design, neural network architectures, and systematic generalization capabilities.

arXiv:2606.19584v1cs.CV

제어 가능하고 일반화 가능한 지각을 위한 언어 지시 비전 임베딩

Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

비전 파운데이션 모델은 일반적으로 정적 특징 추출기로 학습되어, 작업 적응의 부담이 큰 다운스트림 모델로 전가됩니다. 우리는 대안 패러다임을 제안합니다: 시각적 특징을 언어 모델에 단순히 공급하는 대신, 언어 자체를 사용하여 비전 인코더를 동적으로 안내합니다. 우리의 방법인 언어 지시 비전 임베딩(LIVE)은 언어를 고수준 지시로 활용하여 추론 시간에 작업 중심 임베딩을 생성함으로써, 작업별 재학습의 필요성을 제거합니다. 이를 통해 인코더는 입력의 문맥적으로 관련된 측면에 집중할 수 있게 되어, 더 제어 가능하고 일반화 가능한 표현을 만들어냅니다. 실증적으로, LIVE는 시각적 환각을 줄이고(MMVP에서 34점 향상), 시각적 질의 응답에서 매개변수가 자릿수만큼 더 많은 비전-언어 모델을 능가하며, 보이지 않는 지시와 작업으로 일반화되어 적응형이고 지시 기반의 비전 지능으로 가는 직접적인 경로를 제공합니다.

arXiv:2302.02117

Learning to Agree on Vision Attention for Visual Commonsense Reasoning

Visual Commonsense Reasoning (VCR) remains a significant yet challenging research problem in the realm of visual reasoning. A VCR model generally aims at answering a textual question regarding an image, followed by the rationale prediction for the preceding answering process. Though these two processes are sequential and intertwined, existing methods always consider them as two independent matching-based instances. They, therefore, ignore the pivotal relationship between the two processes, leading to sub-optimal model performance. This paper presents a novel visual attention alignment method to efficaciously handle these two processes in a unified framework. To achieve this, we first design a re-attention module for aggregating the vision attention map produced in each process. Thereafter, the resultant two sets of attention maps are carefully aligned to guide the two processes to make decisions based on the same image regions. We apply this method to both conventional attention and the recent Transformer models and carry out extensive experiments on the VCR benchmark dataset. The results demonstrate that with the attention alignment module, our method achieves a considerable improvement over the baseline methods, evidently revealing the feasibility of the coupling of the two processes as well as the effectiveness of the proposed method.