PyTorchKR Papers

“Visual Question Answering”

총 2,807편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2607.29637v1cs.CV

CodeShrink: 효율적인 멀티모달 코드 이해를 위한 적응형 시각적 압축

CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding

소스 코드를 이미지로 렌더링하는 것은 멀티모달 대규모 언어 모델(MLLM)의 입력 비용을 줄이는 유망한 방안을 제공합니다. 이미지 해상도를 조절하면 시각적 토큰(token) 비용과 콘텐츠 충실도 간의 절충(trade-off)을 조율할 수 있습니다. 그러나 해상도 스케일링(scaling)만으로는 두 가지 비효율 원인을 간과하게 됩니다. 바로 줄 바꿈과 들여쓰기로 인해 발생하는 공백 영역과 현재 지시사항과 무관한 코드 영역입니다. 또한 최적의 압축 설정은 입력, 작업, 모델에 따라 달라지므로 고정 비율 전략에는 한계가 있습니다. 본 논문에서는 세 가지 구성 요소로 이루어진 적응형 시각적 압축 프레임워크인 CodeShrink를 제안합니다. 공백 제거 렌더링(Blank-Free Rendering)은 공백 의존적 레이아웃을 조밀한 레이아웃과 명시적 구조 마커로 대체하여 레이아웃으로 인해 발생하는 토큰을 제거합니다. 적응형 압축 구성(Adaptive Compression Configuration)은 강화학습(reinforcement learning)으로 학습된 경량 에이전트(agent)를 활용하여 토큰 효율성과 가독성의 균형을 맞추는 입력별 설정을 예측합니다. 주요 토큰 선택(Dominant Token Selection)은 지시사항과 코드 이미지를 함께 분석하여 추론(inference) 과정에서 작업과 무관한 시각적 토큰을 가지치기(pruning)합니다. 저자들은 코드 질의응답, 클론 탐지, 코드 완성 작업에서 CodeShrink를 평가합니다. CodeShrink는 압축되지 않은 텍스트 전용 입력과 대등하거나 이를 능가하면서도 시각적 토큰 사용량을 최대 71.2%까지 줄이며, 세 가지 작업 모두에서 텍스트 기반 및 시각적 압축 베이스라인을 일관되게 능가합니다. 이러한 결과는 레이아웃 압축, 적응형 구성, 지시사항 인식 가지치기를 결합함으로써 멀티모달 코드 이해를 더욱 효율적으로 만들 수 있음을 보여줍니다. 코드는 https://github.com/vinsontang1/CodeShrink 에서 확인할 수 있습니다.

arXiv:2310.06238공식 코드 · Papers with Code

Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering

In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, strong bias that exists in any modality can lead to the model neglecting the others. Consequently, the model's ability to effectively reason across these diverse modalities is compromised, impeding further advancement. In this paper, we meticulously review each question type from the original dataset, selecting those with pronounced answer biases. To counter these biases, we gather complementary videos and questions, ensuring that no answers have outstanding skewed distribution. In particular, for binary questions, we strive to ensure that both answers are almost uniformly spread within each question category. As a result, we construct a new dataset, named MUSIC-AVQA v2.0, which is more challenging and we believe could better foster the progress of AVQA task. Furthermore, we present a novel baseline model that delves deeper into the audio-visual-text interrelation. On MUSIC-AVQA v2.0, this model surpasses all the existing benchmarks, improving accuracy by 2% on MUSIC-AVQA v2.0, setting a new state-of-the-art performance.

arXiv:2002.10695

Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge

Audio-Visual Scene-Aware Dialog (AVSD) is an extension from Video Question Answering (QA) whereby the dialogue agent is required to generate natural language responses to address user queries and carry on conversations. This is a challenging task as it consists of video features of multiple modalities, including text, visual, and audio features. The agent also needs to learn semantic dependencies among user utterances and system responses to make coherent conversations with humans. In this work, we describe our submission to the AVSD track of the 8th Dialogue System Technology Challenge. We adopt dot-product attention to combine text and non-text features of input video. We further enhance the generation capability of the dialogue agent by adopting pointer networks to point to tokens from multiple source sequences in each generation step. Our systems achieve high performance in automatic metrics and obtain 5th and 6th place in human evaluation among all submissions.

arXiv:2209.10126공식 코드 · Papers with Code

Exploring Modulated Detection Transformer as a Tool for Action Recognition in Videos

During recent years transformers architectures have been growing in popularity. Modulated Detection Transformer (MDETR) is an end-to-end multi-modal understanding model that performs tasks such as phase grounding, referring expression comprehension, referring expression segmentation, and visual question answering. One remarkable aspect of the model is the capacity to infer over classes that it was not previously trained for. In this work we explore the use of MDETR in a new task, action detection, without any previous training. We obtain quantitative results using the Atomic Visual Actions dataset. Although the model does not report the best performance in the task, we believe that it is an interesting finding. We show that it is possible to use a multi-modal model to tackle a task that it was not designed for. Finally, we believe that this line of research may lead into the generalization of MDETR in additional downstream tasks.

arXiv:2311.18651공식 코드 · Papers with Code

LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.

arXiv:2501.02669공식 코드 · Papers with Code

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?

While Vision Language Models (VLMs) are impressive in tasks such as visual question answering (VQA) and image captioning, their ability to apply multi-step reasoning to images has lagged, giving rise to perceptions of modality imbalance or brittleness. Towards systematic study of such issues, we introduce a synthetic framework for assessing the ability of VLMs to perform algorithmic visual reasoning (AVR), comprising three tasks: Table Readout, Grid Navigation, and Visual Analogy. Each has two levels of difficulty, SIMPLE and HARD, and even the SIMPLE versions are difficult for frontier VLMs. We seek strategies for training on the SIMPLE version of the tasks that improve performance on the corresponding HARD task, i.e., S2H generalization. This synthetic framework, where each task also has a text-only version, allows a quantification of the modality imbalance, and how it is impacted by training strategy. Ablations highlight the importance of explicit image-to-text conversion in promoting S2H generalization when using auto-regressive training. We also report results of mechanistic study of this phenomenon, including a measure of gradient alignment that seems to identify training strategies that promote better S2H generalization.

arXiv:2503.13139

Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding

Understanding long video content is a complex endeavor that often relies on densely sampled frame captions or end-to-end feature selectors, yet these techniques commonly overlook the logical relationships between textual queries and visual elements. In practice, computational constraints necessitate coarse frame subsampling, a challenge analogous to "finding a needle in a haystack." To address this issue, we introduce a semantics-driven search framework that reformulates keyframe selection under the paradigm of Visual Semantic-Logical Search. Specifically, we systematically define four fundamental logical dependencies: 1) spatial co-occurrence, 2) temporal proximity, 3) attribute dependency, and 4) causal order. These relations dynamically update frame sampling distributions through an iterative refinement process, enabling context-aware identification of semantically critical frames tailored to specific query requirements. Our method establishes new SOTA performance on the manually annotated benchmark in key-frame selection metrics. Furthermore, when applied to downstream video question-answering tasks, the proposed approach demonstrates the best performance gains over existing methods on LongVideoBench and Video-MME, validating its effectiveness in bridging the logical gap between textual queries and visual-temporal reasoning. The code will be publicly available.

arXiv:2503.15887

DocVideoQA: Towards Comprehensive Understanding of Document-Centric Videos through Question Answering

Remote work and online courses have become important methods of knowledge dissemination, leading to a large number of document-based instructional videos. Unlike traditional video datasets, these videos mainly feature rich-text images and audio that are densely packed with information closely tied to the visual content, requiring advanced multimodal understanding capabilities. However, this domain remains underexplored due to dataset availability and its inherent complexity. In this paper, we introduce the DocVideoQA task and dataset for the first time, comprising 1454 videos across 23 categories with a total duration of about 828 hours. The dataset is annotated with 154k question-answer pairs generated manually and via GPT, assessing models' comprehension, temporal awareness, and modality integration capabilities. Initially, we establish a baseline using open-source MLLMs. Recognizing the challenges in modality comprehension for document-centric videos, we present DV-LLaMA, a robust video MLLM baseline. Our method enhances unimodal feature extraction with diverse instruction-tuning data and employs contrastive learning to strengthen modality integration. Through fine-tuning, the LLM is equipped with audio-visual capabilities, leading to significant improvements in document-centric video understanding. Extensive testing on the DocVideoQA dataset shows that DV-LLaMA significantly outperforms existing models. We'll release the code and dataset to facilitate future research.

arXiv:2408.12528

Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model. Code and models are released at https://github.com/showlab/Show-o.

arXiv:2607.15565v2cs.CV

두 번 묻고, 두 번 보라: 프롬프트 에코잉을 통한 비전-언어 모델의 질문 우선 역설 해결

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

비전-언어 모델(VLM) 프롬프트에서 질문은 이미지 앞과 뒤 중 어디에 위치해야 할까요? 직관적으로는 앞이어야 합니다. 무엇을 묻는지 알아야 모델이 어디를 보아야 할지 알 수 있기 때문입니다. 그러나 여러 시각적 질의응답(VQA) 벤치마크 전반에서 질문을 먼저 배치하는 프롬프팅은 프론티어 VLM에서 권장되는 이미지를 먼저 배치하는 순서보다 일관되게 낮은 성능을 보이며, 저자들은 이 현상을 '질문 우선 역설(question-first paradox)'이라 부릅니다. 저자들은 이 역설의 원인을 VLM 연산의 두 단계 사이의 충돌에서 찾았습니다. 로짓 렌즈(logit-lens) 및 어텐션 프로브에 따르면 질문 우선 프롬프팅은 지각을 유도하여 이미지 패치 표현을 질문과 관련된 개념 쪽으로 이동시킵니다. 그러나 다운스트림에서는 수백 개의 이미지 토큰 뒤에 고립되어, 답변 토큰이 질문에 거의 어텐션을 기울이지 못하고 대신 이미지에만 의존한 채 종종 잘못된 답변을 내놓게 됩니다. 인과적 어텐션 녹아웃(causal attention knockout)을 통해 답변은 질문이 이미지 뒤에 올 때만 질문을 읽는다는 점을 확인했습니다. 이러한 진단을 바탕으로 저자들은 학습이 필요 없는(training-free) 해결책인 '질문 에코잉(question echoing)'을 제시합니다. 즉, 이미지의 양쪽에 질문을 다시 기술하여 하나의 복사본은 지각을 유도하고 다른 하나는 답변 시점에 참조할 수 있도록 하는 것입니다. 이와 유사한 역할 분담은 50년 전 인간의 '부속 질문(adjunct questions)'에 대한 연구 결과에서도 발견되는데, 지문 앞뒤로 질문을 반복하는 것이 이해도를 향상시킨다는 점입니다. 이미지 또한 에코잉하면 인과적 디코더에 의해 손실될 수 있는 전체 이미지 뷰를 복원하여 추가적인 성능 향상을 이끌어냅니다. 이 역설은 5개의 오픈 VLM 전반에서 성립하며, 최대 17.5 그룹 정확도(group-accuracy) 포인트의 손실을 초래합니다. 에코잉된 프롬프트는 이러한 격차의 대부분을 회복하며, NaturalBench 및 Winoground에서 Winoground 기준 최대 19 그룹 정확도 포인트까지 최고 성능의 단일 패스 순서를 뛰어넘으며, 학습이나 미세조정(fine-tuning), 또는 아키텍처 변경 없이 이를 달성합니다. 이 역설은 모델이 무엇을 보는지 유도하는 것과 질문받은 내용에 대한 접근성을 유지하는 것 사이의 긴장 관계를 드러내며, 에코잉은 프롬프트 설계를 통해 이를 해결합니다. 프로젝트 페이지: https://rakshanda-cmu.github.io/ask-twice-look-twice/

arXiv:1803.05268공식 코드 · Papers with Code

Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning

Visual question answering requires high-order reasoning about an image, which is a fundamental capability needed by machine systems to follow complex directives. Recently, modular networks have been shown to be an effective framework for performing visual reasoning tasks. While modular networks were initially designed with a degree of model transparency, their performance on complex visual reasoning benchmarks was lacking. Current state-of-the-art approaches do not provide an effective mechanism for understanding the reasoning process. In this paper, we close the performance gap between interpretable models and state-of-the-art visual reasoning methods. We propose a set of visual-reasoning primitives which, when composed, manifest as a model capable of performing complex reasoning tasks in an explicitly-interpretable manner. The fidelity and interpretability of the primitives' outputs enable an unparalleled ability to diagnose the strengths and weaknesses of the resulting model. Critically, we show that these primitives are highly performant, achieving state-of-the-art accuracy of 99.1% on the CLEVR dataset. We also show that our model is able to effectively learn generalized representations when provided a small amount of data containing novel object attributes. Using the CoGenT generalization task, we show more than a 20 percentage point improvement over the current state of the art.

arXiv:2409.19232

TrojVLM: Backdoor Attack Against Vision Language Models

The emergence of Vision Language Models (VLMs) is a significant advancement in integrating computer vision with Large Language Models (LLMs) to produce detailed text descriptions based on visual inputs, yet it introduces new security vulnerabilities. Unlike prior work that centered on single modalities or classification tasks, this study introduces TrojVLM, the first exploration of backdoor attacks aimed at VLMs engaged in complex image-to-text generation. Specifically, TrojVLM inserts predetermined target text into output text when encountering poisoned images. Moreover, a novel semantic preserving loss is proposed to ensure the semantic integrity of the original image content. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of TrojVLM in maintaining original semantic content while triggering specific target text outputs. This study not only uncovers a critical security risk in VLMs and image-to-text generation but also sets a foundation for future research on securing multimodal models against such sophisticated threats.

arXiv:2505.15687

Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoning

Multimodal pathological image understanding has garnered widespread interest due to its potential to improve diagnostic accuracy and enable personalized treatment through integrated visual and textual data. However, existing methods exhibit limited reasoning capabilities, which hamper their ability to handle complex diagnostic scenarios. Additionally, the enormous size of pathological images leads to severe computational burdens, further restricting their practical deployment. To address these limitations, we introduce a novel bilateral reinforcement learning framework comprising two synergistic branches. One reinforcement branch enhances the reasoning capability by enabling the model to learn task-specific decision processes, i.e., pathology rationales, directly from labels without explicit reasoning supervision. While the other branch dynamically allocates a tailored number of tokens to different images based on both their visual content and task context, thereby optimizing computational efficiency. We apply our method to various pathological tasks such as visual question answering, cancer subtyping, and lesion detection. Extensive experiments show an average +41.7 absolute performance improvement with 70.3% lower inference costs over the base models, achieving both reasoning accuracy and computational efficiency.

arXiv:2611.15418v1cs.CV

ChartGalaxy++: 차트 이해와 생성을 위한 풍부한 주석 데이터셋

ChartGalaxy++: A Richly Annotated Dataset for Chart Understanding and Generation

인포그래픽 차트는 차트, 텍스트, 이미지 등 시각 요소들의 응집력 있는 구성을 통해 데이터를 전달한다. 기존 인포그래픽 차트 데이터셋은 주로 개별 요소에 경계 상자와 범주 레이블을 주석으로 부여한다. 이러한 요소 수준의 주석은 계층적·공간적 관계를 포착하지 못하므로, 인포그래픽 내에서 요소들이 어떻게 조직되고 서로 연관되는지를 이해하거나 잘 조직된 디자인을 생성하는 데 활용하기에 한계가 있다. 이러한 한계를 해결하기 위해 본 논문은 217,195개의 인포그래픽 차트로 구성된 풍부한 주석 데이터셋 ChartGalaxy++를 소개한다. 이 데이터셋의 핵심은 장면 그래프 표현으로, 노드는 시각 요소와 의미적 그룹을 나타내고 에지는 이들의 계층적·공간적 관계를 인코딩한다. 실제 인포그래픽의 경우 시각 요소를 검출하고 점진적으로 그룹화하여 장면 그래프를 구축하며, 합성 인포그래픽의 경우 생성 과정에서 직접 장면 그래프를 도출한다. 본 논문은 1) 이미지-장면 그래프 예측, 2) 장면 그래프로 보강한 인포그래픽 질의응답, 3) 이미지 생성에서의 장면 그래프 보존 평가라는 세 가지 응용을 통해 ChartGalaxy++의 유용성을 입증한다. 이들 응용은 종합적으로 차트 이해와 생성의 발전에 ChartGalaxy++가 지니는 가치를 보여준다.

arXiv:2610.10238v2cs.CV

의료 VLM에서 다중 표현형 병변 해석을 위한 기하 지도 시각 표현 학습

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs

의료 비전-언어 모델(VLM)은 임상 영상 해석에서 점차 높은 잠재력을 보여주고 있습니다. 그러나 이러한 모델들은 여러 병리학적 표현형(phenotype)의 공동 평가가 진단에 요구되는 다중 표현형 병변을 해석하는 데 여전히 어려움을 겪고 있습니다. 기존의 비전-언어 정렬(alignment) 방법으로 생성된 시각 표현은 해부학적 위계와 표현형 하위 클래스 간의 관계를 제대로 보존하지 못합니다. 이는 시각 임베딩(embedding) 공간에서 이러한 관계를 보존하기 위한 기하학적 제약 조건이 결여된 의미론적 지도(semantic supervision)에 의존하기 때문입니다. 더욱이 병변 관련 해부학적 및 표현형 표현의 희소성(sparsity)으로 인해 의료 VLM이 중요한 진단적 증거를 포착하기 어렵습니다. 이러한 한계를 해결하기 위해, 본 논문에서는 의료 VLM의 다중 표현형 병변 해석을 위한 기하 지도 시각 표현 학습 프레임워크인 \textbf{PureVision}을 제안합니다. 이는 기하 지도 표현 학습 모듈인 \textbf{PureEyes}와 해부학 유도 증거 집약 모듈인 \textbf{PureNeurons}를 결합합니다. PureEyes는 해부학적 위계와 표현형 하위 클래스 관계를 인코딩하는 이상적인 공간 분포를 통해 기하학적 지도를 제공합니다. PureNeurons는 시각 표현을 학습된 잠재 공간에 투영하고, 해당 위치를 활용하여 병변 특이적 해부학적 및 표현형 증거를 선택적으로 집약합니다. \textit{LIDC-IDRI}, \textit{CBIS-DDSM}, \textit{3DReasonKnee}에 대한 실험을 통해 PureVision이 시각적 질의응답(VQA) 및 방사선학 보고서 생성에서 병변 그라운딩(grounding)과 표현형 특성화를 개선함을 입증합니다. 코드는 https://anonymous.4open.science/r/purevision-06C2 에서 확인할 수 있습니다.

arXiv:2410.09379공식 코드 · Papers with Code

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

arXiv:1612.05386

The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions

One of the most intriguing features of the Visual Question Answering (VQA) challenge is the unpredictability of the questions. Extracting the information required to answer them demands a variety of image operations from detection and counting, to segmentation and reconstruction. To train a method to perform even one of these operations accurately from {image,question,answer} tuples would be challenging, but to aim to achieve them all with a limited set of such training data seems ambitious at best. We propose here instead a more general and scalable approach which exploits the fact that very good methods to achieve these operations already exist, and thus do not need to be trained. Our method thus learns how to exploit a set of external off-the-shelf algorithms to achieve its goal, an approach that has something in common with the Neural Turing Machine. The core of our proposed method is a new co-attention model. In addition, the proposed approach generates human-readable reasons for its decision, and can still be trained end-to-end without ground truth reasons being given. We demonstrate the effectiveness on two publicly available datasets, Visual Genome and VQA, and show that it produces the state-of-the-art results in both cases.

arXiv:2401.17851공식 코드 · Papers with Code

Instruction-Guided Scene Text Recognition

Multi-modal models have shown appealing performance in visual recognition tasks, as free-form text-guided training evokes the ability to understand fine-grained visual content. However, current models cannot be trivially applied to scene text recognition (STR) due to the compositional difference between natural and text images. We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc. IGTR first devises $\left \langle condition,question,answer\right \rangle$ instruction triplets, providing rich and diverse descriptions of character attributes. To effectively learn these attributes through question-answering, IGTR develops a lightweight instruction encoder, a cross-modal feature fusion module and a multi-task answer head, which guides nuanced text image understanding. Furthermore, IGTR realizes different recognition pipelines simply by using different instructions, enabling a character-understanding-based text reasoning paradigm that differs from current methods considerably. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins, while maintaining a small model size and fast inference speed. Moreover, by adjusting the sampling of instructions, IGTR offers an elegant way to tackle the recognition of rarely appearing and morphologically similar characters, which were previous challenges. Code: https://github.com/Topdu/OpenOCR.

arXiv:1807.03096공식 코드 · Papers with Code

NMT-Keras: a Very Flexible Toolkit with a Focus on Interactive NMT and Online Learning

We present NMT-Keras, a flexible toolkit for training deep learning models, which puts a particular emphasis on the development of advanced applications of neural machine translation systems, such as interactive-predictive translation protocols and long-term adaptation of the translation system via continuous learning. NMT-Keras is based on an extended version of the popular Keras library, and it runs on Theano and Tensorflow. State-of-the-art neural machine translation models are deployed and used following the high-level framework provided by Keras. Given its high modularity and flexibility, it also has been extended to tackle different problems, such as image and video captioning, sentence classification and visual question answering.

arXiv:2410.02713

Video Instruction Tuning With Synthetic Data

The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.