PyTorchKR Papers

최신 논문

총 461편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:1610.02947

End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

We propose a high-level concept word detector that can be integrated with any video-to-language models. It takes a video as input and generates a list of concept words as useful semantic priors for language generation models. The proposed word detector has two important properties. First, it does not require any external knowledge sources for training. Second, the proposed word detector is trainable in an end-to-end manner jointly with any video-to-language models. To maximize the values of detected words, we also develop a semantic attention mechanism that selectively focuses on the detected concept words and fuse them with the word encoding and decoding in the language model. In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks, we participate in four tasks of LSMDC 2016. Our approach achieves the best accuracies in three of them, including fill-in-the-blank, multiple-choice test, and movie retrieval. We also attain comparable performance for the other task, movie description.

arXiv:1707.09074

Learning from Video and Text via Large-Scale Discriminative Clustering

Discriminative clustering has been successfully applied to a number of weakly-supervised learning tasks. Such applications include person and action recognition, text-to-video alignment, object co-segmentation and colocalization in videos and images. One drawback of discriminative clustering, however, is its limited scalability. We address this issue and propose an online optimization algorithm based on the Block-Coordinate Frank-Wolfe algorithm. We apply the proposed method to the problem of weakly supervised learning of actions and actors from movies together with corresponding movie scripts. The scaling up of the learning problem to 66 feature length movies enables us to significantly improve weakly supervised action recognition.

arXiv:1710.00187

Unsupervised Segmentation of Action Segments in Egocentric Videos using Gaze

Unsupervised segmentation of action segments in egocentric videos is a desirable feature in tasks such as activity recognition and content-based video retrieval. Reducing the search space into a finite set of action segments facilitates a faster and less noisy matching. However, there exist a substantial gap in machine understanding of natural temporal cuts during a continuous human activity. This work reports on a novel gaze-based approach for segmenting action segments in videos captured using an egocentric camera. Gaze is used to locate the region-of-interest inside a frame. By tracking two simple motion-based parameters inside successive regions-of-interest, we discover a finite set of temporal cuts. We present several results using combinations (of the two parameters) on a dataset, i.e., BRISGAZE-ACTIONS. The dataset contains egocentric videos depicting several daily-living activities. The quality of the temporal cuts is further improved by implementing two entropy measures.

arXiv:1606.04637

Ego-Surfing: Person Localization in First-Person Videos Using Ego-Motion Signatures

We envision a future time when wearable cameras are worn by the masses and recording first-person point-of-view videos of everyday life. While these cameras can enable new assistive technologies and novel research challenges, they also raise serious privacy concerns. For example, first-person videos passively recorded by wearable cameras will necessarily include anyone who comes into the view of a camera -- with or without consent. Motivated by these benefits and risks, we developed a self-search technique tailored to first-person videos. The key observation of our work is that the egocentric head motion of a target person (ie, the self) is observed both in the point-of-view video of the target and observer. The motion correlation between the target person's video and the observer's video can then be used to identify instances of the self uniquely. We incorporate this feature into the proposed approach that computes the motion correlation over densely-sampled trajectories to search for a target individual in observer videos. Our approach significantly improves self-search performance over several well-known face detectors and recognizers. Furthermore, we show how our approach can enable several practical applications such as privacy filtering, target video retrieval, and social group clustering.

arXiv:1712.00133

Video retrieval based on deep convolutional neural network

Recently, with the enormous growth of online videos, fast video retrieval research has received increasing attention. As an extension of image hashing techniques, traditional video hashing methods mainly depend on hand-crafted features and transform the real-valued features into binary hash codes. As videos provide far more diverse and complex visual information than images, extracting features from videos is much more challenging than that from images. Therefore, high-level semantic features to represent videos are needed rather than low-level hand-crafted methods. In this paper, a deep convolutional neural network is proposed to extract high-level semantic features and a binary hash function is then integrated into this framework to achieve an end-to-end optimization. Particularly, our approach also combines triplet loss function which preserves the relative similarity and difference of videos and classification loss function as the optimization objective. Experiments have been performed on two public datasets and the results demonstrate the superiority of our proposed method compared with other state-of-the-art video retrieval methods.

arXiv:1802.02305

Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder

Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization model, resulting in severe information loss. In this paper, we propose a novel unsupervised video hashing framework dubbed Self-Supervised Video Hashing (SSVH), that is able to capture the temporal nature of videos in an end-to-end learning-to-hash fashion. We specifically address two central problems: 1) how to design an encoder-decoder architecture to generate binary codes for videos; and 2) how to equip the binary codes with the ability of accurate video retrieval. We design a hierarchical binary autoencoder to model the temporal dependencies in videos with multiple granularities, and embed the videos into binary codes with less computations than the stacked architecture. Then, we encourage the binary codes to simultaneously reconstruct the visual content and neighborhood structure of the videos. Experiments on two real-world datasets (FCVID and YFCC) show that our SSVH method can significantly outperform the state-of-the-art methods and achieve the currently best performance on the task of unsupervised video retrieval.

arXiv:1803.00974공식 코드 · Papers with Code

Hashing with Mutual Information

Binary vector embeddings enable fast nearest neighbor retrieval in large databases of high-dimensional objects, and play an important role in many practical applications, such as image and video retrieval. We study the problem of learning binary vector embeddings under a supervised setting, also known as hashing. We propose a novel supervised hashing method based on optimizing an information-theoretic quantity: mutual information. We show that optimizing mutual information can reduce ambiguity in the induced neighborhood structure in the learned Hamming space, which is essential in obtaining high retrieval performance. To this end, we optimize mutual information in deep neural networks with minibatch stochastic gradient descent, with a formulation that maximally and efficiently utilizes available supervision. Experiments on four image retrieval benchmarks, including ImageNet, confirm the effectiveness of our method in learning high-quality binary embeddings for nearest neighbor retrieval.

arXiv:1803.07780

Learning and Recognizing Human Action from Skeleton Movement with Deep Residual Neural Networks

Automatic human action recognition is indispensable for almost artificial intelligent systems such as video surveillance, human-computer interfaces, video retrieval, etc. Despite a lot of progress, recognizing actions in an unknown video is still a challenging task in computer vision. Recently, deep learning algorithms have proved its great potential in many vision-related recognition tasks. In this paper, we propose the use of Deep Residual Neural Networks (ResNets) to learn and recognize human action from skeleton data provided by Kinect sensor. Firstly, the body joint coordinates are transformed into 3D-arrays and saved in RGB images space. Five different deep learning models based on ResNet have been designed to extract image features and classify them into classes. Experiments are conducted on two public video datasets for human action recognition containing various challenges. The results show that our method achieves the state-of-the-art performance comparing with existing approaches.

arXiv:1804.02516공식 코드 · Papers with Code

Learning a Text-Video Embedding from Incomplete and Heterogeneous Data

Joint understanding of video and language is an active research area with many applications. Prior work in this domain typically relies on learning text-video embeddings. One difficulty with this approach, however, is the lack of large-scale annotated video-caption datasets for training. To address this issue, we aim at learning text-video embeddings from heterogeneous data sources. To this end, we propose a Mixture-of-Embedding-Experts (MEE) model with ability to handle missing input modalities during training. As a result, our framework can learn improved text-video embeddings simultaneously from image and video datasets. We also show the generalization of MEE to other input modalities such as face descriptors. We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets. The proposed MEE model demonstrates significant improvements and outperforms previously reported methods on both text-to-video and video-to-text retrieval tasks. Code is available at: https://github.com/antoine77340/Mixture-of-Embedding-Experts

arXiv:1804.09066공식 코드 · Papers with Code

ECO: Efficient Convolutional Network for Online Video Understanding

The state of the art in video understanding suffers from two problems: (1) The major part of reasoning is performed locally in the video, therefore, it misses important relationships within actions that span several seconds. (2) While there are local methods with fast per-frame processing, the processing of the whole video is not efficient and hampers fast video retrieval or online classification of long-term activities. In this paper, we introduce a network architecture that takes long-term content into account and enables fast per-video processing at the same time. The architecture is based on merging long-term content already in the network rather than in a post-hoc fusion. Together with a sampling strategy, which exploits that neighboring frames are largely redundant, this yields high-quality action classification and video captioning at up to 230 videos per second, where each video can consist of a few hundred frames. The approach achieves competitive performance across all datasets while being 10x to 80x faster than state-of-the-art methods.

arXiv:1712.08315공식 코드 · Papers with Code

Deep Hashing with Category Mask for Fast Video Retrieval

This paper proposes an end-to-end deep hashing framework with category mask for fast video retrieval. We train our network in a supervised way by fully exploiting inter-class diversity and intra-class identity. Classification loss is optimized to maximize inter-class diversity, while intra-pair is introduced to learn representative intra-class identity. We investigate the binary bits distribution related to categories and find out that the effectiveness of binary bits is highly correlated with data categories, and some bits may degrade classification performance of some categories. We then design hash code generation scheme with category mask to filter out bits with negative contribution. Experimental results demonstrate the proposed method outperforms several state-of-the-arts under various evaluation metrics on public datasets.

arXiv:1705.03933

An Improved Video Analysis using Context based Extension of LSH

Locality Sensitive Hashing (LSH) based algorithms have already shown their promise in finding approximate nearest neighbors in high dimen- sional data space. However, there are certain scenarios, as in sequential data, where the proximity of a pair of points cannot be captured without considering their surroundings or context. In videos, as for example, a particular frame is meaningful only when it is seen in the context of its preceding and following frames. LSH has no mechanism to handle the con- texts of the data points. In this article, a novel scheme of Context based Locality Sensitive Hashing (conLSH) has been introduced, in which points are hashed together not only based on their closeness, but also because of similar context. The contribution made in this article is three fold. First, conLSH is integrated with a recently proposed fast optimal sequence alignment algorithm (FOGSAA) using a layered approach. The resultant method is applied to video retrieval for extracting similar sequences. The pro- posed algorithm yields more than 80% accuracy on an average in different datasets. It has been found to save 36.3% of the total time, consumed by the exhaustive search. conLSH reduces the search space to approximately 42% of the entire dataset, when compared with an exhaustive search by the aforementioned FOGSAA, Bag of Words method and the standard LSH implementations. Secondly, the effectiveness of conLSH is demon- strated in action recognition of the video clips, which yields an average gain of 12.83% in terms of classification accuracy over the state of the art methods using STIP descriptors. The last but of great significance is that this article provides a way of automatically annotating long and composite real life videos. The source code of conLSH is made available at http://www.isical.ac.in/~bioinfo_miu/conLSH/conLSH.html

arXiv:2606.13141v2cs.AI

긴 비디오에서의 RAG 재고: 무엇을 검색하고 어떻게 활용할 것인가?

Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

검색-증강 생성(RAG)은 텍스트를 넘어 질의와 관련된 청크가 여러 모달리티와 시간적 세분성에 걸쳐 표현될 수 있는 긴 비디오로 확장되고 있습니다. 이러한 설정인 VideoRAG의 발전은 두 가지 한계로 인해 제약을 받습니다. 즉, 기존 벤치마크에서는 비디오 없이도 질의에 답할 수 있어 검색 오류가 가려진다는 점과, 기존 방법들이 청크 수준의 가변성을 무시한 채 질의당 단일 모달리티-세분성 구성만을 적용한다는 점입니다. 본 논문에서는 두 문제를 모두 해결하기 위해 시간 단위 길이의 비디오에 대한 $\langle$질의, 근거 청크, 답변$\rangle$ 트리플릿으로 구성되어 각 답변이 고유한 근거 청크에 의존함으로써 검색과 생성의 분리된 평가를 가능하게 하는 벤치마크인 V-RAGBench와, 여러 구성에 걸쳐 병렬 검색기를 실행하고 청크 적응형 재순위화를 사용하여 각 청크에 대한 최적 구성을 선택한 후 이를 생성 단계로 전달하는 학습(training) 불필요 방식인 CARVE를 제안합니다. V-RAGBench에서 CARVE는 두 단계 모두에서 8개의 최신 VideoRAG 베이스라인보다 뛰어난 성능을 보였으며, 생성기에 제공하는 근거는 단일 구성을 공유하는 대신 여러 구성을 인터리빙합니다. 이러한 성능 향상은 1인칭 시점 및 3인칭 시점의 긴 비디오 전반에서 유지되며 확장된 구성 공간으로도 이어집니다.

arXiv:2609.28437v1cs.CV

MultiVENT-Raw: 원시 동영상 검색 및 추론 벤치마크

MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos

온라인 정보는 점점 더 동영상 형식으로 소비되고 있다. 이 중 상당 부분은 원시 동영상(raw video) 형태로, 휴대폰·핸드헬드 카메라·CCTV로 촬영된 연속 영상이 소셜 미디어 플랫폼과 콘텐츠 공유 서비스에 바로 업로드되는 것이다. 전문적이거나 아마추어가 편집한 영상은 대본화된 발화, 자막(chyron), 그래픽, 메타데이터 등 주제를 맥락화하는 요소를 담는 경우가 많지만, 원시 동영상에는 일반적으로 이런 것이 없어 정보 검색(information retrieval)과 기계 이해(machine understanding)에 훨씬 더 어려운 매체가 된다. 이 분야의 진전을 돕기 위해 우리는 MultiVENT-Raw를 공개한다. 이는 주로 원시 동영상 약 12만 개(총 5,300시간 이상)로 이루어진 다국어 컬렉션으로, 130개의 이벤트와 222개의 이벤트 중심 쿼리와 짝지어져 있으며, 관련 동영상에 대한 사람이标注한 관련성 판정과 사람이 추출한 핵심 사실을 포함한다. MultiVENT-Raw는 검색(retrieval) 과제—쿼리 이벤트와 관련된 컬렉션 내 동영상을 식별하는 것—와 생성(generation) 과제—이벤트 관련 동영상을 대상 사용자를 위한 일관된 보고서로 요약하는 것—를 모두 지원한다. MultiVENT-Raw에서 강력한 베이스라인을 벤치마킹한 결과, 최신 멀티모달 모델 중 일부에게도 두 과제 모두 여전히 도전적임을 보였다.

arXiv:2606.02450v3cs.CV

구조화된 편집 프롬프트와 밀집-희소 융합을 활용한 CoVR-R용 Reason-Then-Retrieve

Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusion

CoVR-R는 추론 인식 합성 비디오 검색(composed video retrieval)을 연구한다. 기준(reference) 비디오와 편집 지시(edit instruction)가 주어졌을 때, 시스템은 편집을 만족하는 대상(target) 비디오를 검색해야 한다. 주요 어려움은 대상이 직접적으로 기술되지 않는다는 점이며, 객체 정체성(object identity), 동작 순서(action order), 최종 상태(final state), 손 상호작용(hand interaction), 장면 전환(scene transition)의 세밀한 변화로부터 추론해야 한다. 우리는 Qwen3.5-27B를 중심으로 제로샷(zero-shot) reason-then-retrieve 파이프라인을 구축한다. 각 갤러리 비디오에 대해, 모델은 검색 지향의 구조화된 설명과 생성된 토큰의 은닉 상태(hidden states)를 토큰 의존적 가중치로 풀링하여 얻은 밀집(dense) 임베딩을 생성한다. 각 쿼리에 대해, 모델은 먼저 기준 비디오와 지시에 대한 편집 추론(edit reasoning)을 수행한 후, 그 은닉 상태가 쿼리 임베딩으로 사용되는 대상 비디오 설명을 생성한다. 우리는 생성된 텍스트에 대한 TF-IDF 분기(branch)로 밀집 검색을 보완하고, 분할(split)별 가중치로 두 순위를 융합한다. 검증에서, 현재 최고 제출은 R@1 80.81, R@5 94.86, R@10 97.11, R@50 98.59를 달성한다. 블라인드 테스트 분할에서는 R@1 89.73, R@5 95.79, R@10 96.63, R@50 97.98를 달성한다.

arXiv:2607.00446v1cs.CV

VideoSearch-R1: 소프트 쿼리 정제를 통한 반복적 비디오 검색 및 추론

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

대규모 비디오 코퍼스에서 초기 검색 실패 시 후속 추론도 실패하는 문제를 해결하기 위해, VideoSearch-R1은 이산 텍스트 공간의 쿼리 재작성 대신 연속 잠재 공간에서 소프트 쿼리 토큰을 생성하는 Soft Query Refinement(SQR)를 도입한 반복적 비디오 검색·추론 에이전트 프레임워크를 제안한다. SQR은 GRPO로 학습되며, VideoSearch-R1은 Video Corpus Moment Retrieval(VCMR) 세 개 벤치마크에서 최첨단 성능을 달성하면서도 명시적 텍스트 수준 쿼리 정제 대비 훨씬 적은 생성 토큰을 요구한다.

arXiv:2606.28326v1cs.IR

ADEPT: An Entropy-Driven Dual-Strategy Agent for Interactive Video Retrieval

This research aims to solve the challenge of video retrieval from massive datasets, caused by ambiguous user queries. Prevailing single-round retrieval paradigms face a performance bottleneck, as they lack effective feedback mechanisms to handle complex search intentions. The root cause is the "Intent-Query Gap", where users' intent cannot be captured by a simple text query. To solve this, we propose the ADEPT framework: a training-free agent that pioneers an entropy-driven decision engine to efficiently guide dialogue by dynamically selecting between ASK and REFINE strategies. Experiments on two challenging datasets demonstrate that ADEPT significantly outperforms all non-interactive, heuristic, and Video-LLM baselines. The core contribution of this work is an efficient and interpretable entropy-driven interactive strategy that sets a new performance benchmark for the field of interactive video retrieval.

arXiv:2606.19682v1cs.CV

Vortex: 지능형 비디오 검색을 위한 멀티모달 융합 시스템

Vortex: Multi-Modal Fusion System for Intelligent Video Retrieval

본 논문은 지능형 멀티미디어 검색과 시간적 추론을 발전시키기 위해 설계된, 우리 팀 FocusOnFun이 호찌민 시 AI 챌린지 2025를 위해 개발한 멀티모달 비디오 검색 시스템인 Vortex를 소개합니다. 이 시스템은 적응형 키프레임 추출, 비전-언어 및 음성 모델을 통한 멀티모달 메타데이터 생성, 그리고 전역적이고 세밀한 의미를 균형 있게 조정하기 위해 Reciprocal Rank Fusion(RRF)을 통해 CLIP 및 SigLIP2 임베딩을 융합하는 하이브리드 검색 전략을 통합합니다. 상호작용성을 향상시키기 위해 Vortex는 Rocchio 기반 관련성 피드백과 순차적 이벤트 정렬을 위한 다단계 시간적 검색 메커니즘을 포함합니다. Milvus와 Elasticsearch를 기반으로 구축된 이 아키텍처는 확장 가능한 인덱싱과 효율적인 검색을 가능하게 합니다. 공식 대회에서 평가된 결과, 우리 FocusOnFun 팀의 시스템은 예선에서 79.6/88점(90.5%)을 달성했으며, 본선에서 추가로 평가되어 질의응답(QA) 과제에서 '최우수(Outstanding)' 결과를 포함한 전반적으로 '우수(Excellent)'한 성과를 거두었습니다. 이는 CLIP과 SigLIP2의 상호 보완적인 강점을 보여주고 하이브리드 검색 접근 방식의 효과를 확인시켜 줍니다. 이 시스템은 지능형, 문맥 인식, 상호작용형 비디오 검색 분야의 향후 연구를 위한 견고한 기반을 마련합니다.

arXiv:2606.19062v1cs.CV

DREAM: 크로스 모달 검색을 위한 이중 목표 인코딩을 통한 비전-언어 모델 확장

DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval

오늘날 미디어 중심의 세계에서 감시, 교육, 엔터테인먼트와 같은 도메인에 걸쳐 비디오 콘텐츠가 기하급수적으로 증가함에 따라 자연어 질의를 통해 의미적으로 관련된 비디오를 검색하는 것이 점점 더 중요해지고 있습니다. 초기 비디오 검색 시스템은 handcrafted features나 얕은 크로스 모달 매핑에 의존하여, 복잡한 의미와 시간적 역학을 포착하는 능력이 제한되었습니다. 대규모 비전-언어 모델이 크로스 모달 정렬을 개선했지만, 세밀한 시간적 의존성과 미묘한 언어적 구조를 모델링하는 데 있어서는 여전히 과제가 남아 있습니다. 본 논문에서는 향상된 시각적 및 텍스트 인코딩을 통해 이러한 한계를 해결하는 새로운 멀티모달 프레임워크인 DREAM: Dual-path Representation Enhancement and Alignment Model을 소개합니다. DREAM은 지역 및 전역 언어 의미를 모두 포착하기 위해 마스크드(masked) 및 순열(permuted) 언어 모델링 목표를 결합하는 하이브리드 언어 모델링 전략을 통합합니다. 시각적 측면에서는 다단계 토큰 상호작용과 coarse-to-fine 어텐션 정제를 통해 공간 및 시간 정보를 통합하는 캐스케이드 그룹 어텐션(cascaded group attention)을 갖춘 계층적 비전 인코더를 설계했습니다. 우리는 널리 사용되는 MSRVTT, MSVD 및 LSMDC 벤치마크 데이터셋에 대한 포괄적인 평가를 통해 DREAM을 검증했으며, 각각 49.4%, 49.7% 및 27.3%의 새로운 최신 기술(SOTA) R1 점수를 달성했습니다. 정성적 분석을 통해 프레임 간에 일관된 어텐션을 유지하고 복잡한 질의를 동적인 비디오 콘텐츠와 정렬하는 모델의 능력을 추가로 확인했습니다. 이러한 발견은 강력하고 문맥 인식 비디오 검색을 가능하게 하는 데 있어 계층적 어텐션과 이중 목표 텍스트 모델링의 효과를 강조하며, 크로스 모달 표현 학습 발전을 위한 향후 연구의 길을 열어줍니다.

arXiv:2606.12295v1cs.CV

MAGMaR 2026 공유 과제 결과

Findings of the MAGMaR 2026 Shared Task

본 개요 논문은 멀티모달 검색을 통한 멀티모달 증강 생성(MAGMaR) 제2차 워크샵의 공유 과제 결과를 제시합니다. 이 공유 과제에서 참가자들은 (i) 비디오 검색 또는 (ii) 검색된 비디오가 주어졌을 때 기사의 그라운딩된 생성(grounded generation) 중 하나에 초점을 맞춘 시스템을 제출했습니다. 팀은 두 과제 중 하나에 제출할 수 있었습니다. 검색 과제의 경우, 총 17개의 시스템을 제출한 2개의 참가 팀이 있었으며, 이 모든 시스템은 작년 공유 과제 우승자에서 파생된 기준선(baseline)을 능가했습니다. 생성 측면에서는 4개 팀이 16개의 시스템을 제출했습니다. 모든 팀은 인간 주석자에 의해 최고라고 라벨링된 생성된 리포트를 최소 하나 이상 보유하고 있었습니다.