Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition
원문 초록
We propose a transfer deep learning (TDL) framework that can transfer the knowledge obtained from a single-modal neural network to a network with a different modality. Specifically, we show that we can leverage speech data to fine-tune the network trained for video recognition, given an initial set of audio-video parallel dataset within the same semantics. Our approach first learns the analogy-preserving embeddings between the abstract representations learned from intermediate layers of each network, allowing for semantics-level transfer between the source and target modalities. We then apply our neural network operation that fine-tunes the target network with the additional knowledge transferred from the source network, while keeping the topology of the target network unchanged. While we present an audio-visual recognition task as an application of our approach, our framework is flexible and thus can work with any multimodal dataset, or with any already-existing deep networks that share the common underlying semantics. In this work in progress report, we aim to provide comprehensive results of different configurations of the proposed approach on two widely used audio-visual datasets, and we discuss potential applications of the proposed approach.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition
VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models
BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimation
PAVE: Patching and Adapting Video Large Language Models
VideoPure: Diffusion-based Adversarial Purification for Video Recognition
Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition
토론
이 논문과 연결된 포럼 글이 아직 없습니다.