Aligning Subtitles in Sign Language Videos
원문 초록
The goal of this work is to temporally align asynchronous subtitles in sign language videos. In particular, we focus on sign-language interpreted TV broadcast data comprising (i) a video of continuous signing, and (ii) subtitles corresponding to the audio content. Previous work exploiting such weakly-aligned data only considered finding keyword-sign correspondences, whereas we aim to localise a complete subtitle text in continuous signing. We propose a Transformer architecture tailored for this task, which we train on manually annotated alignments covering over 15K subtitles that span 17.7 hours of video. We use BERT subtitle embeddings and CNN video representations learned for sign recognition to encode the two signals, which interact through a series of attention layers. Our model outputs frame-level predictions, i.e., for each video frame, whether it belongs to the queried subtitle or not. Through extensive evaluations, we show substantial improvements over existing alignment baselines that do not make use of subtitle text embeddings for learning. Our automatic alignment model opens up possibilities for advancing machine translation of sign languages via providing continuously synchronized video-text data.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
Gender Bias in English-to-Greek Machine Translation
TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration
Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations
Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement
토론
이 논문과 연결된 포럼 글이 아직 없습니다.