Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
원문 초록
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e.g., "dressing") to the action (e.g., "mix yogurt") that produced it. The key challenge is the inevitable visual-linguistic ambiguities arising from the changes in both visual appearance and referring expression of an entity in the video. This challenge is amplified by the fact that we aim to resolve references with no supervision. We address these challenges by learning a joint visual-linguistic model, where linguistic cues can help resolve visual ambiguities and vice versa. We verify our approach by learning our model unsupervisedly using more than two thousand unstructured cooking videos from YouTube, and show that our visual-linguistic model can substantially improve upon state-of-the-art linguistic only model on reference resolution in instructional videos.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
What is the Right Embedding Space for Contrastive Learning in REC?
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
RemoteSAM: Towards Segment Anything for Earth Observation
Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation
4th PVUW MeViS 3rd Place Report: Sa2VA
토론
이 논문과 연결된 포럼 글이 아직 없습니다.