Learning English with Peppa Pig
원문 초록
Recent computational models of the acquisition of spoken language via grounding in perception exploit associations between the spoken and visual modalities and learn to represent speech and visual data in a joint vector space. A major unresolved issue from the point of ecological validity is the training data, typically consisting of images or videos paired with spoken descriptions of what is depicted. Such a setup guarantees an unrealistically strong correlation between speech and the visual data. In the real world the coupling between the linguistic and the visual modality is loose, and often confounded by correlations with non-semantic aspects of the speech signal. Here we address this shortcoming by using a dataset based on the children's cartoon Peppa Pig. We train a simple bi-modal architecture on the portion of the data consisting of dialog between characters, and evaluate on segments containing descriptive narrations. Despite the weak and confounded signal in this training data our model succeeds at learning aspects of the visual semantics of spoken language.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Describe Anything Model for Visual Question Answering on Text-rich Images
Assay2Mol: large language model-based drug design using BioAssay context
Beyond Accuracy: Metrics that Uncover What Makes a 'Good' Visual Descriptor
Dataset Distillation via Vision-Language Category Prototype
DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving
InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
토론
이 논문과 연결된 포럼 글이 아직 없습니다.