Towards Learning Cross-Modal Perception-Trace Models
원문 초록
Representation learning is a key element of state-of-the-art deep learning approaches. It enables to transform raw data into structured vector space embeddings. Such embeddings are able to capture the distributional semantics of their context, e.g. by word windows on natural language sentences, graph walks on knowledge graphs or convolutions on images. So far, this context is manually defined, resulting in heuristics which are solely optimized for computational performance on certain tasks like link-prediction. However, such heuristic models of context are fundamentally different to how humans capture information. For instance, when reading a multi-modal webpage (i) humans do not perceive all parts of a document equally: Some words and parts of images are skipped, others are revisited several times which makes the perception trace highly non-sequential; (ii) humans construct meaning from a document's content by shifting their attention between text and image, among other things, guided by layout and design elements. In this paper we empirically investigate the difference between human perception and context heuristics of basic embedding models. We conduct eye tracking experiments to capture the underlying characteristics of human perception of media documents containing a mixture of text and images. Based on that, we devise a prototypical computational perception-trace model, called CMPM. We evaluate empirically how CMPM can improve a basic skip-gram embedding approach. Our results suggest, that even with a basic human-inspired computational perception model, there is a huge potential for improving embeddings since such a model does inherently capture multiple modalities, as well as layout and design elements.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Graph Collaborative Attention Network for Link Prediction in Knowledge Graphs
A Survey of Link Prediction in N-ary Knowledge Graphs
GenIC: An LLM-Based Framework for Instance Completion in Knowledge Graphs
Topic Modeling and Link-Prediction for Material Property Discovery
Context-Driven Knowledge Graph Completion with Semantic-Aware Relational Message Passing
HJE: Joint Convolutional Representation Learning for Knowledge Hypergraph Completion
토론
이 논문과 연결된 포럼 글이 아직 없습니다.