Discrete Infomax Codes for Supervised Representation Learning
원문 초록
Learning compact discrete representations of data is a key task on its own or for facilitating subsequent processing of data. In this paper we present a model that produces Discrete InfoMax Codes (DIMCO); we learn a probabilistic encoder that yields k-way d-dimensional codes associated with input data. Our model's learning objective is to maximize the mutual information between codes and labels with a regularization, which enforces entries of a codeword to be as independent as possible. We show that the infomax principle also justifies previous loss functions (e.g., cross-entropy) as its special cases. Our analysis also shows that using shorter codes, as DIMCO does, reduces overfitting in the context of few-shot classification. Through experiments in various domains, we observe this implicit meta-regularization effect of DIMCO. Furthermore, we show that the codes learned by DIMCO are efficient in terms of both memory and retrieval time compared to previous methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Are encoders able to learn landmarkers for warm-starting of Hyperparameter Optimization?
Semantic Structure Enhanced Contrastive Adversarial Hash Network for Cross-media Representation Learning
KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos
Dare to Plagiarize? Plagiarized Painting Recognition and Retrieval
Multimodal Information Retrieval for Open World with Edit Distance Weak Supervision
토론
이 논문과 연결된 포럼 글이 아직 없습니다.