Word2vec Skip-gram Dimensionality Selection via Sequential Normalized Maximum Likelihood
원문 초록
In this paper, we propose a novel information criteria-based approach to select the dimensionality of the word2vec Skip-gram (SG). From the perspective of the probability theory, SG is considered as an implicit probability distribution estimation under the assumption that there exists a true contextual distribution among words. Therefore, we apply information criteria with the aim of selecting the best dimensionality so that the corresponding model can be as close as possible to the true distribution. We examine the following information criteria for the dimensionality selection problem: the Akaike Information Criterion, Bayesian Information Criterion, and Sequential Normalized Maximum Likelihood (SNML) criterion. SNML is the total codelength required for the sequential encoding of a data sequence on the basis of the minimum description length. The proposed approach is applied to both the original SG model and the SG Negative Sampling model to clarify the idea of using information criteria. Additionally, as the original SNML suffers from computational disadvantages, we introduce novel heuristics for its efficient computation. Moreover, we empirically demonstrate that SNML outperforms both BIC and AIC. In comparison with other evaluation methods for word embedding, the dimensionality selected by SNML is significantly closer to the optimal dimensionality obtained by word analogy or word similarity tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matrices
LGDE: Local Graph-based Dictionary Expansion
A Likelihood Ratio Test of Genetic Relationship among Languages
Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes
Improving Heterogeneous Graph Learning with Weighted Mixed-Curvature Product Manifold
토론
이 논문과 연결된 포럼 글이 아직 없습니다.