Word Similarity

fastText embeddings exploit subword information to construct word embeddings. Representations are learnt of character $n$-grams, and words represented as the sum of the $n$-gram vectors. This extends the word2vec type models with subword information. This helps the embeddings understand suffixes and prefixes. Once a word is represented using character $n$-grams, a skipgram model is trained to learn the embeddings.

Skip-gram Word2Vec is an architecture for computing word embeddings. Instead of using surrounding words to predict the center word, as with CBow Word2Vec, Skip-gram Word2Vec uses the central word to predict the surrounding words. The skip-gram objective function sums the log probabilities of the surrounding $n$ words to the left and right of the target word $w\_{t}$ to produce the following objective: $$J\_\theta = \frac{1}{T}\sum^{T}\_{t=1}\sum\_{-n\leq{j}\leq{n}, \neq{0}}\log{p}\left(w\_{j+1}\mid{w\_{t}}\right)$$

Continuous Bag-of-Words Word2Vec is an architecture for creating word embeddings that uses $n$ future words as well as $n$ past words to create a word embedding. The objective function for CBOW is: $$ J\_\theta = \frac{1}{T}\sum^{T}\_{t=1}\log{p}\left(w\_{t}\mid{w}\_{t-n},\ldots,w\_{t-1}, w\_{t+1},\ldots,w\_{t+n}\right) $$ In the CBOW model, the distributed representations of context are used to predict the word in the middle of the window. This contrasts with Skip-gram Word2Vec where the distributed representation of the input word is used to predict the context.

A novel low-resource intrinsic metric to evaluate word embedding quality based on graph modularity.