NLP · GloVe

GloVe (Pennington, Socher and Manning, 2014) starts from a different angle than word2vec. It builds a global co-occurrence matrix X (how often word j appears near word i) and fits vectors so that $\mathbf{w}_i^\top\tilde{\mathbf{w}}_j + b_i + \tilde{b}_j \approx \log X_{ij}$.

The Stanford team released pretrained vectors. I loaded the 50-dimensional Wikipedia + Gigaword vectors (glove.6B.50d.txt from glove.6B.zip on the Stanford page) with gensim’s plain-text loader. They’re far better than my small word2vec from last week.

The famous party trick is analogies by vector arithmetic: king − man + woman ≈ queen. Bolukbasi et al. (2016) showed that the same arithmetic exposes gender bias learned from the text. Embeddings mirror their data.

from gensim.models import KeyedVectors

# glove.6B.50d.txt comes from glove.6B.zip on the Stanford GloVe page (plain text, one word + 50 numbers per line)
glove = KeyedVectors.load_word2vec_format("glove.6B.50d.txt", binary=False, no_header=True)
print("vocab:", len(glove.index_to_key), "| dims:", glove.vector_size)
print("nearest to 'python':", [w for w, _ in glove.most_similar("python", topn=5)])

def analogy(a, b, c):
    return glove.most_similar(positive=[b, c], negative=[a], topn=1)[0][0]

print("man : king :: woman :", analogy("man", "king", "woman"))
print("india : delhi :: france :", analogy("india", "delhi", "france"))
print("walk : walked :: swim :", analogy("walk", "walked", "swim"))
print("man : programmer :: woman :", analogy("man", "programmer", "woman"))

Output:

vocab: 400001 | dims: 50
nearest to 'python': ['reticulated', 'spamalot', 'php', 'owl', 'mouse']
man : king :: woman : queen
india : delhi :: france : paris
walk : walked :: swim : swam
man : programmer :: woman : prodigy

Three of the four analogies land exactly. The fourth returns “prodigy” with these small 50-d GloVe vectors. Bolukbasi et al. reported their “homemaker” example using word2vec vectors trained on Google News, so the exact words depend on which embedding you probe. The general point of their paper, that analogy directions encode stereotypes from the training text, is about the method, not this one query.