NLP · word2vec
Dense word vectors learned by predicting context (skip-gram), trained with gensim on the 20 Newsgroups corpus, with nearest neighbours.
BoW vectors are huge, sparse, and know nothing about meaning (“car” and “automobile” share no dimension). word2vec (Mikolov et al., 2013) learns a dense vector (say 100 numbers) per word:
- Skip-gram: predict the context words from the centre word.
- CBOW: predict the centre word from its context.
- Negative sampling turns this into cheap binary classification tasks.
This is the distributional hypothesis: words that appear in similar contexts get similar vectors. I trained skip-gram with gensim on scikit-learn’s 20 Newsgroups posts (about 11k training documents). That’s tiny by word2vec standards, so the neighbours are noisy but recognisable.
import re
from gensim.models import Word2Vec
from sklearn.datasets import fetch_20newsgroups
posts = fetch_20newsgroups(subset="train", remove=("headers", "footers", "quotes")).data
sentences = [re.findall(r"[a-z]+", p.lower()) for p in posts]
sentences = [s for s in sentences if len(s) > 3]
print("documents:", len(sentences), "| tokens:", sum(map(len, sentences)))
model = Word2Vec(sentences, vector_size=100, window=5, min_count=5, sg=1, workers=1, seed=0, epochs=5)
print("vocabulary:", len(model.wv), "| vector shape:", model.wv["computer"].shape)
for w in ["windows", "god", "car", "space"]:
print(f"{w:>8} ->", [n for n, _ in model.wv.most_similar(w, topn=5)])
print("sim(car, engine) =", round(float(model.wv.similarity("car", "engine")), 3),
"| sim(car, god) =", round(float(model.wv.similarity("car", "god")), 3))Output:
documents: 10939 | tokens: 2309684
vocabulary: 20019 | vector shape: (100,)
windows -> ['apps', 'excel', 'microsoft', 'linux', 'povray']
god -> ['christ', 'allah', 'salvation', 'lord', 'eternal']
car -> ['garage', 'bike', 'honda', 'helmet', 'volvo']
space -> ['shuttle', 'redesign', 'planetary', 'goddard', 'exploration']
sim(car, engine) = 0.674 | sim(car, god) = 0.21