NLP · Bag of words and TF-IDF

• Bag of words (BoW): one column per vocabulary word, counting occurrences. Word order is lost.
• TF-IDF: tf × idf, where $\text{idf}(t) = \log\frac{1+N}{1+\text{df}(t)} + 1$ in scikit-learn’s smoothed version. A word that appears in every document gets a low weight, and a distinctive word gets a high one.
• n-grams (ngram_range=(1, 2)) keep a little word order: “not good” ≠ “good”.
• In a corpus this tiny, “the” appears in only half the documents, so its idf isn’t low. Removing stop words keeps it from topping the list.
• Cosine similarity between TF-IDF vectors is a strong baseline for search and deduplication.

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are great pets",
    "stock prices fell as markets closed",
]
cv = CountVectorizer()
X = cv.fit_transform(docs)
print(cv.get_feature_names_out()[:8], X.shape)
print(X.toarray()[0])

tf = TfidfVectorizer(ngram_range=(1, 2), stop_words="english")
T = tf.fit_transform(docs)
vocab = tf.get_feature_names_out()
row = T[0].toarray().ravel()
print("top TF-IDF terms in doc 0:", [vocab[i] for i in row.argsort()[::-1][:3]])

query = tf.transform(["a cat on a mat"])
print("similarity to each doc:", cosine_similarity(query, T).round(3))

Output:

['and' 'are' 'as' 'cat' 'cats' 'closed' 'dog' 'dogs'] (4, 19)
[0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 1 0 2]
top TF-IDF terms in doc 0: ['sat mat', 'mat', 'cat']
similarity to each doc: [[0.658 0.    0.    0.   ]]