Text Features (Bag of Words, TF-IDF)

September 2026 · 15 min read

A bag-of-words vector gives each vocabulary term a column and counts its occurrences in a document. TF-IDF starts from those counts, adjusts them using how widely each term appears across the training documents, and can normalize the resulting vector.

We will keep the same three short documents throughout. That lets you follow the numbers from words to columns to weights, rather than meeting the formula before seeing what it is measuring.

Start with a tiny collection of documents

Our training documents are D1: “red parcel late”, D2: “blue parcel late late”, and D3: “red parcel arrived”. Treat each message as one document. For this example, lowercase word tokens are enough; no stemming or stop-word removal is used.

The vocabulary, in the column order used below, is [arrived, blue, late, parcel, red]. The row for D2 is therefore [0, 1, 2, 1, 0]: no “arrived”, one “blue”, two “late”, one “parcel” and no “red”.

The representation is a “bag” because these single-word counts discard order. “refund after delivery” and “delivery after refund” have the same unigram counts, despite expressing different sequences. Keep that limitation in mind when choosing the representation.

Separate occurrence count from document frequency

The term frequency in this chapter is the raw count within one document. For “late” in D2 it is 2. The document frequency counts how many training documents contain the term at least once. “Late” appears in D1 and D2, so its document frequency is also 2—but for a different reason. Repeating “late” again inside D2 would increase its term count without increasing its document frequency.

Inverse document frequency gives a larger multiplier to terms that appear in fewer training documents. To match the code, we use scikit-learn's smoothed convention. Let $N$ be the number of training documents and $\mathrm{df}(w)$ the number containing word $w$:

The logarithm is natural log. Here $N=3$, so “late” receives $1+\log(4/3)\approx1.2877$. “Parcel” appears in all three documents and receives 1. The additive constants define this particular convention; other TF-IDF formulations differ.

Common terms are relatively downweighted here, not automatically deleted. A rare term may be useful, but it could also be a typo or an identifier. IDF measures rarity in the fitted corpus, not truth, relevance or semantic importance.

Multiply each count by its fitted IDF

For a term in one document, its unnormalized TF-IDF weight is simply count × IDF. In D2, “late” has weight $2\times1.2877\approx2.5754$. “Blue” has weight 1.6931 and “parcel” has weight 1. All absent words remain zero.

Those multipliers are learned once from the training corpus. We do not recompute a different IDF vector for each document. The same “late” column receives the same IDF multiplier in D1 and D2; their counts determine their different weights.

Normalize length when you want to compare direction

A longer document can have larger counts simply because it contains more words. L2 normalization divides the weighted row by its Euclidean length, making every nonzero row have length 1. For D2 the length is

Dividing gives approximately [0, 0.5225, 0.7948, 0.3086, 0]. The coordinates' squares sum to 1; the coordinates themselves need not sum to 1. These weights are not probabilities. A dot product between two nonzero L2-normalized rows equals their cosine similarity.

Length normalization can discard useful magnitude: duplicating every token in a document doubles its raw count-weighted vector but leaves its normalized direction unchanged. If document length matters for the task, consider a separate length feature. A logarithmic term-frequency option can also reduce the influence of repetition before normalization; treat it as a choice to validate.

New text must use the fitted vocabulary

Now transform “green parcel”. The word “green” was absent from training, so there is no column for it. This fitted word-vectorizer ignores it. “Parcel” uses its existing column and its saved IDF of 1.

An all-zero row does not mean the original message had no meaning. It means this representation found no retained features. Monitor empty vectors and vocabulary coverage when the language or domain changes. An unseen word is not assigned a large IDF merely because it is rare: it is outside this fitted feature space.

Decide what counts as a token

Preprocessing is part of the model's feature contract. The default CountVectorizer lowercases text and uses a word pattern that retains tokens of at least two word characters. That can drop a one-letter code and handle punctuation differently from what your application expects. Inspect its analyzer on representative inputs.

Do not remove words such as “not” automatically when they change the task's meaning. Word bigrams can preserve short phrases such as “not arrived”; character n-grams can share pieces across spelling variants. Both enlarge the feature space and change what is represented. For Unicode and case policies, use the separate text-normalization chapter.

Bag-of-words and TF-IDF remain lexical representations. Synonyms such as “parcel” and “package” use distinct columns unless another processing step connects them. Learned embeddings approach that problem differently; evaluate the representations against the task rather than treating one as a universal replacement.

Fit vocabulary and IDF inside training

Even without labels, reading the full evaluation corpus to learn vocabulary, document frequencies or frequency cutoffs uses information from the held-out distribution. For ordinary inductive evaluation, fit the vectorizer on training documents and call only transform on validation and test documents.

Keep the vectorizer and classifier in a pipeline during cross-validation. Each fold must learn its own vocabulary and IDF. Choose n-gram ranges, frequency cutoffs and model regularization using validation, then refit the chosen pipeline on the final training set.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer

train_docs = ['red parcel late',
              'blue parcel late late',
              'red parcel arrived']
vectorizer = CountVectorizer()
counts = vectorizer.fit_transform(train_docs)
print(vectorizer.get_feature_names_out())
print(counts.toarray())  # Dense display is only for this tiny example.

weighted_encoder = TfidfTransformer(norm=None, smooth_idf=True)
weighted = weighted_encoder.fit_transform(counts)
normalized_encoder = TfidfTransformer(norm='l2', smooth_idf=True)
normalized = normalized_encoder.fit_transform(counts)
print(np.round(weighted_encoder.idf_, 4))
print(np.round(weighted.toarray(), 4))
print(np.round(normalized.toarray(), 4))

# Reuse vocabulary AND fitted IDF values for new documents.
new_counts = vectorizer.transform(['green parcel', 'green', ''])
print(normalized_encoder.transform(new_counts).toarray())
# [0, 0, 0, 1, 0], then two all-zero rows.

A small reference implementation on already-tokenized text

The C++ version learns a sorted vocabulary and counts each word at most once per document when calculating document frequency. Transformation counts all occurrences, applies the saved IDF and optionally normalizes. It uses a dense vector for clarity and assumes tokenization has already happened consistently.

#include <cmath>
#include <iomanip>
#include <iostream>
#include <map>
#include <set>
#include <stdexcept>
#include <string>
#include <vector>

using Tokens = std::vector<std::string>;
struct Encoder {
    std::map<std::string, std::size_t> index;
    std::vector<double> idf;
};

// Tokens have already been normalized and split consistently.
Encoder fit(const std::vector<Tokens>& documents) {
    std::set<std::string> vocabulary;
    for (const auto& doc : documents) vocabulary.insert(doc.begin(), doc.end());
    if (vocabulary.empty()) throw std::invalid_argument("empty vocabulary");
    Encoder encoder;
    for (const auto& word : vocabulary) {
        std::size_t next = encoder.index.size();
        encoder.index.emplace(word, next);
    }
    std::vector<int> df(vocabulary.size(), 0);
    for (const auto& doc : documents) {
        std::set<std::string> present(doc.begin(), doc.end());
        for (const auto& word : present) ++df[encoder.index.at(word)];
    }
    for (int frequency : df)
        encoder.idf.push_back(1 + std::log((1.0 + documents.size()) / (1.0 + frequency)));
    return encoder;
}

std::vector<double> transform(const Encoder& encoder, const Tokens& doc,
                              bool normalize = true) {
    std::vector<double> values(encoder.idf.size(), 0);
    for (const auto& word : doc) {
        auto found = encoder.index.find(word);
        if (found != encoder.index.end()) ++values[found->second];
    }
    double squared_length = 0;
    for (std::size_t j = 0; j < values.size(); ++j) {
        values[j] *= encoder.idf[j];
        squared_length += values[j] * values[j];
    }
    if (normalize && squared_length > 0)
        for (double& value : values) value /= std::sqrt(squared_length);
    return values;
}

int main() {
    std::vector<Tokens> documents = {
        {"red", "parcel", "late"},
        {"blue", "parcel", "late", "late"},
        {"red", "parcel", "arrived"},
    };
    Encoder encoder = fit(documents);
    std::cout << std::fixed << std::setprecision(4);
    for (double value : transform(encoder, documents[1])) std::cout << value << ' ';
    std::cout << '\n';  // 0.0000 0.5225 0.7948 0.3086 0.0000
}

References and further reading

• Scikit-learn: text feature extraction — bag-of-words, sparse matrices, n-grams and the limits of lexical representations.
• Scikit-learn: CountVectorizer — tokenization, vocabulary construction and frequency filters.
• Scikit-learn: TfidfTransformer — the exact smoothed IDF, term-frequency and normalization conventions used here.
• Scikit-learn: common pitfalls — fitting data-dependent transformations only on the training portion.