Token Embeddings: Just a Table Lookup

Updated September 2026 · 4 min read

An embedding layer takes a token ID and returns a list of learned numbers. That is the operation: vector = table[token_id]. Let’s make it concrete.

Before the first block → token embeddings. View the model diagram →

First, give each token an ID

Suppose our sentence is “cat sat cat”. For this small example, the tokenizer treats each word as one token and assigns cat → 4 and sat → 1. The sentence becomes [4, 1, 4].

The IDs are just labels. ID 4 is not “more cat” than ID 1, and nearby IDs need not have related meanings. Real tokenizers can split words into smaller pieces; we use whole words here to keep the lookup visible.

An ID selects one row

Imagine a table with one row per token. Each row stores a vector: simply a fixed-length list of numbers. To embed cat, take row 4. To embed sat, take row 1. Put those rows in sentence order.

Try the two occurrences of cat below. They select the same table row, but fill different positions in the sentence.

Nothing multiplies the vector by 4. Nothing searches for the closest row. The ID is an address: table[4] directly selects the stored vector.

Count rows and columns separately

Our vocabulary has 5 tokens, and each vector has 3 numbers. The table therefore has shape [5, 3]: five rows, three columns. Its embedding dimension is 3.

The sentence has 3 tokens, so its output has shape [3, 3]. A ten-token sentence would produce [10, 3]. Sentence length changes the number of output rows; it does not change the width of each vector.

Who chooses the numbers?

When training a model from scratch, the table usually starts with random values. The model makes predictions, a loss measures its errors, and the optimizer adjusts the table along with the other model weights. Useful vectors are learned through that process. With a pretrained model, you load its already learned table.

The columns do not come with labels such as “animal”, “size”, or “happiness”. They are numbers the rest of the network learns to use. An ID identifies a token; its vector provides adjustable features.

There is one shared row for cat. Training examples that use it can help update that row. Reading a new sentence during ordinary inference does not rewrite the table.

The same token can acquire different context

At lookup time, both occurrences of cat get [1, 0, 1]. They can acquire different representations later, when the model uses position information and surrounding context. In a causal decoder, the second cat can see the earlier “cat sat”; the first cannot see those future tokens.

The whole lookup in PyTorch

This code uses exactly the table above. freeze=False makes the table trainable; running the lookup itself does not update its values.

import torch
from torch import nn

table = torch.tensor([
    [-1., 0.,  1.],  # 0: the
    [ 0., 2., -1.],  # 1: sat
    [ 1., 1.,  0.],  # 2: on
    [-1., 1.,  1.],  # 3: mat
    [ 1., 0.,  1.],  # 4: cat
])
embedding = nn.Embedding.from_pretrained(table, freeze=False)
ids = torch.tensor([4, 1, 4])
vectors = embedding(ids)  # selects table[4], table[1], table[4]
print(vectors)
# tensor([[1., 0., 1.], [0., 2., -1.], [1., 0., 1.]], ...)

For a fresh random table, use nn.Embedding(5, 3). The two arguments mean five possible IDs and three numbers per vector. PyTorch’s Embedding documentation.

Next: let the tokens exchange information

You now have one vector per token. Attention explains how those vectors use context. If the ID step is still unclear, read tokenization first.