An embedding is a dense vector of numbers that a model learns to represent something — a word, an image, a user — so that similar things get similar…
See why treating words or categories as bare IDs loses all their meaning.
How do you feed a word like 'cat' to a model that only does math? The naive answer is a one-hot vector: assign each word a slot, set it to 1, the rest to 0. It works, but it throws away all meaning — 'cat' and 'dog' are exactly as different as 'cat' and 'airplane', because every one-hot vector is equidistant from every other.
For large vocabularies these vectors are also huge and mostly zeros. We want a representation where similar things are actually close together — that is what an embedding provides.
Understand embeddings as meaning turned into position in a vector space.
An embedding maps each item to a dense vector — a list of, say, a few hundred numbers — arranged so that similar items sit close and dissimilar ones sit far apart. 'Cat' and 'dog' end up near each other; 'cat' and 'airplane' end up far.
Because meaning becomes geometry, you can measure similarity with distance, and even do arithmetic: in word embeddings, the vector for 'king' minus 'man' plus 'woman' lands near 'queen'. The dimensions aren't hand-labeled — they emerge to encode useful distinctions.
See how training pressures a model to build useful embeddings.
Embeddings are not written by hand — they are learned. You start each item with a random vector, then train the model on a task, and gradient descent adjusts those vectors to do the task well. The vectors that help end up capturing the patterns the task depends on.
A classic objective is predicting a word from its neighbors (or vice versa): to predict well, the model must place words used in similar contexts near each other, so meaning falls out for free. The same idea extends to images, audio, and users.
# vocabulary of 10,000 words, each a 128-number vector embeddings = table(shape = [10000, 128]) # learned during training vec_cat = embeddings[id("cat")] # -> 128 numbers vec_dog = embeddings[id("dog")] similarity = cosine(vec_cat, vec_dog) # high: similar meanings
The embedding table is just a lookup: each item's id indexes a row of learned numbers. The rows start random and are trained; afterward, related items like cat and dog have vectors with high cosine similarity, which is meaning made measurable.
Turn learned representations into search, features, and transfer — carefully.
Once you have good embeddings, a lot becomes easy. Semantic search ranks results by vector distance. Recommendations find items near a user's vector. Clustering and visualization group items by proximity. And embeddings from a big pretrained model transfer to new tasks, giving a strong starting representation without training from scratch.
Two rules. Vectors from different models are not comparable — always embed with the same model on both sides. And embeddings inherit the biases and blind spots of their training data, so associations you didn't intend can appear. Also, an embedding is only as good as the task and data it was trained on; a domain-specific problem may need domain-specific embeddings.
An embedding is a dense, learned vector that represents an item so similar items sit close in a vector space, replacing meaningless one-hot IDs with measurable meaning. The vectors start random and are learned during training, with useful structure like semantic similarity emerging from the task. Embeddings power search, recommendations, clustering, and transfer learning — but only compare vectors from the same model, and remember they inherit their training data's biases.
Imagine building a 'similar products' feature for a store. Explain why learned embeddings beat category IDs for measuring similarity, how you would obtain the embeddings, and one check you would run to confirm nearby vectors really are similar products.
What is an embedding?
Embeddings turn items into coordinates where nearness reflects similarity, capturing meaning as geometry rather than as arbitrary IDs.
Why are one-hot vectors a poor representation of meaning?
One-hot encoding assigns each item an isolated slot, discarding all similarity information; embeddings fix this by placing related items close together.
How does a model get useful embeddings?
Training pressures the vectors toward configurations that solve the task, and useful structure like semantic similarity emerges as a byproduct.
What is a key pitfall when using embeddings?
Embeddings only share a space if made by the same model; they also inherit training-data biases, so both need care.