Understand how vectors encode meaning for search, clustering, and recommendations.
Build the simplest useful picture of an embedding: a learned vector that turns the meaning of a word, sentence, or item into coordinates you can measure. This module answers what an embedding is and why we need one.
An embedding is a fixed-length vector of numbers that represents the meaning of a thing.
Why this matters: It matters because once meaning is a vector, similarity becomes a distance you can compute.
Computers store words as text and items as ID numbers. Neither can be compared by meaning: the strings "car" and "automobile" share no letters, and item #4012 is not "closer" to #4013 just because the numbers are adjacent. We need a representation where distance means similarity.
An embedding is that representation: a fixed-length list of numbers (a vector) that an model produces for a word, sentence, image, or product. Similar things get vectors that point in similar directions, so meaning becomes geometry you can measure.
When a shopping site shows "customers also viewed" or a help search finds the right article even though you used different words, embeddings are usually behind it. Each product or article was embedded once; your query is embedded live and matched to the nearest vectors.
The takeaway pattern: embed your items once, store the vectors, then compare any new input against them by distance.
Slide to see how content is turned into a vector you can compare.
Take "dog", "puppy", and "calculator". Your intuition already says the first two are close in meaning.
A good embedding gives "dog" and "puppy" vectors that point in nearly the same direction, while "calculator" points elsewhere.
Cosine similarity returns about 0.8 for dog/puppy and about 0.1 for dog/calculator. The number, not the spelling, captures meaning.
Each number in the vector is a dimension the model learned during training. No single dimension means "is an animal" in a human-readable way; meaning is spread across all of them. That is why embeddings are called distributed representations.
See where the numbers come from: a model is trained so that things appearing in similar contexts get similar vectors. This is what "learned" means and why the geometry carries meaning instead of being random.
Embeddings are learned by training a model to predict context, pulling related items together.
Why this matters: It matters because 'learned' is the difference between geometry that means something and random numbers.
The signal is context. Words that appear in similar surroundings tend to mean similar things — "king" and "queen" both show up near "throne", "royal", "reign". A model is trained to predict the context of an item, and the vectors that make those predictions easy end up encoding meaning.
During training a score measures how wrong each prediction is. The model adjusts vectors to lower the loss, which pulls items that share context together and pushes unrelated items apart. Run this over a huge corpus and the geometry settles into something meaningful.
Because relationships are encoded as directions, you can do arithmetic on the vectors. Taking the vector for "king", subtracting "man", and adding "woman" lands you very close to the vector for "queen".
No one programmed that rule. It fell out of training on context — which is the whole point of calling embeddings learned representations.
The model reads "the cat sat on the ___" and must predict the missing word from the current vectors.
It guesses "car"; the real word was "mat". The loss is high because the guess was far off.
It shifts the vectors a little so "mat" becomes more likely next time. Repeat billions of times and similar contexts produce similar vectors.
Because the model learns from patterns of context rather than memorizing exact strings, it achieves : a word or sentence it has rarely seen still lands near related ones, as long as it appeared in similar contexts. That is what lets embeddings handle paraphrases and typos gracefully.
The same text gives different vectors depending on three choices: which model produces them, how many dimensions they have, and which distance metric scores similarity. This module makes those tradeoffs concrete so you choose deliberately.
Embedding dimension, source model, and distance metric are the levers you actually choose.
Why this matters: It matters because the same text gives different results depending on these choices.
Decision this forces: Choose the right level of rigor for Embeddings as Learned Representations: baseline, production design, or advanced optimization.
You rarely train embeddings yourself; you call a that produces them. But you still make three decisions that change quality, cost, and risk.
The trap: these are not free choices that you can mix later. Every vector in one search index must come from the same model and dimension, or the distances become meaningless.
Pick a small, well-known general model (for example a 384-dimension sentence model) and cosine similarity. It works surprisingly often.
Only reach for a larger or domain-tuned model after you can show the default misses real queries — otherwise you pay more for no gain.
Record the model name and dimension next to your stored vectors so you never accidentally mix two models in one index.
| Option | Fit | Operational burden | Risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Small general model (~384 dims) | Good for everyday text, search, and prototypes. | Cheap to run; can embed locally. | May miss specialized jargon or long documents. | Start here for almost any first build. | Low | Low |
| Large hosted model (~1536 dims) | Stronger on nuance and longer text. | Per-call cost and an API dependency. | More memory and slower search at scale. | Use when the small model demonstrably misses real queries. | Medium | Medium |
| Domain-tuned model | Best on niche vocabulary (legal, medical, code). | Needs sourcing, evaluation, and maintenance. | Can underperform off-domain inputs. | Use only after measuring that general models fail on your domain. | High | High |
Bad similarity results usually trace back to a choice made here, not to the math. The three most common ways it goes wrong:
You re-embed half your items with a new model. Old and new vectors live in different spaces, so distances are nonsense and results look random. Always re-embed the whole when you change models.
Using raw Euclidean distance on vectors meant for cosine similarity lets long vectors dominate, so popular-but-irrelevant items rank first.
More dimensions feels safer but multiplies memory and search cost while often adding little quality. Pick a default and only grow it on evidence.
Put it together: embed your items once, embed an incoming query, and rank items by cosine similarity. That three-step shape is the core of semantic search and retrieval, and it is only a few lines of code.
In practice you embed everything once, then compare a new query against stored vectors.
Why this matters: It matters because semantic search and retrieval are just nearest-neighbour lookups over embeddings.
Almost every use of embeddings is the same three-step shape, run by an model:
That is semantic search. The same shape powers retrieval for chatbots, recommendations, and deduplication — only the items change.
Query: "a feline pet". Even though none of those words appear in the documents, the ranked output is: "A cat is a small pet" (0.71), "Dogs are loyal animals" (0.39), "The stock market fell" (0.04).
Notice the top hit shares zero keywords with the query — it wins purely on meaning. That is the payoff of embeddings over exact-match search.
from sentence_transformers import SentenceTransformer, util model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim general model docs = ["A cat is a small pet", "Dogs are loyal animals", "The stock market fell"] doc_vecs = model.encode(docs) # embed items ONCE query_vec = model.encode("a feline pet") # embed the query LIVE scores = util.cos_sim(query_vec, doc_vecs)[0] # cosine similarity to each doc best = scores.argmax() print(docs[best], float(scores[best]))
model.encodeutil.cos_simargmaxSame model for docs and query, store the doc vectors once, and let cosine similarity pick the closest. Swap in any list of items and you have semantic search.
It prints "A cat is a small pet" with a score around 0.7. "feline" and "cat" are near each other in the embedding space, so that doc has the highest cosine similarity even though it shares no words with the query.
Each document string is turned into a vector and kept in a list, paired with its text.
The query "a feline pet" is embedded with the same model, so it lands in the same space as the documents.
Cosine similarity scores every document; the highest score wins. "A cat is a small pet" beats "The stock market fell" even with no shared keywords.
When an AI writes embedding code for you, check these before trusting it:
Embeddings fail quietly: a wrong-but-confident similarity score looks just like a right one. This module recalls the core idea, names the realistic failure modes, and gives you a validation habit using labelled pairs.
Embeddings break quietly, so you check them with labelled pairs and a recall test.
Why this matters: It matters because a wrong-but-confident similarity score is worse than an obvious error.
Decision this forces: Choose how much validation an embedding use deserves: eyeball it, a labelled-pair test, or monitored retrieval metrics.
Module 1: an embedding turns meaning into a vector, so similarity becomes a distance — high cosine similarity means close in meaning. Module 2: those vectors are learned by training a to predict context, which is why the geometry carries meaning instead of being arbitrary.
Hold both ideas: the danger is that a vector always returns a confident distance, even when the embedding is the wrong tool or has gone stale. So we validate.
Write 10-20 pairs you KNOW should be similar and a few you know should not. This is your ground truth.
Confirm the should-be-similar pairs score higher than the should-be-different ones. If not, your model or metric is wrong for this data.
Changed the model, dimension, or re-indexed? Run the same labelled set again. A silent drop here is your early warning of a broken pipeline.
| Option | Fit | Operational burden | Risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Eyeball a few queries | Fine for a prototype or demo. | Almost none — just look at results. | Misses systematic errors you didn't think to test. | Use while you are still exploring whether embeddings even help. | Low | Low |
| Labelled-pair test | Catches most real failures before users do. | Write 10-20 pairs once; re-run on change. | Low if the pairs reflect real usage. | Use for anything users will actually depend on. | Medium | Medium |
| Monitored retrieval metrics | Best for high-stakes or large-scale search. | Needs logging, recall@k tracking, and alerts. | Lowest — drift is caught automatically. | Use when wrong results cost money, safety, or trust. | High | High |
Embeddings rarely throw an error when they go wrong. Watch for these silent failures:
Close the book for a moment and reconstruct the chain from memory: What is an embedding? Why is it called learned? Which three choices shape it? What three steps turn it into search? And how do you check it before trusting it?
If you can answer those without peeking, you hold the whole lesson: meaning becomes a vector, training makes that geometry meaningful, model/dimension/metric are your levers, embed-then-rank is the pattern, and a labelled-pair test is the guardrail.
Solo build: take 20 short documents of your own, embed them with a small sentence model, and write a function that returns the 3 most similar to any query using cosine similarity. Then add a labelled-pair test of 5 known-similar and 5 known-different pairs and confirm the ordering holds. NEXT RUNG: store the vectors in a vector database (e.g. FAISS or a hosted index) and feed the top results into an LLM prompt — that is retrieval-augmented generation (RAG), the natural next lesson.
Raw text like the word 'cat' and an ID number like 42 both fail at the same job. What can embeddings do that those representations cannot?
Embeddings encode meaning as a fixed-length vector, so items that are semantically similar end up geometrically close — something raw text strings and arbitrary ID numbers cannot do.
During training, a loss function shapes what the embedding space looks like. Which statement best describes what it does?
The loss function is the training signal that makes the geometry meaningful — rewarding closeness for related pairs and penalising it for unrelated ones.
You have two embedding indexes: one built with Model A and one built with Model B. A colleague suggests merging them into a single index to simplify search. What is the key risk?
Each model learns its own coordinate system; mixing vectors from different models is like mixing measurements in metres and miles — the numbers are incomparable.
In a minimal embed-then-rank pipeline, what is the role of the single line that computes similarity scores between the query vector and every item vector?
The similarity computation is the decision point of the pipeline — it is what converts raw vectors into a ranked list, making it the line that determines which item wins.
Your embedding-powered search worked great at launch, but six months later users complain it returns irrelevant results. Which combination of failure modes should you investigate first?
Stale vectors and out-of-domain inputs are the two most common causes of embedding quality degrading over time — content changes and user language shifts can both silently break retrieval.