Learn queries, keys, values, heads, and why attention scaled language models.
Attention starts from token states, not raw text. This module grounds query-key-value in the vectors produced by embeddings and earlier layers.
Token states are the layer inputs attention operates on.
Why this matters: Without token states, query-key-value attention has nothing to compare.
Before attention, each visible token has a vector state. In the first layer, that state comes from token and positional embeddings. In later layers, it already contains information mixed from earlier attention and feed-forward blocks.
Problem anchor: a developer sees a diagram with Q, K, and V but cannot tell what they are projections of. They are projections of token states, not separate words floating outside the model.
The token “bank” starts with a learned representation. After attention sees “river,” later states can carry the river-bank sense more strongly than the finance-bank sense.
The model has not looked up a dictionary entry. It has updated a vector using context from surrounding tokens.
Queries ask what a position needs; keys advertise what each position contains. Their scores determine attention weights.
Query-key dot products produce attention scores.
Why this matters: Scores explain why some words influence the next prediction more than others.
Each token state is projected into a query and a key. A query represents what the current position is looking for. A key represents what a position can match. Their dot product becomes a score.
Scores are scaled, masked, and normalized. In a causal language model, the mask prevents a position from looking at future tokens during generation.
query = {"refund": 2, "weather": 0}
keys = {
"policy": {"refund": 3, "weather": 0},
"forecast": {"refund": 0, "weather": 4},
}
def dot(a, b):
return sum(a[k] * b[k] for k in a)
scores = {name: dot(query, key) for name, key in keys.items()}
print(scores)
print(max(scores, key=scores.get))input datadecision ruleprint(...)Run this as a tiny model of the mechanism. The point is to predict behavior before trusting the output.
It prints a higher score for policy, then policy. The refund query matches the refund-heavy key, so that position receives more attention in this toy example.
The value vectors are the information being blended into the next state. Multi-head attention lets the model run several such views in parallel.
Values carry information that scores decide how to mix.
Why this matters: This is the step that actually moves information between positions.
After scores become weights, the model takes a weighted sum of value vectors. The result is a new context-aware vector for that position.
Multi-head attention repeats this with separate learned projections. One head might track syntax, another might track references, and another might track local phrase patterns. The model learns those roles; we do not assign them by hand.
Sentence: “Maya gave Priya the notebook because she had the exam.” The token “she” needs context. Attention can weight earlier names and surrounding clues while building the next-token state.
The model may still pick the wrong referent if the sentence is ambiguous. Attention provides a route for information, not a guarantee of human-level interpretation.
Transformers need positional information because attention alone is permutation-friendly; causal models also mask future tokens.
Masks and position rules constrain which tokens can interact.
Why this matters: They explain autoregressive generation and order-sensitive behavior.
Self-attention compares positions, but it needs positional information to know which token came first, second, or later. Models use positional embeddings or related schemes such as rotary position methods.
Causal language models also use an attention mask so a position cannot attend to future tokens. During training, this lets many next-token predictions be learned in parallel without leaking the answer.
The same tokens appear in both phrases, but the order changes the meaning. Positional information lets the model distinguish subject, verb, and object relationships.
The causal mask adds another rule: while predicting the next token, the model can use earlier tokens, not future ones.
Attention is often over-sold as a transparent explanation. Use it carefully.
KV cache stores previous keys and values during decode. It saves compute but can dominate memory at long context or high concurrency.
KV cache stores attention keys and values from previous tokens.
Why this matters: It is central to long-context latency, memory pressure, and serving tradeoffs.
Generation has a prefill phase and a decode phase. Prefill processes the prompt and builds keys and values for every prompt token. Decode adds one token at a time.
A KV cache lets each new token attend over stored past keys and values instead of recomputing the entire prefix. That reduces repeated compute but stores tensors for every active sequence, layer, and head.
A chatbot handles 100 concurrent long conversations. Each one has a long prompt and keeps generating token by token. Throughput drops, not because attention stopped working, but because KV cache memory and scheduling now dominate serving.
From memory, trace one token through attention: state vector, query-key score, mask, softmax weights, weighted values, multi-head mixing, and KV cache reuse during decode.
Explain a slow long-context generation run to an engineer: identify prefill, decode, KV cache growth, and the memory/latency tradeoff. Next rung: study serving LLMs with vLLM or KV-cache optimization.
A token's state enters an attention layer as a vector. What does that vector represent at that point in the network?
Each layer refines the token's vector, so by the time it reaches a given attention layer it encodes contextual information built up across all prior layers — not just the original embedding.
During attention scoring, a query vector is compared against key vectors to produce raw scores. What role does softmax play immediately after?
Softmax turns the raw dot-product scores into a proper probability distribution (summing to 1), which then acts as a weighted recipe for blending the value vectors.
In multi-head attention, the model runs several attention operations in parallel rather than just one. In one or two sentences, explain the practical benefit of doing this.
Multiple heads let the model jointly attend to information from different representation subspaces, capturing richer and more varied relationships than a single attention operation could.
A decoder-only language model uses a causal mask during training. Which of the following best describes what the causal mask enforces?
The causal mask sets future-token scores to negative infinity before softmax, so they become zero weight — ensuring the model only uses past context when predicting the next token.
When serving a language model, the KV cache stores key and value vectors from the prompt so they don't need to be recomputed at every decode step. What is the primary memory cost this introduces?
The KV cache trades compute for memory: it avoids redundant recomputation but requires storing large K and V matrices for every layer across the full context, which becomes a significant bottleneck at long sequences or large batch sizes.