Attention is the mechanism inside a transformer that lets the model weigh how much each token in the input should influence every other token when…
Understand attention as a way for each token to gather context from the whole sequence at once.
Attention lets a transformer decide, for every token, which other tokens matter most to its meaning. In the sentence 'the bank of the river,' the word 'bank' should attend strongly to 'river' to resolve its sense — attention is what supplies that pull.
The output for each token is a weighted blend of information from all tokens, where the weights say how relevant each one is. Nothing is hard-coded; the model learns these relationships from data.
Break attention into the three vectors that decide what each token retrieves.
For each token the model computes three vectors. The query says what this token is looking for. The key says what each token offers. The value is the information a token carries if it is attended to.
A token's query is compared against every key to produce scores; high score means 'this token is relevant to me.' Think of it as a soft dictionary lookup where the match is by similarity, not exact keys.
scores = Q · Kᵀ # every query dotted with every key scores = scores / sqrt(d) # scale by key dimension weights = softmax(scores) # rows sum to 1 output = weights · V # blend the values
This is the whole attention operation. The dot products measure query-key similarity; dividing by sqrt(d) keeps the numbers stable; softmax turns them into weights that sum to 1; multiplying by V produces each token's context-aware output.
Distinguish self-attention within one sequence from running many attention heads in parallel.
Self-attention is attention applied within a single sequence: the queries, keys, and values all come from the same tokens, so each token attends to the others in its own sentence. This is what builds contextual meaning inside a transformer layer.
Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value projections. One head might track subject-verb agreement, another nearby word order, another long-range references.
Their outputs are concatenated and combined, so the layer captures several kinds of relationship at once instead of a single averaged one.
See why attention replaced recurrent models as the backbone of modern language models.
Older recurrent networks (RNNs) read a sequence one token at a time, which is slow to train and tends to forget information from far back. Attention compares all tokens directly, so distant words are one step apart, not hundreds.
It is also parallelizable: every token's attention is computed at the same time on modern hardware. The 2017 paper 'Attention Is All You Need' showed this design outperformed recurrence and became the foundation of today's large language models.
Transformer attention lets each token weigh every other token when forming meaning. Each token becomes a query, key, and value; queries score against keys, softmax turns scores into weights, and the weights blend the values into a context-aware output. Self-attention does this within one sequence, multi-head attention runs several in parallel, and this design's parallelism and long-range reach are why it replaced RNNs.
Take the sentence 'The trophy did not fit in the suitcase because it was too big.' Decide which word 'it' should attend to, and explain how a query-key match would give that word a high attention weight while other words stay low.
What is attention in a transformer?
Attention produces, for each token, a weighted blend of information from all tokens, with learned weights reflecting relevance.
What are query, key, and value vectors?
Queries are matched against keys to produce attention weights, which then blend the value vectors into each token's output.
What is multi-head attention?
Multiple heads let one layer track several patterns (word order, agreement, long-range links) simultaneously, then combine them.
Why did attention replace RNNs for sequence modeling?
RNNs process tokens sequentially and forget distant context; attention connects any two tokens in one step and parallelizes across hardware, which is why transformers took over.