See how text becomes tokens and why context is a scarce design budget.
0/0
A model never sees your characters or words directly — it sees tokens, small word-pieces produced by a tokenizer. This module explains what a token is and why models work this way before we count them.
A token is the chunk of text a model encodes; usually a word-piece, not a whole word.
Why this matters: Everything else (cost, limits, truncation) is measured in tokens, so this is the base unit.
After this you'll be able to
Explain what a token is and how it differs from a word or a character.
Describe why models use sub-word tokens instead of whole words.
Estimate roughly how many tokens a short English passage uses.
Why this exists: a model can only do math on numbers, not letters. Tokenization is the step that turns your text into a sequence of numbered pieces the model can process. Almost every practical limit you hit — cost, length caps, truncation — is counted in those pieces, so it pays to know what they are.
Before reading on, predict: if you typed the word "unbelievable", do you think the model reads it as one piece, or several? Hold your guess.
A token is a piece of text, not a word
In plain wordsThink of tokens like LEGO bricks for language: common words are one brick, but a rarer word like "unbelievable" gets built from a few bricks ("un", "bel", "iev", "able").
When you send text to an , it is first split by a into a list of . A token is usually a common word or a word-piece — not always a whole word, and not a single letter.
Common words are usually one token: "the", "cat", and "run" are each a single token.
Rare or long words split into several: "unbelievable" may become "un" + "believ" + "able".
Spaces and punctuation count too: a leading space is often part of the token, and "." or "," can be their own token.
A useful rule of thumb for English: about 1 token ≈ 4 characters ≈ ¾ of a word. So 100 words is roughly 130 tokens. (The answer to the prediction above: "unbelievable" is usually several tokens, not one.)
Take the sentence: "Tokenizers split unbelievable words."
A typical tokenizer might break it into these eight tokens (a leading space is shown as ·):
"Token" · "izers" — even a common-looking word can split into a stem plus an ending.
" split" — one token, with its leading space attached.
" un" · "believ" · "able" — the rare word "unbelievable" becomes three pieces.
" words" · "." — the period is its own token.
Five English words became eight tokens. That gap is exactly why you should count tokens rather than words when a limit matters.
Granularity: characters → tokens → words
Slide to see the unit a model could split text on, and why sub-word tokens are the practical middle ground.
How the tokenizer learns its pieces
Modern tokenizers use an algorithm such as Byte-Pair Encoding (BPE). It starts from individual bytes and repeatedly merges the most frequently adjacent pair into a new token, building a fixed vocabulary (often 50,000–100,000 tokens).
Practical consequence: the same tokenizer is used to encode AND decode, so the model and your token-counting tool must use the model's own tokenizer. Counting with the wrong one gives wrong numbers.
Which statement best describes what a token is?
Tokens are sub-word pieces. Common words are usually one token; rare or long words split into several. They are not characters and not always whole words.
Now that text is tokens, the next question is how many the model can process in a single call. That ceiling is the context window — and it counts both your prompt and the model's reply.
The context window is the maximum number of tokens (prompt plus reply) a model processes per call.
Why this matters: It sets a hard ceiling on how much you can send and how long a reply can be.
After this you'll be able to
Define the context window as a fixed token budget per request.
Explain that the window holds the prompt AND the generated reply together.
Recognize what happens when input exceeds the window.
Quick recall from module 1: before reading on, what is the rough rule of thumb for English — about how many tokens is one word?
Answer: roughly 1 word ≈ 1.3 tokens (or 1 token ≈ ¾ of a word). Hold that number — you will use it in a moment to check whether text fits the window.
The window is a fixed token budget
In plain wordsThe context window is like a whiteboard of a fixed size: you can write a lot, but once it's full, nothing more fits — and whatever the model writes back has to share the same board.
A is the maximum number of a model can take in for a single request. It is a hard limit set by the model, commonly 8K, 128K, or up to ~1M tokens depending on the model.
The crucial part beginners miss: the window holds the AND the reply together. If the window is 8,000 tokens and your prompt is 7,500, the model has at most 500 tokens left to answer in.
Input tokens: your instructions, the conversation history, any documents you paste in.
Output tokens: everything the model generates back.
Both share one budget: input + output must fit inside the window.
You want to summarize a 10,000-word report using a model with an 8,000-token window.
Estimate the input: 10,000 words × 1.3 ≈ 13,000 tokens.
Compare to the window: 13,000 > 8,000 — it does not fit, even before leaving room for the summary.
The fix: split the report into chunks that each fit, summarize each, then summarize the summaries (a "map-reduce" approach).
The window is why "just paste the whole thing in" stops working past a certain size — and why chunking exists.
Filling the window
Slide to see how much of an 8K-token window the prompt uses — and how little room is left for the reply.
Window size vs. attention cost
Bigger windows are not free. The transformer's attention compares tokens against each other, so cost grows steeply as the window grows. A larger window also means more input tokens to pay for and slower responses.
So "use the biggest window" is rarely the right default — match the window to the job, and keep the prompt lean.
A model has an 8,000-token context window. Your prompt is 7,800 tokens. What is the main problem?
The window is shared by input and output. A 7,800-token prompt leaves only ~200 tokens for the reply, so the model cannot produce a full answer.
The window is finite and tokens cost money, so every request is a budgeting decision. This module is about estimating token counts and choosing how to split the window between input and output.
Budgeting means measuring tokens and choosing what to include so the request fits the window.
Why this matters: Tokens cost money and the window is finite, so every prompt is a budgeting decision.
After this you'll be able to
Estimate token counts for a prompt before sending it.
Reserve output room so the reply isn't truncated.
Choose a strategy when content is too big for the window.
Decision this forces: When content exceeds the window, choose how to fit it: truncate, summarize, or retrieve only the relevant chunks.
Every prompt is a budget
In plain wordsBudgeting tokens is like packing a fixed-size suitcase: you can't bring everything, so you decide what's essential, what to compress, and how much space to leave for what you'll buy on the trip (the reply).
Because the is fixed and you usually pay per , building a means answering three questions:
How many tokens does my input use? Estimate (words × 1.3) or count exactly with the model's tokenizer.
How many tokens does the reply need? Reserve that room up front — set a max-output limit.
What do I cut if it doesn't fit? Trim history, summarize, or retrieve only the relevant passages instead of pasting everything.
How to budget a request, step by step
Measure the fixed parts
Count the tokens in your system instructions and any required context. These are non-negotiable and come out of the budget first.
Reserve output room
Decide the longest reply you need (e.g. 800 tokens) and subtract it from the window. The rest is what's left for variable input.
Fit the variable input
Fill the remaining budget with the highest-value content. If it overflows, summarize or retrieve only the relevant pieces rather than truncating blindly.
What to do when content is too big for the window
Option
Fit
Effort
Risk of losing information
When to choose
Cost
Complexity
Truncate (cut the overflow)
Always fits — you just drop tokens past the limit.
Almost none.
High — you may cut the most important part without noticing.
Only for clearly-trailing, low-value text (e.g. boilerplate footers).
Low
Low
Summarize then send
Fits, since the summary is far smaller.
Extra model calls to produce summaries.
Medium — detail is compressed but the gist survives.
When you need the overall meaning of a large document, not exact wording.
Medium
Medium
Retrieve only relevant chunks
Fits — you send only the passages that match the question.
Needs a search/retrieval step over chunks.
Low for the question at hand, if retrieval is good.
When the answer lives in a small part of a large corpus (the typical RAG case).
Medium
Medium
Pick the smallest-effort option whose information-loss risk you can accept for THIS task. Default to retrieval when exact details matter.
You need exact figures from one section of a 200-page manual. Which budgeting strategy fits best?
When you need exact details from a small part of a large source, retrieval keeps the relevant text intact while fitting the window. Truncation risks cutting the section; summarizing loses the exact figures.
Eyeballing token counts is unreliable. This module shows a tiny, runnable sketch that counts tokens with the model's own tokenizer before you send a request, and how to sanity-check AI-generated counting code.
A minimal code sketch encodes text with a tokenizer and reports the count before you send it.
Why this matters: Estimating tokens by eye is unreliable; a quick count prevents over-limit errors and surprise cost.
After this you'll be able to
Read a minimal token-counting snippet and explain each line.
Predict the token count of a short string before running it.
Verify AI-generated counting code uses the correct tokenizer.
Count, don't guess
The (words × 1.3) rule is fine for a quick gut-check, but it's wrong for code, non-English text, and unusual formatting. To know for sure, run the text through the model's and read the real count.
Before reading the code, predict: how many tokens do you think the string "Hello, world!" is — 1, 3, or 4?
import tiktoken
# Load the tokenizer that matches your model.
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Hello, world!"
tokens = enc.encode(text) # text -> list of token idsprint(len(tokens)) # how many tokens this text uses# Fits a budget?
MAX_INPUT = 7000
assert len(tokens) <= MAX_INPUT, "Prompt too long for the window!"
Syntax breakdown
encoding_for_model("gpt-4o")
Picks the exact tokenizer that model uses — the count is only correct if this matches.
enc.encode(text)
Turns the string into the list of integer token ids the model would see.
len(tokens)
The token count — what you compare against the window and what you're billed on.
assert ... <= MAX_INPUT
A guardrail that stops the request early if the prompt won't fit.
This loads the model's own tokenizer, encodes the text into token ids, and reports the count. The assert line is the cheap guardrail: it fails loudly before you spend a request on an over-limit prompt.
Predict: How many tokens is "Hello, world!" — and which characters become their own tokens?
Four tokens: "Hello", ",", " world", "!". The comma and exclamation mark are separate tokens, and " world" carries its leading space.
Verify AI-generated counting code before you trust its numbers. When an assistant writes token-counting code for you, check these four things:
Right tokenizer? It must match your actual model. Counting GPT text with a Llama tokenizer gives wrong numbers.
Counts input AND output? A budget check that ignores the reply will still overflow the window.
Handles the leading space / special tokens? Chat models add hidden formatting tokens per message; a raw string count undercounts.
Fails loudly, not silently? There should be an assert or check, not a comment that says "should be fine".
An assistant gives you token-counting code that uses a different model's tokenizer than the one you'll call. What's the risk?
Different models tokenize differently, so the same text yields different token counts. Using the wrong tokenizer gives a misleading number and you can still overflow the real window.
Knowing the happy path isn't enough. This module covers the classic ways tokenization and context windows bite in practice — silent truncation, the lost middle, and runaway cost — then a short self-check to lock it in.
Long inputs silently truncate, important text in the middle gets ignored, and token sprawl inflates cost.
Why this matters: Knowing how the window fails is what turns the concept into a reliable habit.
After this you'll be able to
Name the three most common context-window failure modes.
Explain why long inputs can be ignored even when they technically fit.
Recall the token rule of thumb and the shared-budget idea from earlier modules.
Decision this forces: As a conversation grows, decide what to keep in the window each turn: full history, trimmed history, or a running summary.
Before reading on, predict from memory: in module 2 we said the context window holds two things that share one budget. What are they?
Answer: the (input) and the reply (output) share the same window. Keep that in mind — the first failure mode below is exactly what happens when you forget it.
How tokens and windows break in practice
These three failures cause most real-world surprises with .
1. Silent truncation (the output gets cut)
The prompt is so large that almost no room is left for the reply. The model stops mid-sentence and the answer looks complete but isn't. Fix: reserve output tokens up front.
2. Lost in the middle (it fits but gets ignored)
Even when a long input fits, models attend most strongly to the beginning and end. Information buried in the middle of a very long prompt is often effectively ignored — so "it fit" does not mean "it was used". Fix: put the most important text near the start or end, and don't over-stuff the window.
3. Runaway cost (token sprawl)
In a chat loop, the whole history is re-sent every turn, so usage — and cost — grows with each message. A long conversation quietly becomes expensive. Fix: trim or summarize old turns instead of carrying everything.
Diagnostic signal: if a model ignores an instruction you definitely included, suspect either truncation (it was cut) or lost-in-the-middle (it was buried), not that the model "can't follow directions".
Use when
A conversation or input is growing long and you must decide what to keep in the window each turn.
Avoid when
The whole input is small and comfortably under the window — trimming then just adds complexity for no benefit.
Rule of thumb
Keep the system instructions and the most recent, most relevant turns; summarize or drop the rest. Always leave headroom for the reply.
You pasted a long instruction into the middle of a very large prompt that fit the window, but the model ignored it. What is the most likely cause?
If the prompt fit but a middle instruction was ignored, the cause is usually lost-in-the-middle. Move critical instructions toward the start or end, and avoid over-stuffing the window.
Putting it together
Synthesis
Reconstruct the chain from memory before reading on: text becomes ____, those fill a fixed ____ shared by prompt and reply, so every request is a ____ decision, which you verify by ____ the tokens, while watching for truncation, lost-in-the-middle, and runaway cost.
The fill-ins: tokens → context window → budgeting → counting. If you can rebuild that sentence, you have the lesson.
Reference architecture
What a token is
The unit a model actually reads.
The context window
How many tokens fit at once.
Counting and budgeting
Decide what gets in and what to trim.
Counting in code
A small, runnable check.
Suggested build order
See text as tokens
Treat input as token pieces, not words — count in tokens whenever a limit matters.
Know your window
Find the model's context window and remember the prompt and reply share it.
Budget the request
Reserve output room, then fit the highest-value input into what's left.
Count in code
Encode with the model's own tokenizer and assert the prompt fits before sending.
Guard the failure modes
Watch for truncation, lost-in-the-middle, and runaway cost; trim or retrieve as the input grows.
Decision checklist
Can you estimate a prompt's token count before sending it?
Did you leave room in the window for the reply, not just the prompt?
When content was too big, did you choose truncate / summarize / retrieve deliberately?
Are you guarding against truncation, lost-in-the-middle, and runaway cost?
Try it:
Solo build: write a short script that takes a long document and a target model, estimates its token count with the model's tokenizer, reports whether it fits the window with room for a 1,000-token reply, and — if not — picks one strategy (truncate, summarize, or retrieve) and explains why. NEXT rung: extend it into a chunking-and-summarization pipeline (map-reduce) for documents larger than the window.
Before looking back at the lesson, which statement best describes what a token is?
Models read text in tokens, which are pieces of text that do not always match word or character boundaries.
You have a model with an 8,000-token context window and you send a 7,500-token prompt asking for a detailed report. What is the main budgeting problem?
The context window is a shared budget for both the input prompt and the generated output.
A document is too large to fit in the context window, but you only need the section about refund policy. What is usually the best strategy?
When content is too big, selecting relevant chunks preserves the information needed while staying within the token budget.
In one or two sentences, explain what you should check before trusting AI-generated code that counts tokens for a specific model.
Reference answer: Check that the code uses the correct tokenizer or encoding for the target model, because different models may split the same text into different tokens; also verify it on a short test string if possible.
Token counts are only useful if the code uses the tokenizer that matches the model you plan to call.
A long conversation still technically fits in the context window, but the model starts missing earlier instructions. What response best applies the lesson’s guidance?
Long inputs can still lead to ignored details, so conversations often need trimming or a running summary to keep important context prominent.