A large language model does not read characters or words — it reads tokens, chunks of text (roughly four characters or three-quarters of a word in…
See how text is chopped into the tokens a model actually reads.
Before a model sees your text, a tokenizer splits it into tokens — pieces that are usually a few characters long. A rough rule for English is that one token is about four characters, or roughly three-quarters of a word, so 100 tokens is about 75 words.
Each token maps to an ID number, and the model works entirely with those IDs. Common words are often a single token, while rare or long words split into several. Spaces and punctuation count too.
"tokenization" -> ["token", "ization"] (2 tokens) "cat" -> ["cat"] (1 token) "antidisestablish" -> ["anti", "dis", "establish"] (3 tokens) " \n" -> whitespace also becomes tokens
A common word like 'cat' is one token; a longer or rarer word breaks into familiar sub-pieces. Even spaces and newlines are tokenized, which is why token counts don't match word counts exactly.
Understand why models tokenize into sub-pieces rather than whole words.
A model needs a fixed vocabulary of tokens. Using whole words would make it enormous and still miss new words, typos, and names. Using single characters would make sequences too long. Subword tokenization is the middle path: keep common words whole, and build rare words from smaller pieces.
This means any string can be represented, even one never seen in training. One consequence: languages underrepresented in training, and code or non-Latin scripts, often use more tokens per idea — the same sentence can cost more tokens in one language than another.
Learn the token budget that bounds everything a model can consider.
The context window is the maximum number of tokens a model can process in a single call. Critically, it is shared: the system prompt, the conversation history, any retrieved documents, and the model's own response all draw from the same budget. If the input fills the window, there is no room left to answer.
Models advertise their window size — from a few thousand tokens to hundreds of thousands or more. Everything the model can 'see' at once must fit; anything beyond it is outside the model's attention for that call.
A large context window lets you include more, but stuffing it full raises cost and latency and can bury the important parts so the model attends less well to any one of them. Include what the task needs, not everything you have.
Connect tokens and windows to cost, truncation, and real limits.
Tokens are the unit of billing: providers charge per input and output token, so a longer prompt or reply costs more, and trimming needless text directly saves money. Tokens are also the unit of limits — max input and max output are token counts, not word counts.
When your text exceeds the window, you must shorten it: truncate, summarize, or retrieve only the relevant parts (as RAG does) instead of pasting whole documents. Silently overflowing the window drops content the model then can't use.
Watch for: assuming word count equals token count (it doesn't, and it varies by language and code); forgetting the response shares the window, so leaving no room for the answer; and pasting entire documents when only a few passages are relevant. Count tokens with the model's tokenizer when budgeting.
Language models read tokens — subword chunks roughly four characters or 0.75 words each — produced by a tokenizer, not raw words or letters. Subword tokenization keeps the vocabulary small while representing any string. The context window is the shared token budget for prompt plus response; everything must fit. Because tokens are the unit of both pricing and limits, managing them controls cost and prevents truncation — count with the model's tokenizer and include only what the task needs.
You need to answer questions over a 200-page manual with a model whose context window can't hold it. Explain why pasting the whole manual fails, estimate roughly how tokens relate to its word count, and describe how you would fit only the relevant content into the window.
What is a token in a language model?
Models read tokens, not words or letters; common words are often one token while rare ones split into several subword pieces.
Why do models use subword tokenization?
Subwords balance vocabulary size and sequence length, letting the model spell any string from a fixed set of pieces.
What does the context window include?
Input and output share the context window, so a prompt that fills it leaves no room for an answer.
Why does tokenization matter in practice?
Billing and max-length limits are counted in tokens, so managing them controls cost and prevents dropped content.