A large language model (LLM) works by predicting the next token — a word or word-piece — over and over, given all the text so far. It was trained on…
Define an LLM and the token it operates on.
A large language model reads text as TOKENS — words or word-pieces ("embedding" might be one token, "tokenization" two). Its one job is to predict the next token given all the tokens so far.
To write a sentence it predicts a token, appends it, and predicts again — repeating until done. Everything an LLM does is this loop of next-token prediction.
Explain how a simple objective yields complex behaviour.
Predicting the next token sounds trivial, but to do it well across the whole internet the model must internalize grammar, facts, reasoning steps, and coding patterns — because all of those help predict what comes next.
At enough scale, abilities like translation and step-by-step reasoning EMERGE from that single objective, without being programmed in directly.
LLMs use the transformer architecture. Its attention mechanism lets each token look at and weigh every earlier token, so the model keeps track of context across a long passage.
Sketch pre-training then instruction/preference tuning.
First, PRE-TRAINING: the model reads enormous amounts of text, adjusting its billions of parameters to predict the next token better. This gives raw language ability.
Then, ALIGNMENT (instruction tuning + preference tuning like RLHF): the model is trained on examples of helpful, safe responses so it follows instructions and behaves like an assistant rather than an autocomplete.
Explain confident errors as a consequence of prediction.
Because the model generates statistically likely text — not verified lookups — it can produce a fluent, confident answer that is wrong. It has no built-in sense of 'I don't know.'
This is why grounding techniques like RAG (feeding it real documents) and verification matter for factual use.
An LLM predicts the next token over and over, using a transformer whose attention mechanism weighs all prior tokens. Pre-training on vast text builds language ability in billions of parameters; alignment tuning turns it into a helpful assistant. Because it predicts likely text rather than looking facts up, it can hallucinate — so ground and verify factual use.
Explain in three sentences why an LLM can write a flawless-sounding paragraph that contains a made-up fact — using the words token, prediction, and hallucination.
How does an LLM generate text?
Next-token prediction, repeated, is the whole mechanism — fluency and reasoning emerge from doing it well at scale.
What lets a transformer keep track of context across a passage?
Attention is the transformer's core: it relates each token to the others, capturing long-range context.
Why do LLMs hallucinate?
Prediction optimizes for plausibility, not truth — hence confident errors, mitigated by grounding and verification.