Pretraining is the phase where a large language model learns language and knowledge by predicting the next token across a massive text corpus…
Understand the next-token objective that teaches a base model language.
Pretraining teaches a model by a deceptively simple game: given a stretch of text, predict the next token. Run this over trillions of tokens of books, web pages, and code, and to predict well the model must absorb grammar, facts, reasoning patterns, and style — because all of those help guess what comes next.
The output is a base model: broadly capable but not yet aligned to follow instructions or chat. Later stages — instruction tuning and preference tuning — shape that raw capability into a helpful assistant. Pretraining is where the knowledge and the cost live.
See the predictable relationship between scale and model quality.
Scaling laws are the empirical observation that a model's pretraining loss decreases in a smooth, predictable way as you increase three resources: the number of parameters, the amount of training data, and the compute spent. Plotted on log scales, the improvement follows clean curves.
This predictability is powerful: teams can train small models, fit the curve, and forecast how a much larger run will perform before spending millions on it. Scaling laws turned 'make it bigger' from a hope into a plan.
Learn why balancing model size and data beats simply going bigger.
Early on, teams poured compute into ever-larger models but trained them on too little data. The Chinchilla study showed this was suboptimal: for a fixed compute budget, model size and training tokens should scale together, roughly in balance — a guideline of about 20 training tokens per parameter.
A Chinchilla-optimal model that is smaller but trained on much more data can beat a larger, undertrained one at the same compute — and being smaller, it is also cheaper and faster to run at inference. This reshaped how modern models are sized.
Because a model is trained once but served millions of times, teams often train smaller models on even more data than compute-optimal suggests — accepting higher training cost for a cheaper, faster model at inference.
Connect scale to new abilities and to how you choose a model.
Beyond smoothly falling loss, larger models can show emergent abilities — capabilities like multi-step arithmetic or in-context learning that are near-absent in small models and appear once scale crosses a threshold. How sharp and real these jumps are is debated, but the practical point holds: some tasks simply need a bigger model.
For a practitioner, the lessons are concrete. Bigger is not free — cost and latency rise with size — so pick the smallest model that clears your task's bar. And a smaller model trained on more data can outperform a larger, undertrained one, so parameter count alone is a poor quality signal.
Watch for: judging a model by parameter count alone (training data and method matter as much); assuming more scale always helps your specific task (data quality and fine-tuning often matter more downstream); and ignoring that the biggest model is also the most expensive to serve.
Pretraining builds a base model by predicting the next token over a massive corpus, absorbing language and knowledge. Scaling laws show loss falls predictably as parameters, data, and compute grow together, letting teams forecast large runs. Chinchilla refined this: balance model size with training tokens (~20 per parameter), since a smaller, well-trained model beats a larger undertrained one and is cheaper to serve. Some abilities emerge only at scale, but parameter count alone is a weak quality signal.
You have a fixed compute budget for a new base model. Explain how scaling laws and the Chinchilla rule would guide your choice of model size versus training tokens, and why you might train even smaller-and-longer than compute-optimal given inference costs.
What is the pretraining objective of a large language model?
Next-token prediction over trillions of tokens produces a general base model; instruction and preference tuning refine it afterward.
What do scaling laws describe?
Scaling laws are smooth empirical curves that let teams forecast a large model's performance from smaller runs.
What was the key Chinchilla insight?
Chinchilla showed a smaller model trained on more data can beat a larger, undertrained one at equal compute — and is cheaper to serve.
What is an emergent ability?
Some abilities appear as a threshold effect with scale; the practical takeaway is that certain tasks require a sufficiently large model.