Nearly every machine learning model learns the same way: a loss function scores how wrong its predictions are, and gradient descent repeatedly nudges…
Understand loss as the score a model is trying to make as small as possible.
Learning needs a target to aim at. A loss function turns 'how wrong were the predictions?' into a single number the model can try to shrink. If the model predicts perfectly, the loss is near zero; the worse it does, the higher the loss.
The loss you choose defines what 'good' means. For predicting numbers, mean squared error averages the squared gap between prediction and truth. For classification, cross-entropy penalizes confident wrong answers heavily. Pick the loss that matches your task.
See how following the gradient walks the model downhill to lower loss.
Picture the loss as a hilly surface where every point is a setting of the model's parameters and the height is the loss. Training wants the lowest valley. The gradient at your current point tells you which way is steepest uphill — so you step the opposite way, downhill, to reduce the loss.
Repeat: measure the gradient, take a small step down, measure again. Each step lowers the loss a little, and over many steps the parameters settle near a minimum where predictions are good.
Computing the gradient over the entire dataset every step is accurate but slow. Stochastic gradient descent (SGD) instead estimates it from one example or a small mini-batch, so it takes many quick, slightly noisy steps. Mini-batch SGD is the standard: fast, and the noise even helps escape shallow traps.
Tune the step size — the single most important training knob.
The learning rate sets how big each downhill step is. Too large and you leap past the valley and bounce around or diverge, the loss jumping or shooting to infinity. Too small and training crawls, taking forever to improve.
There is a sweet spot in the middle. In practice people start around a small value, watch the loss curve, and often decay the rate over time — big steps early, careful steps as they near the minimum.
loss = mse(predict(x, w), y) # how wrong we are grad = d_loss / d_w # slope of loss vs each weight w = w - lr * grad # step downhill # repeat for many batches until loss stops falling
The whole learning loop in three lines: score the error, find the slope of the loss with respect to each weight, and nudge the weights against that slope by lr. The learning rate lr scales the step — the term that most decides whether training converges or blows up.
Diagnose the loss-curve symptoms that reveal a broken training run.
The loss curve tells you what is wrong. Loss exploding to NaN or oscillating wildly means the learning rate is too high — lower it. Loss barely moving means it is too low, or the features are unscaled — raise the rate or normalize inputs. Loss dropping on training data but rising on validation means overfitting — add regularization or more data.
Also scale your inputs: wildly different feature ranges make the loss surface lopsided and gradient descent zig-zag. Watch training and validation loss together, not just the final number.
Models learn by minimizing a loss function — a single number measuring prediction error — using gradient descent, which repeatedly steps the parameters opposite the gradient to go downhill. The learning rate sets the step size and is the key knob: too high diverges, too low crawls. Mini-batch SGD makes this fast. The loss curve diagnoses failures: exploding loss means lower the rate, a training-vs-validation gap means overfitting.
Imagine training a model whose loss jumps around and never settles. Decide which knob you would change first and in which direction, name one input-preparation step that could also be at fault, and describe what a healthy loss curve should look like.
What does a loss function measure?
The loss quantifies error — near zero for perfect predictions — and defines the target that gradient descent minimizes; the choice of loss defines what 'good' means for the task.
How does gradient descent reduce the loss?
Each step moves parameters against the gradient, lowering the loss a little; over many steps the model reaches a low-loss valley.
What happens if the learning rate is too high?
Too-large steps leap past the valley and destabilize training; too-small steps make it crawl, so the learning rate is the key knob to tune.
Your training loss falls but validation loss rises. What is happening?
A gap where training improves but validation worsens is the classic overfitting signature, addressed with regularization, more data, or early stopping.