A neural network is a stack of layers of simple units that each compute a weighted sum of their inputs and pass it through a non-linear activation…
Build up a network from neurons to layers and see what it computes.
A neural network is built from neurons, each doing something simple: multiply its inputs by weights, add a bias, and pass the result through an activation function. Stack many neurons side by side and you have a layer; stack layers and the output of one feeds the input of the next.
The first layer sees the raw input, the last layer produces the prediction, and the hidden layers in between transform the data into increasingly useful representations. Learning means finding the weights and biases that make the final output correct.
Understand why activation functions are what give a network its power.
Without activation functions, stacking layers is pointless: a chain of weighted sums is still just one big weighted sum, so the network could only draw straight-line boundaries no matter how deep it is.
The non-linear activation breaks that. Applied after each layer, it lets the network bend and combine features into curved, complex decision boundaries. ReLU — which passes positive values through and zeroes out negatives — is the common default because it is simple and trains well.
Run data through the network to get a prediction and a loss.
The forward pass is the network making a prediction. The input flows into the first layer, each layer computes its weighted sums and activations, and the values propagate forward until the final layer emits an output — a number, or class probabilities.
That output is compared to the true answer with a loss function, giving a single error score. The forward pass produces the prediction; the loss says how far off it was, which is what training will try to reduce.
h = relu(W1 @ x + b1) # hidden layer: weighted sum + activation y = W2 @ h + b2 # output layer loss = cross_entropy(y, target)
Input x is transformed by the first layer's weights and a ReLU into hidden features h, then the output layer produces y, and the loss scores it against the target. Every W and b here is a knob backpropagation will nudge.
See how the network assigns blame backward and updates every weight.
Backpropagation answers 'how much did each weight contribute to the error?' Starting from the loss at the output, it applies the chain rule from calculus to pass the error backward layer by layer, computing the gradient of the loss for every weight and bias.
With those gradients, gradient descent nudges each weight in the direction that lowers the loss. One training step is: forward pass to get the loss, backward pass to get the gradients, then update. Repeat over many batches and the network learns.
In deep networks, gradients can shrink toward zero (vanishing) so early layers barely learn, or blow up (exploding) so training destabilizes. ReLU activations, careful weight initialization, normalization layers, and gradient clipping are the standard fixes. If a deep net won't learn, suspect the gradients first.
A neural network stacks layers of neurons, each computing a weighted sum and a non-linear activation, which is what lets it model complex, curved patterns. The forward pass turns input into a prediction and a loss. Backpropagation then works backward with the chain rule to compute every weight's gradient of the loss, and gradient descent updates them. Repeat forward-backward-update over many batches, and watch for vanishing or exploding gradients in deep nets.
Sketch a network for classifying handwritten digits: how many outputs it needs, why it needs hidden layers with activations rather than one linear layer, and what one training step (forward, backward, update) would do to its weights.
What is a neural network made of?
Neurons do weighted sums with activations; stacking layers lets the network transform inputs into useful representations and a final prediction.
Why are non-linear activation functions essential?
A chain of pure weighted sums is still linear; the non-linearity after each layer is what gives depth its expressive power.
What does the forward pass produce?
The forward pass computes the output layer by layer; the loss then measures its error, setting up the backward pass.
What does backpropagation do?
Backprop assigns each weight its share of the error as a gradient; those gradients drive the weight updates that make the network learn.