How Does Backpropagation Use the Chain Rule?

TL;DR

Backpropagation uses the chain rule to calculate how sensitive a network’s cost is to every weight, bias, and preceding activation. Multiplying local derivatives along each path, then adding contributions from multiple paths, produces the gradient components needed to adjust the network’s parameters and repeatedly move the cost downhill.

Transcript

The hard assumption here is that you've watched part 3, giving an intuitive walkthrough of the backpropagation algorithm. Here we get a little more formal and dive into the relevant calculus. It's normal for this to be at least a little confusing, so the mantra to regularly pause and ponder certainly applies as much here as anywhere else. Our main ... Read More

Key Insights

• Backpropagation is an application of the chain rule that measures how small changes to weights, biases, and activations influence a neural network’s cost. These sensitivities become components of the gradient used to determine parameter adjustments that efficiently decrease the cost.
• The final-layer cost for one training example is the squared difference between the network’s activation and the desired output. Its derivative with respect to that activation is proportional to the output error, so larger discrepancies allow small activation changes to have a larger effect on cost.
• A neuron’s activation is computed by applying a nonlinear function, such as sigmoid or ReLU, to a weighted sum called z. Naming this intermediate value separates the calculation into manageable stages and makes the chain rule relationship between parameters, activation, and cost explicit.
• The derivative of cost with respect to a weight is the product of three local derivatives: cost with respect to activation, activation with respect to z, and z with respect to the weight. Multiplying these sensitivities traces the weight’s influence through the network.
• The derivative of z with respect to its incoming weight is the previous neuron’s activation. Consequently, a weight change has more influence when the neuron feeding into it has a stronger activation, which connects the calculus to the idea that neurons firing together wire together.
• The derivative of z with respect to its bias is 1, so the cost derivative for a bias closely resembles the derivative for a weight. The only substitution is replacing the weight-related local derivative with the bias-related local derivative.
• The sensitivity of z to the preceding activation equals the connecting weight. Tracking this sensitivity allows the same chain rule calculation to continue backward through earlier layers, revealing how earlier weights and biases affect the final cost.
• A previous-layer neuron can influence cost through multiple neurons in the next layer. Its total effect is therefore found by adding the derivative contributions from all relevant paths, after which the same backward process can be repeated for parameters feeding into that layer.

Questions & Answers

Q: How does backpropagation use the chain rule?

Backpropagation uses the chain rule by breaking a parameter’s influence on cost into a sequence of local effects. For a final-layer weight, it multiplies the derivative of cost with respect to activation, the derivative of activation with respect to the weighted sum z, and the derivative of z with respect to the weight. The resulting product measures how a small weight change affects cost.

Q: What does the derivative of cost with respect to a weight mean?

The derivative of cost with respect to a weight measures the ratio between a tiny change in that weight and the resulting change in cost. It therefore describes the cost function’s sensitivity to the weight. Calculating this value for every weight and bias produces the gradient components that indicate which parameter adjustments can most efficiently decrease the network’s cost.

Q: How is the cost for one training example calculated?

For the simple final layer discussed, the cost for one training example is the square of the difference between the final activation and the desired output y. In a layer with multiple output neurons, the cost adds the squared differences between every final-layer activation and its corresponding desired output. The full cost averages the individual costs across many training examples.

Q: Why is the weighted sum called z in backpropagation?

The weighted sum receives the special name z so the stages of a neuron’s computation can be represented separately. A weight multiplied by the previous activation, together with a bias, determines z. A nonlinear function such as sigmoid or ReLU then transforms z into the activation, which helps determine cost. This separation makes the chain rule factors easier to identify and calculate.

Q: What derivatives are needed for a final-layer weight?

Three derivatives are multiplied for a final-layer weight. The derivative of cost with respect to the final activation is 2 times the difference between that activation and the target. The derivative of activation with respect to z is the derivative of the chosen nonlinear function. The derivative of z with respect to the weight equals the preceding neuron’s activation.

Q: How is the derivative for a neural network bias calculated?

The bias derivative follows almost the same chain rule expression as the weight derivative. The relevant difference is the local derivative of z with respect to the parameter. Because the derivative of z with respect to its bias is 1, that factor replaces the derivative of z with respect to the weight. The remaining activation and cost sensitivity factors stay the same.

Q: How does backpropagation move through earlier layers?

Backpropagation first calculates how cost depends on activations near the output. The derivative of a weighted sum z with respect to the preceding activation equals the connecting weight. This provides the sensitivity needed to continue applying the chain rule backward. The process can then be repeated to calculate how cost depends on the earlier weights and biases feeding into that preceding layer.

Q: What changes when neural network layers have multiple neurons?

The basic chain rule calculations remain essentially the same when layers have multiple neurons, but additional indices identify individual neurons and connecting weights. A key difference is that one earlier activation may influence several neurons in the next layer. Its effect on cost must therefore include the sum of contributions from all those paths before backpropagation continues into earlier parameters.

Summary & Key Takeaways

• A simple network with one neuron per layer shows the core calculus behind backpropagation. The final activation depends on a weighted sum passed through a nonlinear function, while the cost is the squared difference between that activation and the desired output. The objective is to measure how parameters influence this cost.

• The derivative of cost with respect to a final-layer weight is decomposed into three local derivatives. These measure how cost responds to activation, how activation responds to the weighted sum, and how that sum responds to the weight. Their product gives the required sensitivity through the chain rule.

• For layers containing multiple neurons, the same equations remain applicable with additional indices for neurons and weights. A previous activation can affect cost through several later neurons, so contributions from every path must be added. Repeating this calculation backward supplies the partial derivatives forming the gradient used to reduce cost.