This is part of our lunch study on the foundations of LLMs. This talk explains how derivatives for optimization can be computed efficiently, starting from simple numerical and symbolic approaches and building up to backpropagation.
The notebook compares finite differences, symbolic differentiation, and automatic differentiation, using a small neural network as the running example. It shows how forward-mode automatic differentiation can compute derivatives by carrying derivative information through ordinary calculations, but also why this approach becomes inefficient when there are many parameters.
The central idea is backpropagation: run the calculation forward once, save the intermediate values, and then propagate derivatives backward through the computation graph using the chain rule. This makes it possible to compute all parameter gradients efficiently.
The talk also connects the hand-built implementation to PyTorch, showing how the same ideas appear in requires_grad and .backward(). It closes with a few edge cases and extensions, including nondifferentiable functions such as ReLU and automatic differentiation with complex numbers.
Presented as a Jupyter notebook. There is no Docker container for this session; users should run backpropagation.ipynb directly in Jupyter.