Adaptive Basis Functions: The Missing Link between Linear Fitting and Neural Networks

How do you go from simple least-squares fits to modern feed-forward neural networks? This session gives a mathematician’s view, building intuition for neural networks by tracing the arc from good old linear regression to deep learning—without hand-waving analogies to brains.

We start with familiar ground:

• Linear regression as matrix multiplication, with exact analytic solutions.
• How this extends to multiple dimensions, and how fitting libraries like scikit-learn are just solving extended matrix equations.

What if the relationship isn’t linear?

• If you know the true functional form (e.g., projectiles, exponentials), you can try curve fitting and general minimizers (like Minuit).
• In practice, we often don’t know—so we need approaches that can fit anything.

Enter universal approximators:

• Taylor and Fourier series: arbitrary shapes from sums of fixed basis functions.
• Polynomial fits, Legendre and other orthogonal functions—you can make any smooth curve, but only by summing enough carefully-chosen terms.

The leap:

• Adaptive basis functions are basis functions (like sigmoids) whose shape, width, and position are parameters learned from the data.
• Unlike Taylor series, these functions shift and scale horizontally to where the data needs them.

Here comes neural networks:

• A hidden layer of a neural network is just a collection of these adaptive, parameterized functions.
• One layer = one flexible sum of basis functions with trainable shifts and scales.
• More layers introduce new nonlinearities, letting the network “fold” space and find symmetries, so it can fit complex, high-dimensional patterns efficiently.

Key takeaways:

• Feed-forward neural networks generalize classic function-fitting techniques by letting the basis functions themselves adapt.
• This viewpoint demystifies neural nets: they’re not black boxes, but the next step in a long tradition of function approximation.
• The “universal approximation theorem” is just the statement that enough adaptive basis functions (i.e., wide enough layers) can fit any function.
• No shaky metaphors about actual neurons are required!

Presented as a Jupyter notebook in a publicly accessible JupyterLite website. You don’t need to install anything to run all of the interactive examples; just click on this link!