Universal function approximators
Neural networks are universal function approximators. In machine learning, we're usually trying to do one thing: find a function that maps inputs to outputs based on examples we've seen. Neural networks give us a flexible way to represent that function, flexible enough that, in principle, they can approximate almost any mapping we care about. That's the essence of the Universal Approximation Theorem (Cybenko and related work), and it's the reason a handful of core architectures can be adapted to images, text, and tabular data alike: the same underlying principles behind convolutional networks for vision, transformers for LLMs, and simple feedforward networks for tabular prediction, rather than needing a fundamentally different tool for each. With that motivation in place, let's look at the basic building block that makes this possible.
The perceptron
A perceptron is a mathematical function that maps a vector of inputs to a single scalar output .
- : the weight matrix. Together with the bias , these are the model's learned parameters.
- : the activation function. Turns the weighted sum into the output .
1. Summation form
2. Matrix form
Forward propagation
Forward propagation is the process of computing the output from the input .
Let , , . Use Sigmoid activation.
Activation functions
The activation function determines the non-linear behaviour of each neuron. The four most commonly used functions are below. We will understand their use better in the workshop.
Multi-layer perceptron
An MLP is a feedforward network composed of multiple layers of perceptrons (neurons). Each neuron in one layer is connected to every neuron in the next layer (fully connected).
Architecture:
- Input layer: represents the input features. No computation here.
- Hidden layer(s): layers between input and output. Apply weights, bias, and activation.
- Output layer: produces the network's final prediction.
Compact form
Numerical Forward propagation
Forward propagation through the MLP applies and at every layer, with . Here is a full numerical pass on the network above.
Let . Hidden layers use ReLU; output uses Sigmoid.
Layer 1 (hidden)
Layer 2 (hidden)
Layer 3 (output)
Before the workshop you should know
A digit is a grid of 28×28 intensities. Flatten it to a column in ℝ⁷⁸⁴. Shape is a contract the rest of the net depends on. That vector is what the network actually sees, one number per pixel.
∈ℝ⁷⁸⁴
28 × 28
Next week opens Wednesday 25 Nov at 09:00