Fatal Error ClubFatal Error
Roadmap

Week 8 · Neural Networks

Forward propagation + intro to Keras

01

Universal function approximators

Neural networks are universal function approximators. In machine learning, we're usually trying to do one thing: find a function that maps inputs to outputs based on examples we've seen. Neural networks give us a flexible way to represent that function, flexible enough that, in principle, they can approximate almost any mapping we care about. That's the essence of the Universal Approximation Theorem (Cybenko and related work), and it's the reason a handful of core architectures can be adapted to images, text, and tabular data alike: the same underlying principles behind convolutional networks for vision, transformers for LLMs, and simple feedforward networks for tabular prediction, rather than needing a fundamentally different tool for each. With that motivation in place, let's look at the basic building block that makes this possible.

02

The perceptron

A perceptron is a mathematical function that maps a vector of inputs x=[x1,x2,…,xn]Tx = [x_1, x_2, \ldots, x_n]^T to a single scalar output aa.

  • W=[w1,w2,…,wn]W = [w_1, w_2, \ldots, w_n]: the weight matrix. Together with the bias bb, these are the model's learned parameters.
  • σ(z)\sigma(z): the activation function. Turns the weighted sum into the output a=σ(z)a = \sigma(z).
w1w2wnbσax1x2xn
zz

1. Summation form

2. Matrix form

z=z =x1w1x_1 w_1+x2w2x_2 w_2+⋯\cdots+xnwnx_n w_n+bb
z=[w1w2⋯wn][x1x2⋮xn]+bz = \begin{bmatrix} \textcolor{#5ec8c0}{w_1} & \textcolor{#e8c547}{w_2} & \cdots & w_n \end{bmatrix}\begin{bmatrix} \textcolor{#5ec8c0}{x_1} \\ \textcolor{#e8c547}{x_2} \\ \vdots \\ x_n \end{bmatrix} + b
z=∑i=1nwixi+bz = \sum_{i=1}^{n} w_i x_i + b
z=Wx+bz = Wx + b
a=σ(z)a = \sigma(z)
03

Forward propagation

Forward propagation is the process of computing the output aa from the input xx.

Let x=[2, 3]Tx = [2,\ 3]^T, W=[0.5, −1.2]W = [0.5,\ -1.2], b=1b = 1. Use Sigmoid activation.

z=Wx+bz = Wx + b

z=(0.5×2)+(−1.2×3)+1=1−3.6+1=−1.6z = (0.5 \times 2) + (-1.2 \times 3) + 1 = 1 - 3.6 + 1 = -1.6

a=σ(z)=11+e1.6≈11+4.953a = \sigma(z) = \dfrac{1}{1 + e^{1.6}} \approx \dfrac{1}{1 + 4.953}

a≈0.168a \approx 0.168

04

Activation functions

The activation function σ\sigma determines the non-linear behaviour of each neuron. The four most commonly used functions are below. We will understand their use better in the workshop.

1yz
Sigmoidσ(z)=11+e−z\sigma(z) = \dfrac{1}{1 + e^{-z}}
1-1yz
Tanhtanh⁡(z)=ez−e−zez+e−z\tanh(z) = \dfrac{e^{z} - e^{-z}}{e^{z} + e^{-z}}
yz
ReLUReLU(z)=max⁡(0,z)\mathrm{ReLU}(z) = \max(0, z)
yz
Leaky ReLULReLU(z)=max⁡(αz,z)\mathrm{LReLU}(z) = \max(\alpha z, z)
05

Multi-layer perceptron

An MLP is a feedforward network composed of multiple layers of perceptrons (neurons). Each neuron in one layer is connected to every neuron in the next layer (fully connected).

Architecture:

  • Input layer: represents the input features. No computation here.
  • Hidden layer(s): layers between input and output. Apply weights, bias, and activation.
  • Output layer: produces the network's final prediction.
InputHidden layersOutput
x1x_1
x2x_2
a1(1)a_1^{(1)}
a2(1)a_2^{(1)}
a3(1)a_3^{(1)}
a1(2)a_1^{(2)}
a2(2)a_2^{(2)}
a3(2)a_3^{(2)}
a1(3)a_1^{(3)}

Compact form

z(1)=W(1)x+b(1),a(1)=σ(z(1))z^{(1)} = W^{(1)} x + b^{(1)},\quad a^{(1)} = \sigma(z^{(1)})
z(2)=W(2)a(1)+b(2),a(2)=σ(z(2))z^{(2)} = W^{(2)} a^{(1)} + b^{(2)},\quad a^{(2)} = \sigma(z^{(2)})
z(3)=W(3)a(2)+b(3),a(3)=σout(z(3))z^{(3)} = W^{(3)} a^{(2)} + b^{(3)},\quad \textcolor{#38bdf8}{a^{(3)}} = \sigma_{\mathrm{out}}(z^{(3)})
06

Numerical Forward propagation

Forward propagation through the MLP applies z(l)=W(l)a(l−1)+b(l)z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)} and a(l)=σ(z(l))a^{(l)} = \sigma(z^{(l)}) at every layer, with a(0):=xa^{(0)} := x. Here is a full numerical pass on the network above.

Let x=[12]x = \begin{bmatrix} 1 \\ 2 \end{bmatrix}. Hidden layers use ReLU; output uses Sigmoid.

Layer 1 (hidden)

z(1)=W(1)x+b(1),a(1)=ReLU(z(1))z^{(1)} = W^{(1)} x + b^{(1)},\quad a^{(1)} = \mathrm{ReLU}(z^{(1)})
W(1)=[0.5−0.2−1.00.30.10.2],b(1)=[00.50]W^{(1)} = \begin{bmatrix} \textcolor{#5ec8c0}{0.5} & \textcolor{#5ec8c0}{-0.2} \\ \textcolor{#e8c547}{-1.0} & \textcolor{#e8c547}{0.3} \\ \textcolor{#e07a5f}{0.1} & \textcolor{#e07a5f}{0.2} \end{bmatrix},\quad b^{(1)} = \begin{bmatrix} \textcolor{#5ec8c0}{0} \\ \textcolor{#e8c547}{0.5} \\ \textcolor{#e07a5f}{0} \end{bmatrix}
z(1)=[0.5−0.2−1.00.30.10.2][12]+[00.50]=[0.10.10.5]z^{(1)} = \begin{bmatrix} \textcolor{#5ec8c0}{0.5} & \textcolor{#5ec8c0}{-0.2} \\ \textcolor{#e8c547}{-1.0} & \textcolor{#e8c547}{0.3} \\ \textcolor{#e07a5f}{0.1} & \textcolor{#e07a5f}{0.2} \end{bmatrix}\begin{bmatrix} 1 \\ 2 \end{bmatrix} + \begin{bmatrix} \textcolor{#5ec8c0}{0} \\ \textcolor{#e8c547}{0.5} \\ \textcolor{#e07a5f}{0} \end{bmatrix} = \begin{bmatrix} \textcolor{#5ec8c0}{0.1} \\ \textcolor{#e8c547}{0.1} \\ \textcolor{#e07a5f}{0.5} \end{bmatrix}
a(1)=ReLU(z(1))=[0.10.10.5]a^{(1)} = \mathrm{ReLU}(z^{(1)}) = \begin{bmatrix} \textcolor{#5ec8c0}{0.1} \\ \textcolor{#e8c547}{0.1} \\ \textcolor{#e07a5f}{0.5} \end{bmatrix}

Layer 2 (hidden)

z(2)=W(2)a(1)+b(2),a(2)=ReLU(z(2))z^{(2)} = W^{(2)} a^{(1)} + b^{(2)},\quad a^{(2)} = \mathrm{ReLU}(z^{(2)})
W(2)=[0.50.5000.50.50.500.5],b(2)=[000]W^{(2)} = \begin{bmatrix} \textcolor{#7eb8da}{0.5} & \textcolor{#7eb8da}{0.5} & \textcolor{#7eb8da}{0} \\ \textcolor{#c9a0dc}{0} & \textcolor{#c9a0dc}{0.5} & \textcolor{#c9a0dc}{0.5} \\ \textcolor{#f2a65a}{0.5} & \textcolor{#f2a65a}{0} & \textcolor{#f2a65a}{0.5} \end{bmatrix},\quad b^{(2)} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}
z(2)=[0.50.5000.50.50.500.5][0.10.10.5]+[000]=[0.10.30.3]z^{(2)} = \begin{bmatrix} \textcolor{#7eb8da}{0.5} & \textcolor{#7eb8da}{0.5} & \textcolor{#7eb8da}{0} \\ \textcolor{#c9a0dc}{0} & \textcolor{#c9a0dc}{0.5} & \textcolor{#c9a0dc}{0.5} \\ \textcolor{#f2a65a}{0.5} & \textcolor{#f2a65a}{0} & \textcolor{#f2a65a}{0.5} \end{bmatrix}\begin{bmatrix} \textcolor{#5ec8c0}{0.1} \\ \textcolor{#e8c547}{0.1} \\ \textcolor{#e07a5f}{0.5} \end{bmatrix} + \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} = \begin{bmatrix} \textcolor{#7eb8da}{0.1} \\ \textcolor{#c9a0dc}{0.3} \\ \textcolor{#f2a65a}{0.3} \end{bmatrix}
a(2)=ReLU(z(2))=[0.10.30.3]a^{(2)} = \mathrm{ReLU}(z^{(2)}) = \begin{bmatrix} \textcolor{#7eb8da}{0.1} \\ \textcolor{#c9a0dc}{0.3} \\ \textcolor{#f2a65a}{0.3} \end{bmatrix}

Layer 3 (output)

z(3)=W(3)a(2)+b(3),a(3)=σ(z(3))z^{(3)} = W^{(3)} a^{(2)} + b^{(3)},\quad a^{(3)} = \sigma(z^{(3)})
W(3)=[0.8−0.50.2],b(3)=0.1W^{(3)} = \begin{bmatrix} \textcolor{#38bdf8}{0.8} & \textcolor{#38bdf8}{-0.5} & \textcolor{#38bdf8}{0.2} \end{bmatrix},\quad b^{(3)} = 0.1
z(3)=[0.8−0.50.2][0.10.30.3]+0.1=0.09z^{(3)} = \begin{bmatrix} \textcolor{#38bdf8}{0.8} & \textcolor{#38bdf8}{-0.5} & \textcolor{#38bdf8}{0.2} \end{bmatrix}\begin{bmatrix} \textcolor{#7eb8da}{0.1} \\ \textcolor{#c9a0dc}{0.3} \\ \textcolor{#f2a65a}{0.3} \end{bmatrix} + 0.1 = 0.09
a(3)=σ(0.09)=11+e−0.09≈11+0.914a^{(3)} = \sigma(0.09) = \dfrac{1}{1 + e^{-0.09}} \approx \dfrac{1}{1 + 0.914}

a(3)≈0.522a^{(3)} \approx 0.522

07

Before the workshop you should know

A digit is a grid of 28×28 intensities. Flatten it to a column in ℝ⁷⁸⁴. Shape is a contract the rest of the net depends on. That vector is what the network actually sees, one number per pixel.

00
00

∈ℝ⁷⁸⁴

28 × 28

Next week opens Wednesday 25 Nov at 09:00