Extra resources · Generative modeling
Filling in the Holes
A working derivation of the Variational Autoencoder: why a plain autoencoder's latent space can't be sampled from, the probability theory that fixes it, and the one trick that makes the fix trainable.
01 The problem
Why plain autoencoders aren't enough
An autoencoder pairs two ordinary neural networks around a narrow bottleneck. If you have been through Week 8, you already know how to build both halves: a stack of layers, each one a weighted sum then an activation, trained end to end by the gradient descent from Week 9. The only new idea here is the shape.
The encoder is one of those networks, wired to shrink: it reads an input x (a photo, a sentence, a sound) and, layer by layer, compresses it down to a short vector z, far fewer numbers than x has. The decoder is a second, separate network wired to grow: it reads z and, layer by layer, expands it back out into something the same shape as x, called x̂. Between them sits the bottleneck, the narrowest layer either network has, and everything the decoder will ever know about x has to fit through it.
The encoder and decoder, drawn as networks
Same feedforward stack as Week 8: a weighted sum, then an activation, layer after layer. The only design choice that makes this an autoencoder is the shape: the encoder narrows down to z, the decoder widens back out, and z is the single narrowest layer either network has.
Four terms carry the whole idea. Pin them down before anything else:
Encoder
A neural network with weights φ. Reads x and outputs the latent vector z — the same kind of feedforward stack from Week 8, just shaped to shrink.
Decoder
A second, separate neural network with weights θ. Reads z and outputs a reconstruction x̂, shaped to match x. Mirrors the encoder, expanding instead of shrinking.
Latent vector z
The bottleneck's output. Every fact about x that the decoder gets to use has to survive being squeezed through this one short vector first.
Reconstruction loss
Compares x̂ to x (mean-squared error, typically) and backpropagates through both networks at once: the same gradient descent from Week 9, just with a longer forward pass.
Training never touches zdirectly. It only compares the decoder's output x̂ to the original x with a reconstruction loss, then backpropagates that error through the decoder, through z, and into the encoder: the same chain rule from Week 9, just with two networks stacked end to end instead of one. Every weight in both networks nudges a little, reconstructions get sharper, and z quietly becomes a compact address the decoder has learned to trust.
Compression isn't generation, though. To generate something new you'd want to invent a point in that latent space you've never seen and decode it into something plausible. Try it yourself below: start right on top of a point the encoder actually produced, then slide away from it.
Experiment: nudge a trained z
Still on top of it: this z sits close enough to the point the encoder actually produced during training. The decoder recognizes the neighborhood and reconstructs cleanly.
That failure is structural, not a glitch in this one run. Nothing in the reconstruction loss asks the space between training points to mean anything. The encoder is free to scatter similar inputs however it likes, leaving pockets between them where no training example ever landed. The decoder has never seen those coordinates and has no idea what to do with them. Sample from a gap, and you decode garbage, exactly what just happened above.
Intuition
The fix is almost embarrassingly simple to state: instead of encoding x to one point, encode it to a small neighborhood, a probability distribution. Every input now claims territory, not a single address. If training pushes those neighborhoods to overlap and tile the space, the gaps disappear.
Zoom back out from one nudged point to three whole classes, and toggle between the plain autoencoder and a VAE to see the same gap-versus-no-gap story play out across a full latent space:
Same three classes, one invented z
Autoencoder: each image collapses to one point, so three classes become three tight islands. The crosshair is a z nobody encoded. Decode it and the chip is noise, because the decoder never trained on that coordinate.
That shift (point to neighborhood) is the entire architectural change a VAE makes to the encoder. Everything else in this course is about making that shift mathematically precise and actually trainable.
02 The setup
Setting up the probability problem
To make "encode to a neighborhood" precise, back up and describe generation the way a statistician would.
Assume every observation x was produced by first drawing a hidden, unobserved variable z from some prior distribution, then feeding z through a generative process that produces the visible data. For a face, z might implicitly encode things like pose, lighting, and expression, without ever being told what those factors are called.
If that story is right, the total probability of seeing a particular x is the probability of every path that could have produced it, integrated over every possible z:
Three names are worth fixing before going further:
p(z)
The prior over latent codes. Chosen to be trivial to sample from, almost always a standard normal, N(0, I).
p(x|z)
The likelihood, modeled by the decoder. Given a code, how probable is this observation?
p(z|x)
The posterior. Given an observation, what codes could plausibly have produced it: exactly what encoding needs.
Bayes' rule gives the posterior from the other two:
The problem sits in that denominator. p(x) is the same integral from above, over every possible zin a potentially high-dimensional space, with no closed form for anything but the simplest decoders. It isn't merely annoying to compute. It is intractable.
The workaround
qφ(z|x) (parameterized by a neural network with weights φ) to approximate it as closely as possible. That network isthe encoder. Its job stops being "produce a point" and becomes "produce a distribution close to the true, unreachable posterior."03 The bound
Deriving the ELBO
"As close as possible" needs a way to measure distance between two distributions. The standard tool is the Kullback-Leibler divergence, which is zero exactly when two distributions match, grows as they diverge, and, critically, is never negative.
Write the KL divergence between our approximation and the true posterior:
Substitute Bayes' rule, written as p(z|x) = p(x,z) / p(x), into the ratio, and split the log:
log p(x) doesn't depend on z, so it slides outside the expectation:
Move terms around, and something useful falls out. log p(x)splits exactly into one piece we can compute and optimize, plus one piece we can't compute but know is never negative:
The second term is the same intractable KL we started with, but now it's sitting on the outside as leftover, not something we need to evaluate.
Because that leftover KL term can only be zero or positive, the ELBO (Evidence Lower BOund) is a lower bound on the true log-probability of the data:
Maximizing the ELBO does two things at once: it pushes up a lower bound on the quantity we actually wanted, log p(x), and, since that's the only place the missing slack can go, it simultaneously pushes qφ closer to the true posterior.
The bound, made visible
Drag the slider: as training shrinks the gap, ELBO rises toward the true (fixed) log p(x). It can approach it, but by construction can never pass it.
This bound is the trick that makes intractable posteriors survivable across all of variational inference, not just VAEs: replace an impossible exact computation with a bound you can actually climb.
04 The loss
From bound to loss function
An abstract bound doesn't train a network by itself. It needs to become a loss with clearly separable, computable pieces. Expand the ELBO's joint probability as p(x,z) = p(x|z)·p(z) and regroup, and it factors into exactly two interpretable terms:
Read those as two separate jobs assigned to the network:
Reconstruction term
Eq[log pθ(x|z)] : how well does the decoder reconstruct xfrom a code sampled out of the encoder's distribution? This is the same job a plain autoencoder already does. In practice it's usually mean-squared error for continuous data, or binary cross-entropy when data is normalized to [0,1].Regularization term
DKL(qφ(z|x) ‖ p(z)) : how far is the encoder's distribution for thisinput from the shared prior N(0, I)? Minimizing this pulls every input's neighborhood toward the same well-behaved region: centered, unit-scaled, and smoothly overlapping with its neighbors.Put a minus sign on the whole thing (we minimize losses, and we've been maximizing a bound) and the familiar two-part VAE loss appears:
This is the moment the story closes the loop back to 01. A vanilla autoencoder only ever pays the reconstruction term. Nothing punishes it for scattering codes with gaps between them. The KL term is the entire fix: it's the tax the encoder pays for being disorganized, and paying it is what fills in the holes.
In practice
05 The trick
The reparameterization trick
There's a wrinkle in actually running this loss through backpropagation, and it has nothing to do with the math being wrong. One operation in the forward pass, sampling, simply has no gradient.
Here's the forward pass as literally implemented: the encoder reads x and outputs two vectors, μ and σ. A latent code is then drawn, z ~ N(μ, σ²). That z goes to the decoder, which reconstructs x̂, and the loss is computed.
To update the encoder's weights, gradients need to travel backward from the loss, through the decoder, through z, and into μ and σ. But z came from a random sampling operation. It is not a differentiable function of μ and σ; it's a stochastic node. Ask an autodiff framework for ∂z/∂μthrough a sampling step, and there's nothing coherent to give back.
The computational graph
Direct sampling: z is drawn straight from N(μ,σ²). The sampling step has no gradient, so backprop cannot get past it. The encoder never receives a training signal.
The reparameterization trick sidesteps the problem instead of solving it head-on. Rewrite the same random variable as a deterministic function of μ and σ, plus noise drawn from a fixed, parameter-free distribution:
Nothing about the distribution of z changes. It is still N(μ, σ²). What changes is where the randomness lives. εis sampled from a distribution that never depends on the network's weights, so it sits entirely outside the path backpropagation needs to travel. z becomes an ordinary differentiable function of μ, σ, and a constant (a multiply and an add), and gradients flow through it like through any other layer.
Playground: adjust μ and σ, then sample
Draw a sample to see ε and z.
Every click draws a fresh ε from a fixed N(0,I) and computes z deterministically from the current sliders. The curve and the dots on its axis show the distribution z is really being pulled from. Moving μ shifts it, σ widens or narrows it, and none of that math needs to touch a sampling operation directly.
Why this works
06 Recap
Putting it together
Stack all five pieces and here's the full pipeline, forward pass and loss in one line each:
loss = reconstruction(x, x̂) + DKL( q_φ(z|x) ‖ N(0,I) )
Once training is done, generating something brand new no longer needs the encoder at all. The KL term spent the whole training run pulling every input's little neighborhood toward N(0, I), so sampling z ~ N(0, I)directly and handing it to the decoder is exactly the scenario the model was trained for. That's the entire trick from 01, finally paid off: a latent space with no holes is a latent space you can safely wander into without ever having seen a real input.
A few directions this opens onto, each changing one piece of the picture above without touching the rest:
- β-VAE multiplies the KL term by a coefficient β, trading reconstruction sharpness for a more disentangled, evenly-spread latent space.
- Conditional VAEs feed a label or attribute into both encoder and decoder, so the latent space only has to explain the variation a label doesn't already account for.
- VQ-VAE replaces the continuous Gaussian latent with a discrete codebook, trading the reparameterization trick for a different differentiability workaround, and opening the door to autoregressive latent-space models.
- Diffusion models train a very similar ELBO, but expand the single encoding step into a long chain of small denoising steps.
Test yourself below before moving on to any of them.
Check your understanding
Built as a self-contained course. Equations render with KaTeX. Diagrams stay on this page.