Unit 20 · From Noise to Pictures: VAEs and Diffusion

Drawing from a hat of noise

Imagine this

Before Ganesh Chaturthi, a sculptor in a small workshop makes one idol after another. Every morning his helper hands him a lump of clay. No two lumps are the same: one is a little bigger, one a little lopsided, one a little wetter.

The sculptor's hands make the same moves every time. Yet every idol comes out slightly different, and every one of them still looks like Ganesha. The skill is fixed. The lump is random. Put the two together and new idols keep coming.

The question. How can a machine draw a face that has never existed?

Here is the whole idea of this unit in one line: to create is to sample. A machine that creates is a machine that takes easy randomness — the lump of clay — and turns it into something that looks like the real data. We call such a machine a generator. The randomness goes in, the sculptor's fixed skill is a function $g$ , and a new example comes out: $x=g(z)$ .

By hand, with heights. Say the heights of grown men in a town follow a bell curve with mean 170 cm and spread 7 cm, written $N(170,\ 7^2)$ . How do we make up a new, realistic height? Draw a standard random number $z$ from $N(0,1)$ — the easiest bell curve there is, centred at 0 with spread 1 — and then stretch and shift it:

$x=\mu+\sigma z=170+7z.$

Four draws of the hat give $z=-1,\ 0,\ 0.5,\ 2$ . The machine turns them into

$\begin{aligned}170+7(-1)&=163\\ 170+7(0)&=170\\ 170+7(0.5)&=173.5\\ 170+7(2)&=184.\end{aligned}$

Four new men who never existed, each with a believable height. The skill ( $\times7$ , then $+170$ ) never changed. Only the lump $z$ did.

Not every shape is a bell. Waiting times at a bus stop, say, follow an exponential curve: short waits are common and long waits are rare. With $\lambda=1$ the chance that the wait is at most $x$ minutes is $F(x)=1-e^{-x}$ . Now take the plainest noise of all, a number $u$ spread evenly between 0 and 1, and solve $u=F(x)$ for $x$ :

$x=-\ln(1-u).$

With $u=0.5$ : $x=-\ln 0.5\approx0.6931$ minutes. This trick is called the inverse CDF (the CDF is the "chance of at most $x$ " curve, $F$ ). It works for any shape of data in one dimension: run the curve $F$ backwards on even noise.

The hard part. For heights and waiting times someone can write the curve down. For pictures, nobody can. A 20 × 20 grey picture is already 400 numbers, and there is no formula for "all the pictures that look like a face". So the generator has to be learned from examples. That is what the whole unit is about: two ways to learn the sculptor's hands — the VAE (Act II) and diffusion (Act III).

Try: On heights, press the four draws of the text: the balls come out at 163, 170, 173.5 and 184. Press draw 500 and watch the green pile take the bell shape of $N(170,\ 7^2)$ . Switch to waiting times and press draw u = 0.5: it comes out at 0.6931. Then try two bumps: plain bell-curve noise in, two humps out — the steep part of the curve splits the pile in two.

Why does this work?

Look at the curve as a funnel. Where it is steep, a narrow band of inputs is stretched over a wide band of outputs, so the balls spread thin. Where it is flat, a wide band of inputs is squeezed into a narrow band of outputs, so the balls pile up. So the curve decides where the pile is high and where it is low: output density = input density ÷ slope. The inverse CDF picks exactly the curve whose slope makes the pile come out right. A learned generator does the same thing in hundreds of dimensions, with a neural network as the curve.

Four ways to learn the sculptor's hands. Every modern generator turns noise into data. They differ in how the noise goes in and how the machine is trained.

family	how it draws	where the noise goes in	good at	weak at
autoregressive (Unit 19)	one piece at a time, each piece chosen from a probability list	a dice roll at every token	text; exact likelihood	slow: one step per piece
VAE (§3–§7)	pick a point on a small map, decode it in one go	the point $z\sim N(0,I)$	fast; a smooth map you can walk	blurry pictures
GAN	one pass of a generator network	a random vector $z$	sharp pictures, fast	unstable training; can forget whole kinds of data
diffusion (§8–§13)	start from static, remove a little noise many times	the starting static, and a little at every step	sharp, varied pictures; easy to steer	many steps (§12 makes it faster)

A GAN in one paragraph. A GAN (a generative adversarial network) trains two networks against each other, like a forger and a detective. The forger turns noise into fake pictures. The detective looks at real pictures and fakes and says "real" or "fake". The forger improves by fooling the detective, the detective improves by catching the forger. It gives sharp pictures, but the contest is hard to balance, and the forger may learn to draw only a few kinds of picture that fool the detective. We will not build one here: the VAE and diffusion are the ones whose maths this course has prepared you for.

Rule of thumb. Text: autoregressive. A fast generator with a smooth map of the data: VAE. The best pictures today: diffusion — often run inside a VAE's map (§13).

Trap

A machine that hands back the training pictures is not creating — it is copying. We want new examples that come from the same distribution as the training data: new heights that are believable, not the heights of the men we measured. A generator is judged by whether its samples look like they came from the same hat, not by whether they match any one example.

The realization

$x=g(z),\qquad z\sim\text{easy noise}$

A generator is a fixed function applied to fresh noise. In one dimension we can write the function down: stretch-and-shift for a bell curve, the inverse CDF for anything else. For pictures the function must be learned — and learning it is the whole art.

Pause & predict

In the heights machine you change the curve from $170+7z$ to $170+14z$ . What happens to the green pile of outputs?

Pause & predict

In the waiting-times machine $x=-\ln(1-u)$ , where do most of the outputs pile up, and why?

If you want the algebra · 2 proofs, step by step

Prove it · running the CDF backwards gives the right shape

Claim. Let $F$ be a CDF that rises steadily (so it has an inverse), and let $u$ be spread evenly on $(0,1)$ . Then $x=F^{-1}(u)$ has exactly the CDF $F$ .

Ask for the chance that the output is at most some value

a

\begin{aligned}&P(x\le a)\\ &=P\big(F^{-1}(u)\le a\big)\\ &=P\big(u\le F(a)\big).\end{aligned}

F

rises steadily, so applying it to both sides keeps the "

\le

" the right way round.

For even noise on

(0,1)

, the chance of landing below any number

c

between 0 and 1 is just

c

. So

P(u\le F(a))=F(a)

— exactly the CDF we wanted. ∎ Exponential with

\lambda=1

F(x)=1-e^{-x}

, so

u=1-e^{-x}

gives

x=-\ln(1-u)

;

u=0.5

gives

\ln2\approx0.6931

Prove it · output density = input density ÷ slope

Claim. If $x=g(z)$ with $g$ increasing, and $z$ has density $p_z$ , then $x$ has density $p_x(x)=p_z(z)\,/\,g'(z)$ at $x=g(z)$ .

A thin band of inputs

[z,\ z+dz]

holds probability

p_z(z)\,dz

. The curve sends it to the band

[x,\ x+g'(z)\,dz]

, which must hold the same probability:

p_x(x)\,g'(z)\,dz=p_z(z)\,dz.

Balls are neither made nor lost on the way through the curve.

Divide by

g'(z)\,dz

p_x(x)=p_z(z)/g'(z)

. Steep curve, thin pile; flat curve, tall pile. ∎ For

x=170+7z

g'=7

, so the output bell is 7 times wider and 7 times lower than

N(0,1)

— that is

N(170,\ 7^2)

. In many dimensions the slope becomes a Jacobian determinant (Unit 4's volume factor).

The road ahead. The unit has four acts.

To create is to sample (§1–§2): the noise machine, and the two facts about Gaussian noise that everything later uses.
Squeeze, then rebuild (§3–§7): the autoencoder, the VAE's cloud, the reparameterisation trick, the ELBO, and walking the map of faces.
Noise in, noise out (§8–§12): diffusion — destroy a picture with noise, learn to guess the noise, the score as a compass, walk back from static, and the fast jump.
Steering, and the whole course (§13–§14): steering with words, latent diffusion, and every tool of the course inside the two machines.

In one sentence: A generator is a sculptor's fixed skill applied to a random lump of clay — $x=g(z)$ with fresh noise $z$ — and since nobody can write $g$ down for pictures, the machine has to learn it.

The Gaussian toolkit for noise

Imagine this

You have a glass of hot chai, filled to the brim. You pour out one tenth of it and top the glass up with milk. Then you do it again, and again. The glass is always exactly full. But after enough rounds, what is in it is almost all milk, and it no longer matters whether you started with strong chai or weak chai.

The question. Diffusion (Act III) will add noise to a picture again and again. How do we add noise so that the numbers never blow up, and so that we always know exactly where they end up?

Two facts about bell curves from Unit 14 do all the work.

Scaling. Multiply a random number by $a$ and its variance is multiplied by $a^2$ : $\operatorname{Var}(aX)=a^2\operatorname{Var}X$ . (The spread, which is the square root, is multiplied by $|a|$ .)
Adding. Add two independent bell curves and you get a bell curve whose variances add: $\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y$ .

One noising step. Take a number $x$ and fresh noise $\varepsilon\sim N(0,1)$ , and make

$x_{\text{new}}=\sqrt{0.9}\,x+\sqrt{0.1}\,\varepsilon.$

This is the chai glass: keep 90% of the variance of the old drink, top up with 10% of new milk. If $x$ itself was a standard bell curve, the two facts give the new variance at once:

$\begin{aligned}&(\sqrt{0.9})^2\cdot1+(\sqrt{0.1})^2\cdot1\\ &=0.9+0.1=1.\end{aligned}$

The glass stays exactly full. Now fix one starting value, $x=2$ . The first part is no longer random: $\sqrt{0.9}\times2\approx1.8974$ . Only the noise part is random, with spread $\sqrt{0.1}\approx0.3162$ . So

$x_{\text{new}}\sim N\big(1.8974,\ 0.3162^2\big).$

In words: shrink the signal a little, add a little noise, and the total size stays the same. Repeat it $n$ times and the signal shrinks to $\sqrt{0.9}^{\,n}x$ while the noise variance grows to $1-0.9^{n}$ . After many steps the starting point is forgotten, and every start ends in the same cloud, $N(0,1)$ . This one step, repeated, is the forward process of diffusion (§8).

Try: Press 1 step: the pile sits at 1.8974 with spread 0.3162. Press 10 steps a few times: the pile slides to 0 and widens to spread 1, and the start is forgotten. Move start to −3 and do it again — same end. Switch to start from a whole N(0, 1) cloud: step as much as you like, the cloud never changes size, because 0.9 + 0.1 = 1.

start2

Why does this work?

Because the two numbers $\sqrt{0.9}$ and $\sqrt{0.1}$ are chosen so that their squares add up to 1. Variance is what adds for independent noise, and a factor inside a variance comes out squared. So the step takes exactly 10% of the variance away from the signal and gives exactly 10% to fresh noise. Nothing grows, nothing shrinks overall; the drink just slowly turns into milk. Any pair $\sqrt{1-\beta}$ and $\sqrt{\beta}$ does the same, which is why diffusion writes every step this way.

Two ways to add noise, side by side.

	just add noise: $x+\sqrt{0.1}\,\varepsilon$	shrink, then add: $\sqrt{0.9}\,x+\sqrt{0.1}\,\varepsilon$
variance after one step (start $N(0,1)$ )	1.1	1
after 100 steps	11 — it keeps growing	still 1
where every start ends up	nowhere fixed: the cloud keeps spreading	the same $N(0,1)$ , whatever the start
used by	score models with growing noise	DDPM (§8) — "variance preserving"

Rule of thumb. Shrink by $\sqrt{1-\beta}$ whenever you add noise of variance $\beta$ . Then the numbers stay the same size forever, and the end of the road is always the same easy cloud.

Trap

Add two independent noises, each with spread 1. The spread of the sum is not 2. Variances add, spreads do not: the variance is $1+1=2$ , so the spread is $\sqrt2\approx1.414$ . Two people pushing a cart in random directions do not push it twice as far — they often partly cancel.

The realization

$\begin{gathered}\operatorname{Var}(aX+bY)\\ =a^2\operatorname{Var}X+b^2\operatorname{Var}Y\\ x\to\sqrt{1-\beta}\,x+\sqrt{\beta}\,\varepsilon\end{gathered}$

For independent noise, variances add and a scale factor comes out squared. So "shrink by $\sqrt{1-\beta}$ , add noise of variance $\beta$ " keeps a standard cloud standard, and slowly washes any starting point into the same $N(0,1)$ .

Pause & predict

Start from the fixed number $x=2$ and take two steps $x\to\sqrt{0.9}\,x+\sqrt{0.1}\,\varepsilon$ , with fresh noise each time. What is the bell curve now?

Pause & predict

You add three independent noises, each with spread 2. What is the spread of the total?

If you want the algebra · 2 proofs, step by step

Prove it · variances of independent noise add

Claim. If $X$ and $Y$ are independent, with means $m_X,m_Y$ , then $\operatorname{Var}(aX+bY)=a^2\operatorname{Var}X+b^2\operatorname{Var}Y$ .

Subtract the mean and call what is left

\tilde X=X-m_X

\tilde Y=Y-m_Y

: then

aX+bY-(am_X+bm_Y)=a\tilde X+b\tilde Y

. Square and average:

\begin{aligned}\operatorname{Var}&=a^2\,\mathbb E[\tilde X^2]\\ &\quad+b^2\,\mathbb E[\tilde Y^2]\\ &\quad+2ab\,\mathbb E[\tilde X\tilde Y].\end{aligned}

(p+q)^2=p^2+q^2+2pq

, then average each part.

For independent

X,Y

the last average splits into

\mathbb E[\tilde X]\,\mathbb E[\tilde Y]=0\cdot0

. What is left is

a^2\operatorname{Var}X+b^2\operatorname{Var}Y

. ∎ With

a=\sqrt{0.9}

b=\sqrt{0.1}

and two standard bells:

0.9+0.1=1

. The sum of two independent bells is again a bell (Unit 14), so knowing the mean and variance is knowing everything.

Prove it · after n steps:

N\big(\sqrt{0.9}^{\,n}x_0,\ 1-0.9^{n}\big)

Claim. Start at a fixed $x_0$ and repeat $x\to\sqrt{0.9}\,x+\sqrt{0.1}\,\varepsilon$ with fresh noise. After $n$ steps, $x_n\sim N\big(\sqrt{0.9}^{\,n}x_0,\ 1-0.9^{n}\big)$ .

True for

n=0

: mean

x_0

, variance

1-1=0

. Suppose it holds for

n

. One more step gives mean

\sqrt{0.9}\cdot\sqrt{0.9}^{\,n}x_0=\sqrt{0.9}^{\,n+1}x_0

. The noise has mean 0, so only the signal carries the mean.

Variance: the old noise is scaled by

\sqrt{0.9}

, the new noise adds 0.1:

\begin{aligned}&0.9\,(1-0.9^{n})+0.1\\ &=1-0.9^{\,n+1}.\end{aligned}

∎ As

n

grows,

0.9^{n}\to0

: mean

\to0

, variance

\to1

. Every start ends in

N(0,1)

. With

n=1

x_0=2

N(1.8974,\ 0.3162^2)

In one sentence: For independent noise variances add and scale factors come out squared, so "shrink by $\sqrt{1-\beta}$ , add noise of variance $\beta$ " keeps the glass exactly full while slowly turning any chai into the same milk, $N(0,1)$ .

The autoencoder: squeeze and rebuild

Imagine this

Your friend is arriving at a crowded railway station, and your cousin has never seen him. You phone your cousin, and the line is bad — you only have time for two numbers. You say: "Face size 7 out of 10. Smile 9 out of 10." Your cousin, who is good at sketching, draws a face from those two numbers and goes looking for the man who matches.

You squeezed a whole face into two numbers. Your cousin rebuilt a face from them. If the sketch looks like your friend, the two numbers held what mattered.

The question. Can a machine learn, by itself, which few numbers describe a picture — and learn to draw the picture back from them?

An autoencoder is two networks back to back. The encoder (you, on the phone) reads a picture $x$ and squeezes it into a short list of numbers, the code $z$ . The decoder (your cousin) reads the code and rebuilds a picture $\hat x$ . Training asks only one thing: make the rebuild match the picture. The loss is the squared error over every pixel, summed over all the training pictures:

$\begin{gathered}L=\sum_{\text{pictures}}\big\|x-\hat x\big\|^2,\\ \hat x=\operatorname{dec}\big(\operatorname{enc}(x)\big).\end{gathered}$

The toy on this page. We made up 4 000 small grey faces, each 20 × 20 = 400 pixels. Two things vary: how big the face is, and how much it smiles (from a frown to a big smile). We trained a small autoencoder on them for this page, with a code of just 2 numbers — so the whole map of codes fits on your screen. 400 numbers in, 2 in the middle, 400 out: a squeeze of 200 times. Nobody told the network about "size" or "smile". The squeeze forced it to find them, because two numbers are only enough if they are the two that matter.

You have seen this before. In Unit 12, PCA squeezed data onto the few directions with the most variation. Make the encoder and decoder plain matrices (no curves at all) and train with squared error: the best such autoencoder squeezes onto exactly the same subspace as PCA. A curved (nonlinear) autoencoder can do more: it can bend its map to follow data that lies on a curved surface.

So can we create? Here is the tempting idea: throw away the encoder, pick a random code, and let the decoder draw. Try it below. On the plain autoencoder it mostly fails. The codes of real faces landed in a lopsided patch, stretched far to one side of 0, with empty regions around it. Nothing in training ever asked the decoder what to draw for a code from an empty region, so it draws whatever it likes — often something that is not a face at all.

Try: On the plain autoencoder, click inside the pink patch: a clean face. Click in the empty area near 0 — right inside the rings: a smudge or an inverted blob, marked not a face. Press 200 random codes from N(0, I): about 65% are not faces. Switch to the VAE and press it again: about 4%. The VAE's map is centred and full.

Why does this work?

Because the squeeze leaves no room for anything else. Two numbers cannot hold 400 pixels, so the network must spend them on whatever changes most from face to face — here, size and smile. Every face that differs only in the small details gets the same code and the same rebuild. That is PCA's idea (Unit 12) done with curves: keep the few directions along which the data really varies, and throw the rest away. And it is also why the empty regions are dangerous: the loss only ever looked at the codes of real faces, so every other point of the map is untested.

Three ways to squeeze, side by side.

	PCA (Unit 12)	linear autoencoder	nonlinear autoencoder
squeeze	project onto the top eigenvectors	a matrix	a network with curves
rebuild	the same eigenvectors, back	another matrix	another network
what it finds	the best flat subspace	the same subspace (maybe with tilted axes)	a curved surface that follows the data
trained by	an eigen-decomposition, in one go	gradient descent	gradient descent
a random code gives	a blend of eigen-pictures	a blend of eigen-pictures	anything — often garbage outside the patch

Rule of thumb. If the data lies near a flat subspace, PCA is enough and needs no training. If it lies on a curved surface, use an autoencoder. If you want to create, a plain autoencoder is not enough: you need a map with no empty regions — the VAE of §4.

Trap

A good rebuild does not make a good generator. The plain autoencoder rebuilds its training faces very well, yet most random codes decode to rubbish. Rebuilding asks "is the decoder right at the codes of real pictures?" Creating asks "is the decoder right everywhere we might pick a code?" Those are different questions.

The realization

$\min_{\text{enc},\ \text{dec}}\sum_{x}\big\|x-\operatorname{dec}(\operatorname{enc}(x))\big\|^2$

An autoencoder squeezes each picture into a short code and learns to rebuild it. The squeeze forces the code to keep only what varies most — with straight lines, it is PCA. But it makes no promise about the empty parts of the map, so it is a compressor, not yet a creator.

Pause & predict

The plain autoencoder's codes lie roughly between −10 and 0 across and between −1 and 4 up. You pick 1 000 random codes from $N(0,I)$ , centred at the origin. What happens?

Pause & predict

You train a linear autoencoder (matrices only, no curves) with a 2-number code on some data, using squared error. PCA on the same data gives a top-2 plane. What does the autoencoder's decoder span?

If you want the algebra · 1 proof, step by step

Prove it · a linear autoencoder finds PCA's subspace

Claim. Take centred data (the rows of $X$ ), a linear encoder $z=E\,x$ and a linear decoder $\hat x=D\,z$ with a $k$ -number code. The smallest possible total squared error $\sum\|x-DEx\|^2$ is reached when $DE$ is the projection onto the top- $k$ principal directions — the same subspace PCA keeps.

Whatever

E

and

D

are,

DE

is a matrix of rank at most

k

: every rebuild

DEx

lies in the column space of

D

, a

k

-dimensional subspace

S

. A

k

-number code can only reach

k

directions.

For a fixed subspace

S

, the closest point of

S

to each

x

is its orthogonal projection

Px

(Unit 3). So the best any linear autoencoder can do with that

S

\sum\|x-Px\|^2

, and it achieves it by choosing

E

to compute the projection's coordinates. Any other point of

S

is farther away, by Pythagoras.

Now choose

S

. The total error is the total spread minus the spread kept in

S

, and Unit 12 showed the spread kept is largest for the top-

k

eigenvectors of the covariance. So the best

S

is PCA's subspace. ∎ The columns of

D

need not be the eigenvectors themselves: any basis of the same plane gives the same rebuilds. Gradient descent usually lands on a tilted, stretched basis of it.

In one sentence: An autoencoder squeezes a picture into a few numbers and learns to rebuild it — describing a friend's face in two numbers over a bad phone line — but its map of codes has empty regions, so a random code usually draws rubbish.

The VAE: a cloud, not a point

Imagine this

You tell a friend where to meet you. If you give the exact house number — "14/B, Lane 3" — and he lands on the house next door, he is lost. If you say "near the temple in Rajpur" — a neighbourhood, not a house — then anywhere in that neighbourhood is a fine answer, and he will find you.

Now imagine every friend gives a neighbourhood instead of a house number, and the town asks everyone to keep their neighbourhoods near the town centre. The neighbourhoods overlap, and there is no empty land left in town.

The question. How do we stop the map of codes from having empty regions, so that every point decodes to something sensible?

The variational autoencoder, or VAE (Kingma and Welling, 2014), makes two changes.

A cloud, not a point. The encoder no longer outputs one code. It outputs a centre $\mu$ and a spread $\sigma$ — a small bell-shaped cloud $N(\mu,\sigma^2)$ on the map. During training we pick a random point $z$ inside the cloud, and the decoder must rebuild the picture from that point. So the decoder learns to draw well over the whole cloud, not just at its centre. A neighbourhood, not a house number.
A pull towards the centre. A second term in the loss pulls every cloud towards the standard cloud $N(0,1)$ : centre 0, spread 1. It is the KL divergence of Unit 14, which measures how far one bell curve is from another. For one number of the code,

$\begin{aligned}&\operatorname{KL}\big(N(\mu,\sigma^2)\,\|\,N(0,1)\big)\\ &=\tfrac12\big(\mu^2+\sigma^2-1-\ln\sigma^2\big).\end{aligned}$

Read it aloud. $\mu^2$ : you pay for sitting far from the centre. $\sigma^2-1-\ln\sigma^2$ : you pay for a spread that is not 1 — too wide or too narrow. Four clouds, by hand:

$\begin{aligned}&\mu=1,\ \sigma=0.5:\\ &\quad\tfrac12(1+0.25-1-\ln0.25)\approx0.8181\\ &\mu=0,\ \sigma=1:\ \ \tfrac12(0+1-1-0)=0\\ &\mu=2,\ \sigma=1:\ \ \tfrac12(4+1-1-0)=2\\ &\mu=0,\ \sigma=0.1:\\ &\quad\tfrac12(0+0.01-1-\ln0.01)\approx1.8076\end{aligned}$

The standard cloud itself costs nothing. Moving it away costs $\mu^2/2$ . And look at the last one: a cloud sitting right at the centre but too sure of itself (spread 0.1) costs more than the one at $\mu=1$ . The KL does not only pull clouds in; it also keeps them from shrinking back into points.

The full loss for one picture adds the two jobs, with a weight $\beta$ on the pull ( $\beta=1$ is the plain VAE):

$\begin{aligned}\text{loss}=\ &\underbrace{\text{rebuild error from a point in the cloud}}_{\text{be useful}}\\ &+\beta\,\underbrace{\operatorname{KL}\big(\text{cloud}\,\|\,N(0,I)\big)}_{\text{stay near the centre}}.\end{aligned}$

(For a code of two numbers, the KLs of the two entries simply add up.) Our toy VAEs were trained with three weights, $\beta=0.1,\ 1,\ 30$ , on the same faces.

Try: Switch between the three weights. At β = 0.1 the clouds are tiny and float high (KL about 8 each), and their shadows leave empty land between them. At β = 1 they are bigger, lower, and their shadows cover the rings. At β = 30 they sink to the floor, stretch into long bars across the whole map, and the rebuilds below all get the same blurred mouth: the smile number has stopped carrying anything (its KL is 0). Click a face below to light up its cloud and read its KL by the formula.

drag the picture to orbit

12 training faces · top: the face · bottom: rebuilt from the centre of its cloud

Why does this work?

Two forces meet. The rebuild term wants each cloud small and apart, so that the decoder can tell the faces apart. The KL term wants every cloud to be the same wide cloud at the centre. The balance is a map where clouds of similar faces overlap. Wherever two clouds overlap, the decoder was trained to draw both faces from nearby points, so it learns to draw something in between — a sensible face. Overlap everywhere means no empty land: every point of the rings has been trained on.

The weight on the pull, side by side. These are the three toy VAEs of the widget.

	$\beta=0.1$ (weak pull)	$\beta=1$ (the plain VAE)	$\beta=30$ (strong pull)
clouds	tiny, almost points	small, overlapping	wide, all at the centre
average KL per face	about 8.0	about 5.0	about 1.2 (the smile number: 0)
rebuilds	sharp	sharp	right size, but every mouth the same blur
random codes	many land in holes	almost all good faces	faces, but no smiles or frowns

Rule of thumb. Too little pull and you have a plain autoencoder with holes. Too much and the code stops carrying information — "posterior collapse". Start at $\beta=1$ and watch the KL of each code number: a number whose KL falls to 0 has stopped being used.

Trap

The KL term is not a nuisance to push to zero. A KL of exactly 0 means the cloud is the standard cloud, whatever the picture — so the code tells the decoder nothing about the picture. At $\beta=30$ this happened to the smile number: its KL is 0, and every rebuilt mouth is the same average blur. A useful code must pay some KL.

The realization

$\begin{aligned}\text{loss}=\ &\mathbb E\,\big\|x-\operatorname{dec}(z)\big\|^2\\ &+\beta\,\operatorname{KL}\big(\text{cloud}\,\|\,N(0,I)\big)\end{aligned}$

A VAE encodes each picture as a cloud, rebuilds from a random point $z$ in it, and pays a KL price — $\tfrac12(\mu^2+\sigma^2-1-\ln\sigma^2)$ for each number of the code — for every cloud that is far from the centre or too narrow. The clouds of similar pictures overlap, the map fills up, and every point decodes to something sensible.

Pause & predict

Two clouds for one code number: A has $\mu=0$ , $\sigma=0.5$ ; B has $\mu=0.5$ , $\sigma=1$ . Which pays the bigger KL?

Pause & predict

You raise the KL weight $\beta$ from 1 to 30. What happens to the clouds and to the rebuilds?

Pause & predict

A code has two numbers. Its cloud has $\mu=(2,0)$ and $\sigma=(1,1)$ . What is its total KL to $N(0,I)$ ?

If you want the algebra · 1 proof, step by step

Prove it · the KL of a bell curve to the standard one

Claim. $\operatorname{KL}\big(N(\mu,\sigma^2)\,\|\,N(0,1)\big)=\tfrac12\big(\mu^2+\sigma^2-1-\ln\sigma^2\big)$ , and it is 0 only at $\mu=0,\ \sigma=1$ .

KL is the average, under

q=N(\mu,\sigma^2)

, of

\ln q(z)-\ln p(z)

(Unit 14). The two logs of bell curves are

\begin{aligned}\ln q&=-\tfrac12\ln(2\pi\sigma^2)\\ &\quad-\tfrac{(z-\mu)^2}{2\sigma^2}\\ \ln p&=-\tfrac12\ln(2\pi)-\tfrac{z^2}{2}\end{aligned}

The

\ln2\pi

parts cancel in the difference.

Average over

z\sim q

\mathbb E[(z-\mu)^2]=\sigma^2

and

\mathbb E[z^2]=\mu^2+\sigma^2

. So

\begin{aligned}\operatorname{KL}&=-\tfrac12\ln\sigma^2-\tfrac12\\ &\quad+\tfrac12(\mu^2+\sigma^2)\\ &=\tfrac12(\mu^2+\sigma^2\\ &\qquad-1-\ln\sigma^2).\end{aligned}

\mathbb E[z^2]

is the variance plus the mean squared.

\mu^2\ge0

is smallest at

\mu=0

. And

s-1-\ln s

with

s=\sigma^2

has slope

1-1/s

, zero only at

s=1

, where it is

0

; it is positive everywhere else. So the KL is 0 exactly for the standard cloud. ∎

\mu=0,\ \sigma=0.1

\tfrac12(0.01-1+4.6052)\approx1.8076

. A too-narrow cloud pays through

-\ln\sigma^2

In one sentence: A VAE gives every picture a neighbourhood, not a house number — a small cloud on the map — and a KL price pulls every cloud towards the town centre, so the clouds overlap and every point of the map has been trained to draw a sensible picture.

The reparameterisation trick

Imagine this

A darts player's hand always wobbles a little. If the wobble happens by surprise, you cannot say what would have happened had he aimed a little to the left. But suppose the wobble is decided in advance — written on a card before the throw: "2 cm right, 1 cm up". Now the dart lands at aim + wobble. Move the aim 1 cm left, and the dart moves exactly 1 cm left. You can ask how the aim moves the dart.

The question. Training a VAE means backpropagation (Unit 7): blame flows from the loss back to every weight. But in the middle of the network we draw a random point from the cloud. How does blame flow backwards through a dice roll?

It cannot, as long as the dice roll is inside the network. "Draw $z$ from $N(\mu,\sigma^2)$ " is not a formula in $\mu$ and $\sigma$ : nudge $\mu$ and the draw simply comes out different, with no slope to follow.

The trick. Roll the dice first, outside the network, as the card with the wobble: $\varepsilon\sim N(0,1)$ . Then build the draw by the stretch-and-shift of §1:

$z=\mu+\sigma\,\varepsilon.$

The randomness has moved into an input. And $z$ is now an ordinary formula in $\mu$ and $\sigma$ , so it has slopes.

By hand. Take $\mu=1$ , $\sigma=0.5$ , and the card says $\varepsilon=0.4$ :

$\begin{aligned}z&=1+0.5\times0.4=1.2\\ \frac{\partial z}{\partial\mu}&=1,\qquad \frac{\partial z}{\partial\sigma}=\varepsilon=0.4.\end{aligned}$

Now the chain rule of Unit 7 does the rest. Say the rest of the network (a stand-in here) gives the loss $L=(z-2)^2$ : the decoder would like $z$ to be 2. Then $\partial L/\partial z=2(1.2-2)=-1.6$ , and the blame splits:

$\begin{aligned}\frac{\partial L}{\partial\mu}&=-1.6\times1=-1.6\\ \frac{\partial L}{\partial\sigma}&=-1.6\times0.4=-0.64.\end{aligned}$

Both slopes are negative, so gradient descent raises $\mu$ (and, with this $\varepsilon$ , $\sigma$ a little) to move the dart towards 2. Blame flows straight through.

Try: In dice inside the network, press ▶ backward pass: the blame stops at the dice, broken — μ and σ get nothing. Switch to reparameterised and press it again: the blame reaches μ as −1.6 and σ as −0.64. Drag ε to −0.4: z = 0.8, and σ's slope becomes +0.96 — the other way, because this wobble points left. The histogram on the right always matches $N(\mu,\sigma^2)$ : the trick changes how we draw, not what we draw.

μ1

σ0.5

ε0.4

Why does this work?

Because $\mu+\sigma\varepsilon$ with $\varepsilon\sim N(0,1)$ is exactly a draw from $N(\mu,\sigma^2)$ — the stretch-and-shift of §1. So nothing about the model changes; only the bookkeeping does. The randomness is now a fixed input for each training step, like a picture, and everything that depends on $\mu$ and $\sigma$ is a smooth formula. Average the slopes over many wobbles and you get the true slope of the average loss.

Two ways to draw, side by side.

	dice inside: draw $z\sim N(\mu,\sigma^2)$	reparameterised: $z=\mu+\sigma\varepsilon$
what comes out	a draw from $N(\mu,\sigma^2)$	a draw from $N(\mu,\sigma^2)$ — the same
where the randomness lives	in the middle of the network	in an input $\varepsilon$ , drawn first
$\partial z/\partial\mu$ , $\partial z/\partial\sigma$	not defined	1 and $\varepsilon$
training	needs slow, noisy tricks	ordinary backprop

Rule of thumb. Whenever a network must sample in the middle, write the sample as "parameters + a formula of outside noise". Diffusion does the same thing in §8: $x_t=\sqrt{\bar\alpha}\,x_0+\sqrt{1-\bar\alpha}\,\varepsilon$ .

Trap

The wobble $\varepsilon$ is not learned and not optimised. It is fresh noise for every picture at every step, and it gets no slope of its own. What learns are $\mu$ and $\sigma$ — and, through them, the encoder's weights.

The realization

$\begin{gathered}z=\mu+\sigma\,\varepsilon,\quad\varepsilon\sim N(0,1)\\ \Rightarrow\ \frac{\partial z}{\partial\mu}=1,\ \ \frac{\partial z}{\partial\sigma}=\varepsilon\end{gathered}$

Roll the dice before the network, not inside it. The draw is the same, but now it is a formula in $\mu$ and $\sigma$ , and the chain rule carries the blame straight back to the encoder.

Pause & predict

With $\mu=1$ , $\sigma=0.5$ , the card says $\varepsilon=-2$ . What are $z$ and $\partial z/\partial\sigma$ ?

Pause & predict

You switch from "dice inside" to the reparameterised form, and draw 10 000 values of $z$ each way. How do the two histograms compare?

If you want the algebra · 1 proof, step by step

Prove it · the slope of an average, through the trick

Claim. For a loss $f$ and $z=\mu+\sigma\varepsilon$ , $\varepsilon\sim N(0,1)$ : $\ \dfrac{\partial}{\partial\mu}\mathbb E\big[f(z)\big]=\mathbb E\big[f'(z)\big]$ and $\dfrac{\partial}{\partial\sigma}\mathbb E\big[f(z)\big]=\mathbb E\big[f'(z)\,\varepsilon\big]$ .

Write the average over the fixed noise:

\mathbb E_\varepsilon\big[f(\mu+\sigma\varepsilon)\big]

. The distribution of

\varepsilon

does not depend on

\mu

\sigma

, so the slope can go inside the average. Before the trick, the distribution itself moved with

\mu

, and there was nothing to differentiate inside.

Inside, the chain rule (Unit 7):

\partial f/\partial\mu=f'(z)\cdot1

\partial f/\partial\sigma=f'(z)\cdot\varepsilon

. ∎ One draw gives one noisy estimate of each slope; a batch of pictures averages them. With

f=(z-2)^2

\mu=1,\ \sigma=0.5,\ \varepsilon=0.4

f'=-1.6

, slopes

-1.6

and

-0.64

In one sentence: Write the random draw as $z=\mu+\sigma\varepsilon$ with the wobble $\varepsilon$ rolled in advance — a dartboard where the wobble is written on a card — and the chain rule can carry blame through the sampling step straight back to the encoder.

The ELBO: the score a VAE maximises

Imagine this

You want to know how high the ceiling of a hall is, but you cannot reach it. What you can measure is a platform you are standing on. The platform is always below the ceiling. Push the platform up, and one of two things must be happening: the ceiling is rising, or the gap between you and the ceiling is closing. Either way, you are doing something useful.

The question. A good generator gives real pictures a high probability $p(x)$ . But for a VAE, $p(x)$ means "add up, over every code, the chance of that code times the chance it draws $x$ " — an impossible sum over the whole map. What can we compute and maximise instead?

The answer is the ELBO, the evidence lower bound. (The "evidence" is $\log p(x)$ .) For any guess $q(z)$ the encoder makes about which codes could have drawn $x$ :

$\begin{aligned}\log p(x)=\ &\underbrace{\mathbb E_{q}\big[\log p(x\mid z)\big]}_{\text{rebuild well}}\\ &-\underbrace{\operatorname{KL}\big(q\,\|\,p(z)\big)}_{\text{stay near the prior}}\\ &+\underbrace{\operatorname{KL}\big(q\,\|\,p(z\mid x)\big)}_{\text{the gap}\ \ge\ 0}.\end{aligned}$

The first two lines together are the ELBO. The ELBO is the platform, $\log p(x)$ is the ceiling, and the gap is how wrong the encoder's guess is about the true codes behind $x$ . Its two parts are exactly the VAE loss of §4, with the sign flipped: rebuild well, and pay KL to the prior.

A whole example, by hand. Take the smallest possible model: just two codes, A and B, each chosen with chance $p(z)=0.5$ . Code A draws our picture $x$ with chance $p(x\mid A)=0.6$ , code B with chance $p(x\mid B)=0.2$ .

$\begin{aligned}p(x)&=0.5(0.6)+0.5(0.2)=0.4\\ \log p(x)&=\ln0.4\approx-0.9163\\ p(A\mid x)&=\frac{0.3}{0.4}=0.75\\ p(B\mid x)&=0.25.\end{aligned}$

That last line is the true posterior: given the picture, A is three times as likely as B. Now let the encoder guess $q=(q_A,\ q_B)$ , and compute the ELBO's parts: rebuild $=q_A\ln0.6+q_B\ln0.2$ , KL $=q_A\ln\frac{q_A}{0.5}+q_B\ln\frac{q_B}{0.5}$ .

the encoder's guess $q$	rebuild	KL to prior	ELBO	gap
(0.5, 0.5) — no idea	−1.0601	0	−1.0601	0.1438
(0.9, 0.1) — too sure	−0.6207	0.3681	−0.9888	0.0725
(0.75, 0.25) — the truth	−0.7855	0.1308	−0.9163	0

Every row adds up: ELBO + gap $=-0.9163$ , always. The "too sure" guess rebuilds best of all, but pays so much KL that its ELBO is lower than the truth's. The ELBO is highest exactly when the guess is the true posterior — and there it touches the ceiling.

Try: Slide q(A) from 0.5 to 0.9, then to 0.75: the gap shrinks to 0 and the platform touches the ceiling at −0.9163. Then make the model better — slide p(x | A) from 0.6 to 0.8: the ceiling itself rises to ln 0.5 ≈ −0.6931, and the best guess moves to 0.8. Pushing the ELBO up does both jobs.

q(A)0.5

p(x | A)0.6

Why does this work?

Because the gap is a KL, and a KL is never negative (Unit 14). So the ELBO is always a floor under $\log p(x)$ . Training pushes the floor up in two ways at once. Improving the decoder raises the ceiling itself — real pictures become more likely. Improving the encoder closes the gap — its guess gets closer to the true codes. We never compute the ceiling, and we never need to.

Two ways to read the same score.

	ELBO = rebuild − KL to prior	ELBO = $\log p(x)$ − gap
what you can compute	both parts, for any picture	neither part (both need the whole map)
what it tells you	the VAE loss, turned upside down	why the loss is a sensible thing to optimise
best guess $q$	balance rebuilding against the pull	the true posterior (gap 0)

Rule of thumb. Train with the left-hand form; understand with the right-hand one. The left form tells you what to compute; the right form tells you why it works — every step that raises it either makes the model better or makes the encoder's guess more honest.

Trap

Maximising the ELBO is not only "rebuild better". It does two jobs at once: it improves the decoder (a higher ceiling) and makes the encoder's guess closer to the truth (a smaller gap). In the table, the guess that rebuilds best, (0.9, 0.1), is not the one with the best ELBO.

The realization

$\begin{aligned}\log p(x)&=\mathbb E_q[\log p(x\mid z)]\\ &\quad-\operatorname{KL}(q\,\|\,p(z))\\ &\quad+\operatorname{KL}\big(q\,\|\,p(z\mid x)\big)\\ &\ge\ \text{ELBO}\end{aligned}$

We cannot compute how likely the model makes a picture, but we can compute a floor under it: rebuild well, stay near the prior. Raising the floor either raises the ceiling or closes the gap — and at the true posterior the floor touches the ceiling.

Pause & predict

In the two-code toy, the encoder guesses $q=(1,0)$ : "certainly A". What happens to the ELBO compared with the truth $(0.75,\ 0.25)$ ?

Pause & predict

You make the decoder better: $p(x\mid A)$ goes from 0.6 to 0.8 (with $p(x\mid B)=0.2$ as before). Where does the best guess $q_A$ move, and what is the new ceiling?

Pause & predict

For some guess $q$ , the rebuild part is −1.2 and the KL to the prior is 0.3, and you know $\log p(x)=-1.1$ . How big is the gap?

If you want the algebra · 2 proofs, step by step

Prove it · log p(x) = ELBO + KL(q ‖ true posterior)

Claim. For any guess $q(z)$ : $\log p(x)=\mathbb E_q\big[\log p(x\mid z)\big]-\operatorname{KL}\big(q\,\|\,p(z)\big)+\operatorname{KL}\big(q\,\|\,p(z\mid x)\big)$ .

Bayes (Unit 14):

p(z\mid x)=p(x\mid z)\,p(z)/p(x)

. Take logs and rearrange:

\begin{aligned}\log p(x)&=\log p(x\mid z)\\ &\quad+\log p(z)\\ &\quad-\log p(z\mid x).\end{aligned}

This holds for every single

z

Add and subtract

\log q(z)

, then average over

z\sim q

(the left side does not depend on

z

\begin{aligned}\log p(x)&=\mathbb E_q\log p(x\mid z)\\ &\quad-\mathbb E_q\log\tfrac{q}{p(z)}\\ &\quad+\mathbb E_q\log\tfrac{q}{p(z\mid x)}.\end{aligned}

The two averages of log-ratios are exactly the two KLs.

A KL is never negative, so

\log p(x)\ge

ELBO, with equality exactly when

q=p(z\mid x)

. ∎ Two-code toy,

q=(0.9,0.1)

: ELBO

-0.9888

, gap

0.9\ln\frac{0.9}{0.75}+0.1\ln\frac{0.1}{0.25}\approx0.0725

, total

-0.9163

Prove it · the ELBO's rebuild term is the VAE's squared error

Claim. If the decoder says "the picture is $\operatorname{dec}(z)$ plus bell-shaped pixel noise of spread $s$ ", then $\log p(x\mid z)=-\|x-\operatorname{dec}(z)\|^2/(2s^2)+\text{const}$ — so maximising the ELBO is minimising the VAE loss of §4.

Each of the

n

pixels is

N(\operatorname{dec}(z)_i,\ s^2)

, independently. The log of their joint density is

\begin{aligned}&-\sum_i\tfrac12\ln(2\pi s^2)\\ &-\sum_i\tfrac{(x_i-\operatorname{dec}(z)_i)^2}{2s^2}.\end{aligned}

The same step that turned likelihood into least squares in Unit 14.

The first part does not depend on the model, so it drops out of any comparison. What is left is

-\|x-\operatorname{dec}(z)\|^2/(2s^2)

: the squared error, with weight

1/(2s^2)

. Flip the sign of the ELBO and you have "squared error + KL". ∎ Our toy used

s=0.25

, so the rebuild weight is

1/(2\times0.0625)=8

per unit of squared error.

\beta

then weighs the KL against it.

In one sentence: The ELBO — rebuild well, stay near the prior — is a platform that always sits under the ceiling $\log p(x)$ ; pushing it up either raises the ceiling or closes the gap, and at the true posterior the two touch.

Walking the latent map

Imagine this

Picture a huge map of a country where every village is a face. Villages next to each other hold faces that look alike. Walk north and the faces grow bigger; walk west and they start to smile. You can stop anywhere — even in a field between villages — and a face is waiting for you there too.

The question. Now that the map is full, how do we use it to make new faces — and what are its limits?

The VAE's map is called its latent space ("latent" means hidden: the code is never seen in the data). Three things become easy.

Sampling. Draw $z\sim N(0,I)$ — the same cloud the KL pulled everything towards — and decode. Each draw is a new face. This is the sculptor of §1: fixed hands (the decoder), a random lump ( $z$ ).
Morphing. Take the codes of two faces and walk in a straight line between them, decoding as you go: one face turns smoothly into the other.
Directions mean something. Because similar faces sit near each other, a direction on the map becomes a property: "this way is bigger", "that way is happier". In big VAEs such directions are found for hair, glasses or age.

The honest limit. VAE pictures are blurry. The rebuild loss is a squared error, and when the decoder is unsure between two possible pictures — a mouth slightly up or slightly down — the squared error is smallest for their average. An average of two sharp mouths is one soft mouth. So wherever the code does not pin a detail down, the VAE paints a blur. Keep this in mind: it is the gap that diffusion (Act III) closes.

Try: Drag the cursor upwards: the faces grow. Drag it from left to right: a big smile turns into a frown. Drag it to the far left edge, outside both rings: the decoder was hardly trained there, and it paints inverted blobs. Press draw 12 random for twelve new faces from $N(0,I)$ . Press morph A → B: the strip walks from a small smiling face to a big frowning one. Compare its middle frame with the pixel average of A and B — two faces laid on top of each other — which is not a face at all.

Why does this work?

Because of the overlapping clouds of §4. During training, every point near the centre was drawn from some face's cloud, and the decoder was asked to rebuild that face from it. Points between two clouds were trained on both faces, so the decoder learned to paint something between them. That is exactly what a smooth morph and a good random sample need: a decoder that was trained everywhere the prior puts its draws.

Two kinds of "in between", side by side.

	pixel average $\tfrac12(x_A+x_B)$	decoded midpoint $\operatorname{dec}\big(\tfrac12(z_A+z_B)\big)$
what it does	lays the two pictures on top of each other	walks halfway on the map, then draws
what you see	two faint outlines, two mouths — a double exposure	one face, middle-sized, with a middle smile
is it a face?	no	yes — the map only holds faces

Rule of thumb. To blend two things, blend their codes, not their pixels. Straight lines in the latent space become curved, sensible paths among real-looking pictures.

Trap

A straight line on the map is not a straight line in pixel space. The pixel-space straight line from A to B passes through double exposures that are not faces at all. The decoder bends the straight latent line into a curve that stays among faces. "Halfway" on the map means "halfway in meaning", not "the average of the pixels".

The realization

$\begin{gathered}x_{\text{new}}=\operatorname{dec}(z),\ \ z\sim N(0,I)\\ x(t)=\operatorname{dec}\big((1-t)\,z_A+t\,z_B\big)\end{gathered}$

A trained VAE is a map where every point is a picture: draw a point from the prior to create, walk a straight line to morph. Its weakness is blur — the squared-error rebuild paints the average wherever the code leaves a detail open.

Pause & predict

You decode codes far outside the rings, at $z=(6,6)$ . What do you expect?

Pause & predict

A VAE decoder is unsure whether a mouth pixel should be 0 (dark) or 1 (bright), each equally likely. Under squared error, what value does it paint?

If you want the algebra · 1 proof, step by step

Prove it · squared error paints the average (where VAE blur comes from)

Claim. If the true value of a pixel is random, the single guess $c$ with the smallest average squared error $\mathbb E[(Y-c)^2]$ is the average $c=\mathbb E[Y]$ .

Expand:

\mathbb E[(Y-c)^2]=\mathbb E[Y^2]-2c\,\mathbb E[Y]+c^2

. This is a bowl in

c

(Unit 9); its slope

-2\,\mathbb E[Y]+2c

is zero at

c=\mathbb E[Y]

. A parabola opening upwards has one bottom.

With

Y

equal to 0 or 1, each half the time,

\mathbb E[Y]=0.5

: the best guess is a grey 0.5 with cost

0.25

, while guessing 0 or 1 costs

0.5

. ∎ A decoder that cannot tell whether the mouth curves up or down paints the average of both — a soft blur. Diffusion avoids this by never deciding everything at once.

In one sentence: A trained VAE is a map of faces you can walk — draw a point from the prior to create, walk a straight line to morph — but where the code leaves a detail open, its squared-error decoder paints the average, so its pictures are soft.

Forward: destroy a picture one small step at a time

Imagine this

Drop a single drop of blue ink into a glass of still water. At first it is a sharp little cloud with a clear shape. A second later it has spread a bit; a minute later it is a faint haze; after an hour the whole glass is an even pale blue, and nothing tells you what shape the drop once had.

Destroying is easy. Every moment the ink simply spreads a little more. Nobody has to plan it.

The question. The VAE drew a picture in one go, and it came out blurry. Diffusion takes the opposite road: it learns to undo destruction. But first we need the destruction itself. How do we turn a picture into pure static — in a way so simple that we know exactly what happened at every step?

A picture is a list of numbers (for our faces, 400 pixels, scaled to run from −1 to 1). We use the step of §2, over and over, with a small noise amount $\beta_t$ at step $t$ :

$\begin{gathered}x_t=\sqrt{1-\beta_t}\,x_{t-1}+\sqrt{\beta_t}\,\varepsilon_t,\\ \varepsilon_t\sim N(0,I).\end{gathered}$

Shrink the picture a little, add a little noise — the chai glass. Every pixel gets its own noise.

By hand, three steps. Take $\beta=0.1$ at every step. The picture is shrunk by $\sqrt{0.9}$ three times, and the three noises pile up. The two facts of §2 add them into one noise:

$\begin{aligned}\bar\alpha_3&=0.9\times0.9\times0.9=0.729\\ x_3&=\sqrt{0.729}\,x_0+\sqrt{1-0.729}\,\varepsilon\\ &\approx0.8538\,x_0+0.5206\,\varepsilon.\end{aligned}$

This is the key shortcut. We never have to walk the steps one by one: to jump straight to step $t$ , multiply the "keep" factors together, $\bar\alpha_t=(1-\beta_1)(1-\beta_2)\cdots(1-\beta_t)$ , and draw one noise:

$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon.$

It is the reparameterisation trick of §5 again: a formula of the picture and outside noise.

The real schedule. DDPM (Ho, Jain and Abbeel, 2020) uses $T=1\,000$ steps, with $\beta_t$ rising in a straight line from 0.0001 to 0.02. Then

$\begin{aligned}\bar\alpha_{100}&\approx0.897\ \ (\text{SNR }8.71)\\ \bar\alpha_t&<0.5\ \text{ from step }260\ \text{on}\\ \bar\alpha_{500}&\approx0.07859\ \ (\text{SNR }0.0853)\\ \bar\alpha_{1000}&\approx4.04\times10^{-5}.\end{aligned}$

The signal-to-noise ratio $\bar\alpha/(1-\bar\alpha)$ compares the variance kept from the picture with the variance of the noise. At step 100 the picture is still 8.71 times louder than the noise. By step 500 the noise is about 12 times louder. At step 1 000 only $\sqrt{4.04\times10^{-5}}\approx0.0064$ of the picture survives: it is pure static, $N(0,I)$ , whatever you started from.

Try: Drag t slowly from 0 to 1 000 and watch the face and the rangoli melt into the same grey static. Press t = 100: $\bar\alpha=0.897$ , signal-to-noise 8.71 — the face is still easy to see, but the rangoli's fine pattern is already a blur: small details vanish first, big shapes last. t = 260: $\bar\alpha$ has just dropped below 0.5. t = 500: 0.07859, you can barely guess the face. Switch to cosine: at t = 500 it keeps $\bar\alpha\approx0.49$ — the picture fades more evenly, instead of being mostly destroyed in the first third.

t100

Why does this work?

Because every step is §2's step, which keeps the total size fixed while swapping picture for noise. Independent noises simply add their variances, so a thousand small noises are exactly one big noise, and a thousand small shrinks are one big shrink $\sqrt{\bar\alpha_t}$ . The destination does not depend on the picture at all: every picture ends in the same $N(0,I)$ . That fixed, easy destination is what will let us start the reverse journey from pure static.

Two schedules, side by side.

	linear (DDPM, Ho et al., 2020)	cosine (Nichol and Dhariwal, 2021)
recipe	$\beta_t$ rises in a straight line, 0.0001 → 0.02	$\bar\alpha_t$ follows a cosine curve from 1 to 0
$\bar\alpha$ at t = 500	0.07859	about 0.49
how the picture fades	fast early: much of the late journey is already static	evenly across all the steps
good for	the original DDPM results	small pictures, where the linear one destroys detail too soon

Rule of thumb. The schedule only decides how fast the picture fades. Whatever it is, the one-jump formula holds with its own $\bar\alpha_t$ — so you can always jump straight to any step.

Trap

No learning happens in the forward process. It is a fixed recipe with no weights: shrink, add noise, repeat. Everything a diffusion model learns is in the reverse direction, in the next sections. The forward process is only there to make training examples: a clean picture, a step $t$ , and the noise that was added.

The realization

$\begin{gathered}x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon\\ \bar\alpha_t=\prod_{s\le t}(1-\beta_s)\end{gathered}$

A thousand small "shrink and add noise" steps are one big one. The picture survives with weight $\sqrt{\bar\alpha_t}$ , the noise has weight $\sqrt{1-\bar\alpha_t}$ , and at the end every picture is the same pure static.

Pause & predict

Two steps with $\beta=0.36$ each. Write $x_2$ as one jump from $x_0$ .

Pause & predict

With the linear schedule, you forward-noise two very different pictures — a face and a cat — to $t=1\,000$ . How different are the two results?

Pause & predict

At step 100 of DDPM, $\bar\alpha=0.897$ . Roughly how much louder is the picture than the noise, in variance?

If you want the algebra · 1 proof, step by step

Prove it · a thousand small noisings are one big one

Claim. If $x_t=\sqrt{1-\beta_t}\,x_{t-1}+\sqrt{\beta_t}\,\varepsilon_t$ with fresh independent noise at every step, then $x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon$ for one standard noise $\varepsilon$ , where $\bar\alpha_t=\prod_{s\le t}(1-\beta_s)$ .

True for

t=1

\bar\alpha_1=1-\beta_1

. Suppose it holds for

t-1

. Substitute:

\begin{aligned}x_t&=\sqrt{\bar\alpha_t}\,x_0\\ &\quad+c\,\varepsilon'+\sqrt{\beta_t}\,\varepsilon_t,\\ c&=\sqrt{(1-\beta_t)(1-\bar\alpha_{t-1})}.\end{aligned}

(1-\beta_t)\bar\alpha_{t-1}=\bar\alpha_t

by the definition of

\bar\alpha

. The last two terms are two independent noises.

Two independent bells add into one bell whose variance is the sum (§2):

\begin{aligned}&(1-\beta_t)(1-\bar\alpha_{t-1})+\beta_t\\ &=1-(1-\beta_t)\bar\alpha_{t-1}\\ &=1-\bar\alpha_t.\end{aligned}

So the noise part is

\sqrt{1-\bar\alpha_t}\,\varepsilon

for a single standard

\varepsilon

. ∎

\beta=0.1

three times:

\bar\alpha_3=0.729

x_3=0.8538x_0+0.5206\varepsilon

In one sentence: The forward process spreads the ink — shrink a little, add a little noise, a thousand times — and because the noises add up into one, any step can be reached in a single jump, $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon$ , ending in the same static for every picture.

The denoiser: guess the noise

Imagine this

An old photograph of your grandparents' wedding has faded and gathered dust. A restorer at a studio in Chandni Chowk looks at it for a long time and says: "This much of what you see is dust." He paints the dust out, and the wedding comes back.

He does not paint the picture from memory. He only has to be good at one thing: spotting the dust.

The question. The forward process is fixed. So what does a diffusion model actually learn?

It learns to be the restorer. A network $\varepsilon_\theta(x_t,t)$ looks at a noisy picture and the step number, and guesses the noise $\varepsilon$ that was added. Training is a plain regression with squared error — the loss of Unit 10 and Unit 15:

Take a real picture $x_0$ and a random step $t$ .
Draw noise $\varepsilon$ and make $x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon$ in one jump (§8).
Ask the network for its guess $\hat\varepsilon=\varepsilon_\theta(x_t,t)$ , and pay $\|\varepsilon-\hat\varepsilon\|^2$ .

That is all. No second network, no guessing game: the right answer — the noise — is known for every training example, because we added it ourselves. (Underneath, it is §6 again: Ho and colleagues showed that this plain squared error is an ELBO, like the VAE's, in which the noisy pictures play the part of the code — with the steps weighted a little differently. The bookkeeping just comes out very simple.)

Knowing the noise is knowing the picture. Solve the one-jump formula for $x_0$ :

$\hat x_0=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\hat\varepsilon}{\sqrt{\bar\alpha_t}}.$

Paint out the dust, then turn the brightness back up. By hand, with one number: $x_0=2$ , $\bar\alpha=0.64$ (so $\sqrt{\bar\alpha}=0.8$ , $\sqrt{1-\bar\alpha}=0.6$ ), and noise $\varepsilon=0.5$ :

$\begin{aligned}x_t&=0.8(2)+0.6(0.5)=1.9\\ \hat\varepsilon=0.5:\ \ \hat x_0&=\frac{1.9-0.6(0.5)}{0.8}=2\\ \hat\varepsilon=0.3:\ \ \hat x_0&=\frac{1.9-0.6(0.3)}{0.8}=2.15.\end{aligned}$

A perfect guess gives the picture back exactly. A guess that is 0.2 too small leaves some noise in, and the rescaling blows it up a little.

What network? For pictures, a U-Net (a network that squeezes the picture down and widens it back up, with shortcuts between matching sizes), or a transformer (Unit 18) that reads the picture as patches. Either way it also reads $t$ , so it knows how much dust to expect.

Try: With $x_0=2$ , $\bar\alpha=0.64$ , $\varepsilon=0.5$ : set the guess $\hat\varepsilon$ to 0.5 and the gold dot lands on 2 with loss 0; set it to 0.3 and it lands on 2.15. Now push $\bar\alpha$ down to 0.05 (very noisy): in the bottom panel every best guess crowds into a small blob near the centre — the average of the whole rangoli. Raise $\bar\alpha$ to 0.95: the best guesses spread out and sit on the rangoli.

x₀2

ᾱ0.64

ε0.5

guess ε̂0.3

Why does this work?

Because squared error teaches averages (the drawer of §7). Many different clean pictures could have produced the same noisy $x_t$ , and the network cannot know which one it was. So it learns the average noise over all of them — and, through the formula, the average clean picture. When the noise is small, only one clean picture fits, and the average is sharp. When the noise is huge, everything fits, and the best guess is the average of the whole dataset. Each guess is honest about how much it can know.

What should the network predict? Two choices, side by side.

the network predicts	the clean picture $\hat x_0$	the noise $\hat\varepsilon$ (DDPM)
loss	$\\|x_0-\hat x_0\\|^2$	$\\|\varepsilon-\hat\varepsilon\\|^2$
size of the target	a whole picture, every step	always $N(0,I)$ : the same size at every step
convert to the other	$\hat\varepsilon=(x_t-\sqrt{\bar\alpha}\,\hat x_0)/\sqrt{1-\bar\alpha}$	$\hat x_0=(x_t-\sqrt{1-\bar\alpha}\,\hat\varepsilon)/\sqrt{\bar\alpha}$
in practice	works; often weaker at low noise	the standard choice since DDPM

Rule of thumb. Predict the noise: the target always has the same size, which keeps training steady at every step. §10 adds a third choice, the score. Any of the three can be turned into the others with one line of algebra.

Trap

The network is not used to draw the final picture in one go. From a very noisy input its best guess $\hat x_0$ is a blurry average — in the widget, a blob near the centre. That is why diffusion does not stop at the first guess. It uses the guess to take one small step back, looks again, guesses again, and sharpens the picture over many steps (§11).

The realization

$\begin{gathered}\min_\theta\ \mathbb E\big\|\varepsilon-\varepsilon_\theta(x_t,\ t)\big\|^2\\ x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon\\ \hat x_0=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\hat\varepsilon}{\sqrt{\bar\alpha_t}}\end{gathered}$

A diffusion model is a restorer trained by squared error to spot the dust: noise a real picture by a known amount, and ask for the noise back. Knowing the noise is the same as knowing the clean picture — as well as it can be known from what is left.

Pause & predict

$x_t=1$ , $\bar\alpha=0.36$ (so $\sqrt{\bar\alpha}=0.6$ , $\sqrt{1-\bar\alpha}=0.8$ ), and the restorer guesses $\hat\varepsilon=0.5$ . What is $\hat x_0$ ?

Pause & predict

A perfect restorer sees pure static at $t=1\,000$ . What is its best guess $\hat x_0$ of the clean picture?

If you want the algebra · 1 proof, step by step

Prove it · the best noise guess gives the best clean guess

Claim. Among all functions of $x_t$ , the one with the smallest average $\|\varepsilon-\hat\varepsilon\|^2$ is $\hat\varepsilon(x_t)=\mathbb E[\varepsilon\mid x_t]$ ; and then $\hat x_0=(x_t-\sqrt{1-\bar\alpha}\,\hat\varepsilon)/\sqrt{\bar\alpha}$ equals $\mathbb E[x_0\mid x_t]$ , the average clean picture behind $x_t$ .

Fix one

x_t

. The guess is a single number (or vector)

c

, and the drawer of §7 says the

c

with the smallest average squared error is the average:

c=\mathbb E[\varepsilon\mid x_t]

. A network trained long enough on squared error learns these conditional averages (Unit 14's least squares, grown up).

Average the identity

x_0=(x_t-\sqrt{1-\bar\alpha}\,\varepsilon)/\sqrt{\bar\alpha}

over everything consistent with this

x_t

. The formula is a straight line in

\varepsilon

, so the average passes through:

\begin{aligned}&\mathbb E[x_0\mid x_t]\\ &=\frac{x_t-\sqrt{1-\bar\alpha}\,\mathbb E[\varepsilon\mid x_t]}{\sqrt{\bar\alpha}}.\end{aligned}

∎ So at huge noise, where every picture is consistent with

x_t

, the best clean guess is the dataset's average — the blob in the widget.

In one sentence: A diffusion model is a restorer who, shown a noisy picture and how noisy it is, guesses the dust by plain squared-error regression — and from the dust guess, $\hat x_0=(x_t-\sqrt{1-\bar\alpha}\,\hat\varepsilon)/\sqrt{\bar\alpha}$ is the best clean picture that can be known.

The score: a compass pointing at the data

Imagine this

You are lost in fog in the hills above a few villages. You hold a strange compass: it always points towards the most likely place to find people. Far out in the fog it points vaguely towards the middle of all the villages. As you come closer, it starts to choose: this village, not that one. Right at the edge of a village it points sharply to its centre.

The question. The restorer's noise guess feels like a trick for one picture. Is there a single field — an arrow at every point of space — that tells us which way real data lies?

There is. It is called the score: the slope (gradient, Unit 6) of the log of the density, $\nabla_x\log p(x)$ . At every point it is an arrow pointing uphill — towards where data is more likely.

One village, by hand. For a bell curve $N(\mu,\sigma^2)$ , $\log p(x)=-\frac{(x-\mu)^2}{2\sigma^2}+\text{const}$ , so its slope is

$\text{score}(x)=-\frac{x-\mu}{\sigma^2}.$

With $\mu=2$ and $\sigma^2=0.25$ : at $x=1$ the score is $-(1-2)/0.25=+4$ (go right, towards 2); at $x=3$ it is $-4$ (go left). The farther away, the stronger the pull; at the centre, 0.

The noise guess is the score in disguise. Given one clean $x_0$ , the noisy $x_t$ is a bell curve with centre $\sqrt{\bar\alpha}\,x_0$ and variance $1-\bar\alpha$ . Its score is $-(x_t-\sqrt{\bar\alpha}x_0)/(1-\bar\alpha)=-\varepsilon/\sqrt{1-\bar\alpha}$ . So

$\text{score}=-\frac{\hat\varepsilon}{\sqrt{1-\bar\alpha}}.$

Check it on §9's numbers ( $x_0=2$ , $\bar\alpha=0.64$ , $x_t=1.9$ ): from the bell curve, $-(1.9-1.6)/0.36\approx-0.8333$ ; from the noise, $-0.5/0.6\approx-0.8333$ . The same arrow, both ways. A trained restorer is a compass.

Two villages. Now the data is two bumps: half the points near $-2$ , half near $+2$ , each with variance 0.01. Noise it to $\bar\alpha=0.64$ : the bumps move to $\pm0.8\times2=\pm1.6$ , and each widens to variance $0.64(0.01)+0.36=0.3664$ . At $x=0.5$ , how much does each village "claim" the point? Compare the two bells (the responsibilities, as in Unit 14's Bayes):

$\begin{aligned}\text{claims}&\propto e^{-(0.5+1.6)^2/(2\times0.3664)},\\ &\quad\ e^{-(0.5-1.6)^2/(2\times0.3664)}\\ &=(0.0125,\ 0.9875).\end{aligned}$

The score is the claims times each village's own pull:

$\begin{aligned}&0.0125\cdot\frac{-(0.5+1.6)}{0.3664}\\ &+0.9875\cdot\frac{-(0.5-1.6)}{0.3664}\approx2.8927.\end{aligned}$

A strong arrow to the right: the compass has picked the nearer village.

Try: In two villages at $\bar\alpha=0.64$ , drag the marker to 0.5: the claims read (0.0125, 0.9875) and the score 2.8927. Now slide $\bar\alpha$ down to 0.05: the two bells merge into one soft hill, and the compass everywhere just points to the middle. Slide it up to 0.99: two sharp villages, and a steep cliff in the score between them — the compass flips from "go left" to "go right" at 0. Switch to one village and drag to 1 and 3: +4 and −4.

ᾱ0.64

x0.5

In two dimensions. The rangoli is 55 small villages. The same rule holds: at every point, the score is the claims of the villages times their pulls — and we can compute it exactly. Below, the height of the landscape is $\log p$ of the noisy rangoli, and the gold arrows running over it are the compass. In this widget, and in all of Act III, the trained network is replaced by this exact answer, so every arrow is true. A real model learns an approximation of this field from examples.

Try: Start at t = 600: one smooth hill, and every arrow points to the middle. Slide down to t = 300, then t = 100: still one hill, but its top flattens into a wide plateau — the outer ring of lamps is beginning to pull. At t = 60 the plateau breaks up into rings and petals, and some arrows turn outwards: the compass starts to choose. At t = 30 the landscape is 55 sharp peaks and each arrow points to its own nearest lamp or petal. Press the probe buttons to read the compass exactly — for example between two lamps.

drag the picture to orbit

t300

probe

Why does this work?

Because of one small miracle, called Tweedie's formula: take one step along the score, of size $1-\bar\alpha$ , and you land exactly on the restorer's best average guess, scaled: $x_t+(1-\bar\alpha)\,\text{score}=\sqrt{\bar\alpha}\,\hat x_0$ . So the arrow points from where you are to the average of the clean pictures that could have made this noisy one. Far away, that average is the centre of everything. Close to one village, it is that village. The compass is just the restorer's guess, turned into a direction.

Three ways to say the same thing, side by side. Any one of them can be computed from the others.

	the noise guess $\hat\varepsilon$	the clean guess $\hat x_0$	the score
answers	"how much of this is dust?"	"what was under the dust?"	"which way is more likely?"
from $\hat\varepsilon$	—	$(x_t-\sqrt{1-\bar\alpha}\,\hat\varepsilon)/\sqrt{\bar\alpha}$	$-\hat\varepsilon/\sqrt{1-\bar\alpha}$
picture	the dust layer	the restored photo	a compass needle
who uses it	DDPM training (§9)	DDIM's jump (§12)	score models, Langevin (§11)

Rule of thumb. Train the noise, think in scores. When you picture a diffusion model, picture a field of arrows that sharpens as the noise goes down.

Trap

The score is not the probability. It points uphill, but it says nothing directly about how high you are. The compass reads 0 at the very top of a village — and also exactly halfway between two equal villages, where the pulls cancel. A zero score means "flat", not "likely".

The realization

$\begin{gathered}\nabla_x\log p_t(x)\\ =\sum_k r_k(x)\,\frac{\sqrt{\bar\alpha}\,\mu_k-x}{v}\\ \text{score}=-\frac{\hat\varepsilon}{\sqrt{1-\bar\alpha}}\end{gathered}$

The score is a compass at every point, pointing towards more likely data. For a mixture it is each village's pull, weighted by how strongly that village claims the point. A noise predictor is a compass: divide its guess by $-\sqrt{1-\bar\alpha}$ .

Pause & predict

Two equal villages at $\pm1.6$ (after noising), and you stand exactly at $x=0$ . What does the compass read?

Pause & predict

You noise the two-village data much more, to $\bar\alpha=0.05$ . At $x=0.5$ , what happens to the compass compared with $\bar\alpha=0.64$ ?

Pause & predict

A trained model sees $x_t$ at $\bar\alpha=0.91$ and predicts $\hat\varepsilon=0.6$ . What is its score there?

If you want the algebra · 2 proofs, step by step

Prove it · the noise guess is the score

Claim. For the noised data $p_t$ , the score is $\nabla\log p_t(x_t)=-\mathbb E[\varepsilon\mid x_t]/\sqrt{1-\bar\alpha}$ . (For one known $x_0$ this is just the score of the bell curve $N(\sqrt{\bar\alpha}x_0,\ 1-\bar\alpha)$ .)

For one clean

x_0

\log p(x_t\mid x_0)=-\frac{\|x_t-\sqrt{\bar\alpha}x_0\|^2}{2(1-\bar\alpha)}+\text{const}

, whose slope is

-(x_t-\sqrt{\bar\alpha}x_0)/(1-\bar\alpha)=-\varepsilon/\sqrt{1-\bar\alpha}

x_t-\sqrt{\bar\alpha}x_0=\sqrt{1-\bar\alpha}\,\varepsilon

by the one-jump formula.

For the whole data,

p_t(x_t)=\sum p(x_0)\,p(x_t\mid x_0)

. The slope of a log of a sum is a weighted average of the separate slopes, weighted by how much each

x_0

explains

x_t

— its posterior:

\begin{aligned}\nabla\log p_t&=\mathbb E_{x_0\mid x_t}\Big[-\tfrac{\varepsilon}{\sqrt{1-\bar\alpha}}\Big]\\ &=-\frac{\mathbb E[\varepsilon\mid x_t]}{\sqrt{1-\bar\alpha}}.\end{aligned}

∎

\nabla\log\sum_k f_k=\sum_k\frac{f_k}{\sum_jf_j}\nabla\log f_k

. With §9:

x_t+(1-\bar\alpha)\nabla\log p_t=\sqrt{\bar\alpha}\,\hat x_0

— Tweedie's formula.

Prove it · the compass of a mixture of villages

Claim. For equal-weight bells $N(m_k,\ vI)$ (the noised rangoli: $m_k=\sqrt{\bar\alpha}\mu_k$ , $v=\bar\alpha s^2+1-\bar\alpha$ ), the score is $\sum_kr_k(x)\,(m_k-x)/v$ , with claims $r_k\propto e^{-\|x-m_k\|^2/(2v)}$ .

p(x)=\frac1K\sum_k f_k(x)

with

f_k

the bells. Each

\nabla\log f_k=(m_k-x)/v

(the one-village formula). Adding noise to a small bell keeps it a bell: its centre shrinks by

\sqrt{\bar\alpha}

, its variance becomes

\bar\alpha s^2+(1-\bar\alpha)

(§2).

The log-of-a-sum rule of the previous drawer gives weights

r_k=f_k/\sum_jf_j

; the common factors of the bells cancel, leaving

r_k\propto e^{-\|x-m_k\|^2/(2v)}

. ∎ Two villages at

\pm1.6

v=0.3664

x=0.5

r=(0.0125,\ 0.9875)

, score

\approx2.8927

. Since every

v

is the same, the score is

(\bar m(x)-x)/v

: from

x

to the claim-weighted average of the villages.

In one sentence: The score $\nabla\log p$ is a compass pointing towards more likely data — fuzzy and central in thick fog, sharp and local near a village — and a noise predictor is that compass in disguise: $\text{score}=-\hat\varepsilon/\sqrt{1-\bar\alpha}$ .

Walking back from static

Imagine this

Film the ink drop spreading in the glass, and play the film backwards. The even haze gathers, a little at a time, into a faint cloud, then a sharper one, until the single drop hangs there again, with its shape.

Played backwards, every small moment looks like "the ink moves a little towards where it is thicker". That is all the reverse film is.

The question. We have a compass that works at every noise level. How do we use it to turn pure static into a picture?

Start from pure static, $x_T\sim N(0,I)$ , and repeat for $t=T,\ T-1,\ \dots,\ 1$ :

Ask the restorer for the noise, $\hat\varepsilon=\varepsilon_\theta(x_t,t)$ — the compass.
Step a little uphill: remove a small part of the guessed noise and undo one step's shrink.
Add a little fresh noise, of the right size for this step.

DDPM's step says exactly how much of each:

$\begin{aligned}x_{t-1}=\ &\frac{1}{\sqrt{\alpha_t}}\Big(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\hat\varepsilon\Big)\\ &+\sigma_t\,z,\qquad z\sim N(0,1),\end{aligned}$

with $\alpha_t=1-\beta_t$ and $\sigma_t=\sqrt{\beta_t}$ . Read it aloud. $\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\hat\varepsilon$ : only the share of the guessed noise that this one step added. $\frac1{\sqrt{\alpha_t}}$ : undo this step's shrink. $\sigma_tz$ : a fresh jiggle.

By hand. $\beta_t=0.1$ (so $\alpha_t=0.9$ ), $\bar\alpha_t=0.729$ , $x_t=1.2$ , $\hat\varepsilon=0.5$ , and the fresh draw is $z=-0.3$ :

$\begin{aligned}\text{mean}&=\tfrac{1}{\sqrt{0.9}}\Big(1.2-\tfrac{0.1}{\sqrt{0.271}}(0.5)\Big)\\ &\approx1.1637\\ \sigma_t&=\sqrt{0.1}\approx0.3162\\ x_{t-1}&=1.1637+0.3162(-0.3)\\ &\approx1.0688.\end{aligned}$

You have met this walker before. Write $\hat\varepsilon$ as the score (§10) and the step has the same shape as Langevin dynamics — a small step along the score, plus a jiggle:

$x\leftarrow x+\eta\,\nabla\log p(x)+\sqrt{2\eta}\,z.$

For $p=N(0,1)$ the score is $-x$ . From $x=3$ with $\eta=0.1$ and no jiggle ( $z=0$ ): $x\leftarrow3+0.1(-3)=2.7$ . That is Unit 9's gradient descent walker, walking downhill on the landscape $-\log p$ — with a jiggle at every step (Unit 9's noise as a friend), so that the walkers spread over the whole distribution instead of all piling onto its highest point.

Try: Press ▶ walk back with 100 steps: the static drifts, gathers, and lands on the rangoli — about 96% of the points on the shape, all 55 parts used. Switch fresh noise off and walk again: every point piles onto the centre dot — 1 part of 55. Put the noise back and choose 10 steps: less than half land on the shape. Drag scrub to replay any moment.

drag the picture to orbit

steps

scrub0

Why does this work?

Because each forward step is tiny. When a step adds only a little noise, undoing it is almost a bell curve too — its centre given by the compass, its spread $\sigma_t$ . (This is Unit 8's Taylor idea: over a small enough step, everything looks simple.) So a thousand small, simple reverse steps undo a thousand small forward steps. And the fresh noise matters: the reverse film must end with the right spread of pictures, and a compass alone only knows the direction of more likely data.

With and without the jiggle, side by side. These are the widget's own counts, 3 000 points, 100 steps, the same starting static.

	DDPM, fresh noise on	fresh noise off (only the "mean" step)	Langevin (a fixed landscape)
each step	compass step + $\sigma_tz$	compass step only	$\eta\cdot$ score $+\sqrt{2\eta}\,z$
where the points end	spread over the whole rangoli	all on the centre dot	spread over $p$ , if run long enough
parts of the rangoli used	55 of 55	1 of 55	—
noise level	walks down from 1 000 to 0	walks down from 1 000 to 0	one fixed level

Rule of thumb. Keep the jiggle. The compass tells each walker where "more likely" is; the jiggle keeps the walkers from all agreeing. Want a sampler without randomness? Use DDIM (§12), which is built to be deterministic — not DDPM with the noise switched off.

Trap

Removing the added noise $z$ does not give "cleaner" samples. It gives less varied ones. Without the jiggle, the early steps pull every walker towards the middle of everything, and they all end in the same place — here, all 3 000 points on the single centre dot of the rangoli. The randomness is not a flaw in the sampler; it is what makes it a sampler.

The realization

$\begin{aligned}x_{t-1}=\ &\frac{1}{\sqrt{\alpha_t}}\Big(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\hat\varepsilon\Big)\\ &+\sqrt{\beta_t}\,z\end{aligned}$

Generation is the ink film played backwards: start from static, and at every step move a little along the compass and add a little fresh noise. It is gradient descent on $-\log p$ with a jiggle, walked down through the noise levels from very blurry to sharp.

Pause & predict

In the worked step, the fresh draw is $z=+0.3$ instead of $-0.3$ . What is $x_{t-1}$ ?

Pause & predict

You run the reverse walk with no fresh noise. What do the finished samples look like?

Pause & predict

One Langevin step for $p=N(0,1)$ , from $x=-2$ , with $\eta=0.25$ and jiggle $z=0.4$ . Where do you land?

If you want the algebra · 2 proofs, step by step

Prove it · where DDPM's step comes from

Claim. If we knew $x_0$ , the step back from $x_t$ would be a bell curve with centre $\tilde\mu=\frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0+\frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t$ . Putting in $\hat x_0$ from $\hat\varepsilon$ gives exactly DDPM's mean $\frac1{\sqrt{\alpha_t}}\big(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\hat\varepsilon\big)$ .

Bayes:

p(x_{t-1}\mid x_t,x_0)\propto p(x_t\mid x_{t-1})\,p(x_{t-1}\mid x_0)

, a product of two bells in

x_{t-1}

— itself a bell. Its centre is the precision-weighted average of the two "votes"

x_t/\sqrt{\alpha_t}

(weight

\alpha_t/\beta_t

) and

\sqrt{\bar\alpha_{t-1}}x_0

(weight

1/(1-\bar\alpha_{t-1})

); simplifying with

\alpha_t\bar\alpha_{t-1}=\bar\alpha_t

gives

\tilde\mu

above. Multiplying bells adds their precisions (Unit 14).

Substitute

x_0=\big(x_t-\sqrt{1-\bar\alpha_t}\,\hat\varepsilon\big)/\sqrt{\bar\alpha_t}

. The

x_t

terms collect to

x_t/\sqrt{\alpha_t}

and the

\hat\varepsilon

term to

-\frac{\beta_t}{\sqrt{\alpha_t}\sqrt{1-\bar\alpha_t}}\hat\varepsilon

. ∎ The spread of that bell is

\tilde\beta_t=\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t

; DDPM found

\sigma_t^2=\beta_t

works as well, and it is the choice used here.

Prove it · Langevin keeps the right distribution

Claim (for a bell). For $p=N(0,1)$ , the Langevin step $x'=x+\eta(-x)+\sqrt{2\eta}\,z$ maps $N(0,v)$ to $N\big(0,(1-\eta)^2v+2\eta\big)$ , whose only fixed point for small $\eta$ is $v\approx1$ — the target itself.

x'=(1-\eta)x+\sqrt{2\eta}\,z

: the two facts of §2 give variance

(1-\eta)^2v+2\eta

. A scaled bell plus an independent bell.

Set

v'=v

v\big(1-(1-\eta)^2\big)=2\eta

, so

v=\frac{2\eta}{2\eta-\eta^2}=\frac{2}{2-\eta}\approx1

for small

\eta

. ∎ Without the jiggle (

z=0

) the variance shrinks by

(1-\eta)^2

every step and every walker ends at 0 — the collapse of the widget. With

\eta=0.1

from

x=3

3-0.3=2.7

In one sentence: To create, play the ink film backwards — start from static, and a thousand times move a little along the compass and add a little fresh noise — which is Unit 9's downhill walker on $-\log p$ , with the jiggle that keeps every walker from ending in the same place.

Faster: DDIM and the smooth path

Imagine this

You are coming down a long staircase in the dark, one step at a time, holding the rail. Then someone switches on the light. Now you can see the landing far below, and you can take the stairs three or four at a time — you know where you are heading.

The question. DDPM needs a thousand steps, and every step runs the whole network once. Can we reach the picture in far fewer steps?

DDIM (Song, Meng and Ermon, 2021) takes big, deterministic jumps. At every visit it does two things with the same noise guess:

Look at the landing: estimate the clean picture, $\hat x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\hat\varepsilon)/\sqrt{\bar\alpha_t}$ (§9).
Jump straight to a much lower noise level $s$ , re-noising the landing with that same $\hat\varepsilon$ :

$x_s=\sqrt{\bar\alpha_s}\,\hat x_0+\sqrt{1-\bar\alpha_s}\,\hat\varepsilon.$

By hand. $x_t=1.9$ , $\bar\alpha_t=0.64$ , $\hat\varepsilon=0.5$ : the landing is $\hat x_0=2$ (as in §9). Jump to $\bar\alpha_s=0.9$ :

$x_s=\sqrt{0.9}(2)+\sqrt{0.1}(0.5)\approx2.0555.$

No fresh noise is drawn. So the walk is deterministic: the same starting static always gives the same picture. And because it never throws a random jiggle in, the path is smooth, and it can be walked with big strides. (In the limit of tiny steps it follows an ordinary differential equation, the probability-flow ODE: a smooth river that carries every point of the static to a picture.) Song and colleagues report that 50 DDIM steps give pictures close to those of the full 1 000-step DDPM.

Try: With 50 steps, press again, same starts a few times: the DDPM paths jitter and land somewhere new every time; the DDIM paths do not move at all. Slide the steps down to 10: both land off the shape more often — fewer steps are not free. Slide up to 100: DDIM puts about 95% of its walkers on the shape.

steps50

Why does this work?

Because the restorer's guess already contains a full picture, $\hat x_0$ . DDPM uses only a sliver of it per step and relies on fresh noise to keep the right spread. DDIM instead keeps the noise it already has — the same $\hat\varepsilon$ — and re-mixes it with the landing in exactly the proportions the forward process would have at level $s$ . So after every jump the point looks like a proper sample of level $s$ , and the next compass reading is trustworthy even after a big stride.

DDPM and DDIM, side by side. The last row is the widget's own count: 400 walkers on our rangoli.

	DDPM (§11)	DDIM
each step	small compass step + fresh noise	estimate $\hat x_0$ , jump, re-noise with the same $\hat\varepsilon$
randomness	at every step	only the starting static
same start twice	two different pictures	the same picture
typical steps	1 000	25–100
same network?	yes	yes — only the sampler changes
our rangoli, 400 walkers	50 steps: about 88% on the shape; 100 steps: about 95%	50 steps: 92.5%; 100 steps: about 95%

Rule of thumb. Train once, sample however you like. DDIM with 50–100 steps is the usual fast choice; add back some randomness when you want more variety between runs.

Trap

Fewer steps are not free. Each jump trusts one compass reading over a long distance; if the stride is too long, the reading is out of date by the time you land. With 10 steps, fewer than half of our walkers land on the rangoli, with DDIM as well as with DDPM.

The realization

$\begin{aligned}x_s=\ &\sqrt{\bar\alpha_s}\,\frac{x_t-\sqrt{1-\bar\alpha_t}\,\hat\varepsilon}{\sqrt{\bar\alpha_t}}\\ &+\sqrt{1-\bar\alpha_s}\,\hat\varepsilon\end{aligned}$

DDIM looks at the landing and jumps: estimate the clean picture, then re-noise it with the same noise guess to a much lower level. No fresh randomness, so the path is smooth, the same start gives the same picture, and a few dozen strides can do the work of a thousand small steps.

Pause & predict

A DDIM jump from $x_t=1.9$ , $\bar\alpha_t=0.64$ , $\hat\varepsilon=0.5$ , all the way to $\bar\alpha_s=1$ (no noise at all). Where does it land?

Pause & predict

You run DDIM twice from the same starting static, with the same trained network. What do you get?

If you want the algebra · 1 proof, step by step

Prove it · a DDIM jump lands at the right noise level

Claim. If the network is exact for a single clean picture (so $\hat\varepsilon=\varepsilon$ and $\hat x_0=x_0$ ), then the DDIM jump $x_s=\sqrt{\bar\alpha_s}\hat x_0+\sqrt{1-\bar\alpha_s}\hat\varepsilon$ gives exactly $\sqrt{\bar\alpha_s}\,x_0+\sqrt{1-\bar\alpha_s}\,\varepsilon$ — a proper level- $s$ sample of the forward process, with the same noise.

Substitute:

\hat x_0=\big(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon-\sqrt{1-\bar\alpha_t}\varepsilon\big)/\sqrt{\bar\alpha_t}=x_0

. The guessed noise removes exactly the noise that was added.

Then

x_s=\sqrt{\bar\alpha_s}x_0+\sqrt{1-\bar\alpha_s}\varepsilon

, which is the one-jump formula of §8 at level

s

. Walking from

t

s

moved along a fixed line in the

(x_0,\varepsilon)

plane: no randomness was added. ∎ With many possible pictures the guesses are averages, so each jump is only approximately right — which is why strides that are too long drift off the data.

In one sentence: DDIM switches on the light on the staircase — at each visit it estimates the clean picture and jumps far down with the same noise guess — so the path is smooth, the same start always gives the same picture, and a few dozen strides replace a thousand steps.

Steering with words: guidance and latent diffusion

Imagine this

You are driving to a friend's house with the map app on. At a busy junction the voice says: "Keep left." Your own habit — "most people go straight here" — pulls you one way; the voice pulls you another. A good voice even says: "More to the left than you think." It takes the difference between where you would go and where you should go, and makes it bigger.

The question. A diffusion model draws something that looks real. How do we make it draw what we ask for — "a cat on a scooter" — and how do big models afford to do it on huge pictures?

Step 1: give the restorer the request. Train the noise guesser with the text as an extra input: $\hat\varepsilon(x_t,t,c)$ , where $c$ is the prompt. A transformer (Unit 18) reads the text into vectors, and the picture network looks at them with cross-attention: every patch of the picture asks the words "which of you is about me?". Sometimes, during training, the prompt is dropped on purpose, so the same network also learns the unconditional guess $\hat\varepsilon(x_t,t)$ .

Step 2: go past the request. Classifier-free guidance (Ho and Salimans, 2022) mixes the two guesses with a weight $w$ :

$\begin{aligned}\tilde\varepsilon=\ &\varepsilon_{\text{uncond}}\\ &+w\,\big(\varepsilon_{\text{cond}}-\varepsilon_{\text{uncond}}\big).\end{aligned}$

By hand: $\varepsilon_{\text{uncond}}=0.2$ , $\varepsilon_{\text{cond}}=0.5$ , $w=3$ gives $0.2+3(0.5-0.2)=1.1$ . The difference 0.3 is "what the request adds"; guidance adds it three times. $w=0$ ignores the prompt, $w=1$ follows it plainly, and $w>1$ says "more to the left than you think".

On our rangoli we can do it exactly: the "prompt" is a class — the petals or the lamps — and the conditional guess is the exact noise guess of that class's parts alone. So guidance, too, is computed with no network at all.

Try: Choose lamps and set w = 0: the prompt is ignored and about half the walkers land on lamps. w = 1: about 97% on lamps, spread evenly over all 30. w = 3, then w = 5: still lamps — but the inner ring, the lamps that sit closest to the petals, empties out, and the walkers crowd the outer ring. Past 1, guidance buys "the most lamp-like lamps" and pays in variety.

Why does this work?

Turn the guesses into scores (§10). The difference $\varepsilon_{\text{cond}}-\varepsilon_{\text{uncond}}$ is, up to a factor, the slope of $\log p(c\mid x)$ — "how much more does this picture look like the request?", by Bayes (Unit 14). Guidance with weight $w$ samples as if that vote were raised to the power $w$ . A louder vote pushes the walkers to the places that are most typical of the class and least like anything else — sharper and more faithful, but less varied.

Step 3: diffuse in a small space. A 512 × 512 colour picture is $512\times512\times3=786\,432$ numbers, and a diffusion model must run a big network on all of them at every step. Latent diffusion (Rombach and colleagues, 2022 — the idea behind Stable Diffusion) first trains a VAE-like autoencoder (§3–§4) that squeezes the picture into a latent of $64\times64\times4=16\,384$ numbers — 48 times fewer — and runs the whole diffusion there. At the end the decoder turns the finished latent into the full picture. Act II and Act III join: the VAE's map becomes the place where diffusion walks.

The steering choices, side by side.

$w$	0	1	3 to 7 (common for pictures)
what it samples	anything, ignoring the prompt	the prompt's distribution, plainly	the most typical pictures of the prompt
faithful to the prompt	no	yes	very
variety	most	the prompt's own	less; too high looks over-cooked

Rule of thumb. Guidance is a dial between "varied" and "on-message". Latent diffusion is how the dial stays affordable: squeeze with a VAE, diffuse the code, decode once at the end.

Trap

$w=1$ is plain conditional sampling — no guidance at all. "Guidance" means $w>1$ : the model is pushed beyond its own conditional answer, away from the generic one. The widget shows it: at $w=1$ every lamp gets its share; only from $w>1$ does the voice start to insist.

The realization

$\begin{gathered}\tilde\varepsilon=\varepsilon_{\text{u}}+w\,(\varepsilon_{\text{c}}-\varepsilon_{\text{u}})\\ 786\,432\ \to\ 16\,384\ \ (48\times)\end{gathered}$

Steer by taking the difference the request makes to the noise guess and making it louder; walk in a VAE's small latent space to keep it cheap. Words guide the picture through the same compass that made it.

Pause & predict

$\varepsilon_{\text{uncond}}=0.4$ , $\varepsilon_{\text{cond}}=0.1$ , and $w=2$ . What is the guided guess?

Pause & predict

In the widget you raise $w$ from 1 to 5 for the class lamps. What happens?

If you want the algebra · 1 proof, step by step

Prove it · guidance is Bayes' rule with a louder class vote

Claim. In scores, $\tilde\varepsilon=\varepsilon_u+w(\varepsilon_c-\varepsilon_u)$ is the noise guess of the tilted density $p(x)\,p(c\mid x)^{w}$ : it samples as if the "does this look like $c$ ?" vote were raised to the power $w$ .

Bayes (Unit 14):

p(x\mid c)=p(c\mid x)\,p(x)/p(c)

. Take logs and slopes in

x

(

p(c)

has no slope):

\begin{aligned}&\nabla\log p(x\mid c)\\ &=\nabla\log p(x)\\ &\quad+\nabla\log p(c\mid x).\end{aligned}

So the class vote's slope is the difference of the two scores.

Scores are

-\varepsilon/\sqrt{1-\bar\alpha}

(§10), so

\varepsilon_c-\varepsilon_u=-\sqrt{1-\bar\alpha}\,\nabla\log p(c\mid x)

. Then

\begin{aligned}\tilde\varepsilon&=\varepsilon_u+w(\varepsilon_c-\varepsilon_u)\\ &=-\sqrt{1-\bar\alpha}\,\nabla\big(\log p(x)\\ &\qquad+w\log p(c\mid x)\big),\end{aligned}

the noise guess of

p(x)\,p(c\mid x)^w

. ∎

w=0

p(x)

w=1

p(x\mid c)

, up to a constant.

w>1

: the class vote counts

w

times, which favours places where the class is most certain. On the rangoli with class "lamps", that is the outer ring.

In one sentence: Guidance is a sat-nav voice saying "more to the left than you think" — $\tilde\varepsilon=\varepsilon_{\text{uncond}}+w(\varepsilon_{\text{cond}}-\varepsilon_{\text{uncond}})$ — and latent diffusion runs the whole walk in a VAE's code, 48 times smaller, before decoding once at the end.

What to carry forward — and the whole course inside one picture

Thirteen pictures hold this unit. If you can draw each one and say its line, you own the unit.

1 · The lump of clay

To create is to sample: $x=g(z)$ , a fixed skill applied to fresh noise.

2 · The chai glass

Shrink by $\sqrt{1-\beta}$ , add noise of variance $\beta$ : the glass stays full, and every start ends in $N(0,1)$ .

3 · Two numbers on the phone

An autoencoder squeezes and rebuilds; with straight lines it is PCA. Its map has empty land.

4 · A neighbourhood, not a house

A VAE's cloud plus a KL price $\tfrac12(\mu^2+\sigma^2-1-\ln\sigma^2)$ fills the map.

5 · The wobble on a card

$z=\mu+\sigma\varepsilon$ : roll the dice first, and blame flows through.

6 · A platform under the ceiling

$\log p(x)=\text{ELBO}+\text{gap}$ : rebuild well, stay near the prior.

7 · A map of faces

Sample from the prior, walk straight lines to morph; squared error paints blur.

8 · The ink drop

$x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon$ : a thousand noisings in one jump.

9 · The restorer

Learn the dust by squared error; $\hat x_0=(x_t-\sqrt{1-\bar\alpha}\hat\varepsilon)/\sqrt{\bar\alpha}$ .

10 · The compass

$\text{score}=-\hat\varepsilon/\sqrt{1-\bar\alpha}$ : fuzzy far away, sharp near a village.

11 · The film backwards

Step along the compass, add a jiggle, a thousand times. No jiggle, no variety.

12 · The light on the stairs

DDIM: estimate the landing, jump with the same noise guess. Same start, same picture.

13 · The sat-nav voice

Steer with $\varepsilon_u+w(\varepsilon_c-\varepsilon_u)$ : take the difference the request makes and say it louder — "more to the left than you think". And to keep it cheap, run the whole walk in a VAE's small code and decode once at the end.

The families of generators, again — now you know what is inside.

	autoregressive (Unit 19)	VAE	GAN	diffusion
trained by	next-token cross-entropy	the ELBO	a forger–detective game	squared error on the noise
draws with	one token at a time	one decoder pass	one generator pass	25 to 1 000 small steps
best at	text, code	smooth maps, compression	fast sharp pictures	the sharpest, most varied pictures; steering
its weak spot	slow for long outputs	blur	unstable, forgets kinds of data	many steps

The whole course, inside two machines. An LLM (Unit 19) and a diffusion model look nothing alike: one writes a word at a time, the other paints a whole picture out of static. Yet follow either one from start to end, and every stage is a tool you built in this course.

An LLM · Unit 19

a promptwords cut into tokens16

vectorsevery token is a list of numbers2 3 16

attention layersdot products pick what to look at; matrices mix1 3 18

a probability listsoftmax over every next word14

a dice rollthe next word, sampled14 19

A diffusion model · this unit

pure statica draw from

N(0,I)

the restorera network of matrices and attention guesses the noise1 5 18

the compassthe score: a gradient of

\log p

6 14

a small step + a jiggledescent with noise, small enough for Taylor8 9

a pictureoften decoded from a VAE's small map12 20

Both are trained the same way: a loss built from probability 14 · slopes by the chain rule and backprop 6 7 15 · downhill steps with Adam 9 10 11

Each numbered circle is a unit of this course (its ring has the colour of its part); tap one to go back to it. Every stage of both machines is built from tools you already own.

The same story, as a table you can read row by row:

what you learned	where it lives now
Vectors and dot products (Unit 3)	attention scores in the LLM; cross-attention from the picture to the prompt
Matrices and rank (Units 1, 5)	every layer; low-rank fine-tuning of both machines
Eigenvectors, SVD, PCA (Units 4, 5, 12)	the autoencoder's squeeze; the latent space of latent diffusion
Derivatives, the chain rule, backprop (Units 6, 7, 15)	training both; the reparameterisation trick; the score is a gradient
Taylor (Unit 8)	why tiny reverse steps are almost bell curves
Gradient descent, noise, Adam (Units 9, 11)	training both; and sampling itself — Langevin is descent with a jiggle
Probability, Gaussians, softmax, KL (Unit 14)	every loss: cross-entropy for the LLM, the ELBO's KL for the VAE, squared error (a Gaussian likelihood) for diffusion; sampling from both
Attention and transformers (Unit 18)	the LLM's whole body; the text reader and often the denoiser of a diffusion model

Rule of thumb. When you meet a new model, ask three questions: what noise goes in, what is it trained to predict, and how does it walk from noise to data? Every generator in this unit is an answer to those three.

Pause & predict

A friend says: "Diffusion models have nothing to do with gradient descent — they are trained differently and they sample differently." Which is the best reply?

In one sentence: A machine creates by learning to walk from noise to data — through a small map (the VAE) or down a thousand steps of fading static along a learned compass (diffusion) — and every step of that walk is built from the vectors, derivatives, descents and probabilities of this course.

Practice arena — sixteen problems, solved in full

Sixteen problems, in the order of the unit: making samples from noise, the Gaussian toolkit, the KL of a cloud, the trick and its slopes, a whole ELBO by hand, the forward jump and a schedule's half-life, the restorer, the compass for one and two villages, one step of DDPM, Langevin and DDIM, guidance, and the latent squeeze. The tags say which are easy and which are hard. Every number here was checked by machine.

Three habits do most of the work. Square roots first: write down $\sqrt{\bar\alpha}$ and $\sqrt{1-\bar\alpha}$ before anything else — almost every formula uses them. Variances add, spreads do not. And check the sign of every score: it must point towards the data.

Problem 1easysampling

A tea stall's daily sales follow $N(200,\ 15^2)$ cups. (a) Turn the standard draws $z=-2,\ 0.4,\ 1.5$ into sales. (b) Which $z$ would give a day of 245 cups?

What this tests. The stretch-and-shift generator of §1. Plan. $x=\mu+\sigma z$ forwards, then solve it for $z$ .

Show the full solution

Step 1 — (a).

200+15(-2)=170

200+15(0.4)=206

200+15(1.5)=222.5

Step 2 — (b).

z=(x-\mu)/\sigma=(245-200)/15=3

answers at a glance: (a) 170, 206, 222.5 cups. (b) $z=3$ .

Remember

The generator never changes; only the noise does. Running it backwards, $z=(x-\mu)/\sigma$ , is what an encoder does.

Problem 2easyinverse CDF

Waiting times follow an exponential curve with $\lambda=2$ : $F(x)=1-e^{-2x}$ . (a) Find the sampler $x=F^{-1}(u)$ . (b) Sample it at $u=0.75$ and $u=0.2$ . (c) Which $u$ gives the median wait, and what is it?

What this tests. The inverse-CDF trick of §1. Plan. Solve $u=1-e^{-2x}$ for $x$ .

Show the full solution

Step 1 — (a).

e^{-2x}=1-u

, so

x=-\tfrac12\ln(1-u)

Step 2 — (b).

u=0.75

-\tfrac12\ln0.25=\tfrac12\ln4\approx0.6931

u=0.2

-\tfrac12\ln0.8\approx0.1116

Step 3 — (c). The median is where half the waits are shorter:

u=0.5

x=\tfrac12\ln2\approx0.3466

answers at a glance: (a) $x=-\tfrac12\ln(1-u)$ . (b) 0.6931 and 0.1116. (c) $u=0.5$ , $x\approx0.3466$ .

Remember

Even noise in, any shape out: run the CDF backwards. The quantiles of the output are the quantiles of $u$ .

Problem 3mediumthe toolkit

One noising step is $x\to\sqrt{0.8}\,x+\sqrt{0.2}\,\varepsilon$ . (a) For the fixed start $x=-1$ , give the bell curve after one step (mean and spread). (b) If instead $x\sim N(0,4)$ , what is the variance after one step? (c) For independent $X$ , $Y$ with variances 1 and 2, find $\operatorname{Var}(3X-2Y)$ .

What this tests. The two facts of §2. Plan. Factors come out squared; independent variances add (a minus sign squares away).

Show the full solution

Step 1 — (a). Mean

\sqrt{0.8}(-1)\approx-0.8944

; spread

\sqrt{0.2}\approx0.4472

Step 2 — (b).

0.8\times4+0.2=3.4

: the cloud shrinks towards variance 1.

Step 3 — (c).

3^2(1)+(-2)^2(2)=9+8=17

answers at a glance: (a) $N(-0.8944,\ 0.4472^2)$ . (b) 3.4. (c) 17.

Remember

Only a start of variance exactly 1 stays at 1. Anything else drifts towards 1, step by step.

Problem 4mediumKL of a cloud

(a) Find $\operatorname{KL}\big(N(0.5,\ 0.8^2)\,\|\,N(0,1)\big)$ . (b) A cloud centred at $\mu=0$ pays a KL of 0.3181 and is narrower than the standard one. Find its spread $\sigma$ .

What this tests. The KL formula of §4, forwards and backwards. Plan. $\tfrac12(\mu^2+\sigma^2-1-\ln\sigma^2)$ ; for (b), try simple spreads below 1.

Show the full solution

Step 1 — (a).

\tfrac12(0.25+0.64-1-\ln0.64)=\tfrac12(-0.11+0.4463)\approx0.1681

Step 2 — (b). With

\mu=0

\sigma^2-1-\ln\sigma^2=2\times0.3181\approx0.636

. Try

\sigma=0.5

0.25-1-\ln0.25=-0.75+1.3863\approx0.636

. ✓

Step 3 — check. A wider cloud can pay the same price too (about

\sigma\approx1.608

): the KL punishes too narrow and too wide.

answers at a glance: (a) $\approx0.1681$ . (b) $\sigma=0.5$ .

Remember

The spread part $\sigma^2-1-\ln\sigma^2$ is 0 only at $\sigma=1$ and grows on both sides.

Problem 5easythe trick

An encoder gives $\mu=-0.5$ , $\sigma=2$ , and the wobble is $\varepsilon=0.3$ . The loss is $L=z^2$ . Find (a) $z$ ; (b) $\partial L/\partial\mu$ and $\partial L/\partial\sigma$ .

What this tests. The reparameterisation trick and the chain rule (§5). Plan. $z=\mu+\sigma\varepsilon$ ; then $\partial L/\partial z$ times 1 and times $\varepsilon$ .

Show the full solution

Step 1 — (a).

z=-0.5+2(0.3)=0.1

Step 2 — (b).

\partial L/\partial z=2z=0.2

\partial L/\partial\mu=0.2\times1=0.2

;

\partial L/\partial\sigma=0.2\times0.3=0.06

answers at a glance: (a) $z=0.1$ . (b) 0.2 and 0.06.

Remember

The slope to $\sigma$ is the slope to $z$ times the wobble: a draw that landed on the other side ( $\varepsilon<0$ ) would push $\sigma$ the other way.

Problem 6harda whole ELBO

Two codes A and B, each with prior 0.5. $p(x\mid A)=0.9$ , $p(x\mid B)=0.1$ . (a) Find $p(x)$ , $\log p(x)$ and the true posterior. (b) For $q=(0.5,\ 0.5)$ , find the rebuild, the KL to the prior, the ELBO and the gap. (c) The same for $q=(0.7,\ 0.3)$ . (d) Check that the gap in (c) equals $\operatorname{KL}(q\,\|\,\text{posterior})$ .

What this tests. §6 end to end. Plan. rebuild $=\sum q\ln p(x\mid z)$ , KL $=\sum q\ln(q/\text{prior})$ , ELBO = rebuild − KL, gap $=\log p(x)-$ ELBO.

Show the full solution

Step 1 — (a).

p(x)=0.45+0.05=0.5

;

\log p(x)=\ln0.5\approx-0.6931

; posterior

(0.45/0.5,\ 0.05/0.5)=(0.9,\ 0.1)

Step 2 — (b). rebuild

=0.5\ln0.9+0.5\ln0.1\approx-1.2040

; KL

=0

; ELBO

\approx-1.2040

; gap

\approx-0.6931+1.2040=0.5108

Step 3 — (c). rebuild

=0.7\ln0.9+0.3\ln0.1\approx-0.0738-0.6908=-0.7645

. KL

=0.7\ln1.4+0.3\ln0.6\approx0.2355-0.1532=0.0823

. ELBO

\approx-0.8468

; gap

\approx0.1537

Step 4 — (d).

0.7\ln\frac{0.7}{0.9}+0.3\ln\frac{0.3}{0.1}\approx-0.1759+0.3296=0.1537

. ✓

answers at a glance: (a) 0.5, −0.6931, (0.9, 0.1). (b) −1.2040, 0, −1.2040, gap 0.5108. (c) −0.7645, 0.0823, −0.8468, gap 0.1537. (d) equal.

Remember

The closer the guess is to the posterior, the smaller the gap — here (0.9, 0.1) would close it completely, with ELBO $=-0.6931$ .

Problem 7easyone jump

A short schedule has $\beta=(0.1,\ 0.2,\ 0.5)$ . (a) Find $\bar\alpha_3$ . (b) Write $x_3$ as one jump from $x_0$ . (c) Evaluate it for $x_0=1$ , $\varepsilon=-0.5$ .

What this tests. The closed form of §8. Plan. Multiply the keep factors $1-\beta$ ; take square roots.

Show the full solution

Step 1 — (a).

0.9\times0.8\times0.5=0.36

Step 2 — (b).

x_3=\sqrt{0.36}\,x_0+\sqrt{0.64}\,\varepsilon=0.6x_0+0.8\varepsilon

Step 3 — (c).

0.6(1)+0.8(-0.5)=0.2

answers at a glance: (a) 0.36. (b) $0.6x_0+0.8\varepsilon$ . (c) 0.2.

Remember

The two weights always satisfy $0.6^2+0.8^2=1$ : the glass stays full.

Problem 8harda schedule's half-life

A schedule uses the same $\beta$ at every step. (a) Find a formula for the first step $t$ at which $\bar\alpha_t$ drops below a level $a$ . (b) Apply it with $\beta=0.01$ , $a=0.5$ . (c) And with $a=0.1$ .

What this tests. $\bar\alpha_t=(1-\beta)^t$ , turned around with logs. Plan. Solve $(1-\beta)^t<a$ ; both logs are negative, so the inequality flips.

Show the full solution

Step 1 — (a).

t\ln(1-\beta)<\ln a

. Divide by the negative

\ln(1-\beta)

t>\ln a/\ln(1-\beta)

. The first whole step is the next whole number above

\ln a/\ln(1-\beta)

Step 2 — (b).

\ln0.5/\ln0.99\approx68.97

, so

t=69

. Check:

0.99^{68}\approx0.5049

0.99^{69}\approx0.4998

. ✓

Step 3 — (c).

\ln0.1/\ln0.99\approx229.1

, so

t=230

. Check:

0.99^{229}\approx0.1001

0.99^{230}\approx0.0991

. ✓

answers at a glance: (a) the first whole $t>\ln a/\ln(1-\beta)$ . (b) 69. (c) 230.

Remember

A constant $\beta$ is a steady decay: every 69 steps it halves what is left. DDPM's $\beta$ is not constant — it starts tiny and grows — so its halfway point (step 260) has to be counted by multiplying the factors one by one.

Problem 9easythe restorer

At $\bar\alpha=0.64$ the restorer sees $x_t=0.5$ and guesses $\hat\varepsilon=-1$ . (a) Find $\hat x_0$ . (b) If the true clean value was 1.4, what noise was really added, and what is the training loss?

What this tests. $\hat x_0$ from $\hat\varepsilon$ , and the other way round (§9). Plan. $\sqrt{\bar\alpha}=0.8$ , $\sqrt{1-\bar\alpha}=0.6$ .

Show the full solution

Step 1 — (a).

\hat x_0=(0.5-0.6(-1))/0.8=1.1/0.8=1.375

Step 2 — (b).

\varepsilon=(x_t-0.8x_0)/0.6=(0.5-1.12)/0.6\approx-1.0333

. Loss

(\varepsilon-\hat\varepsilon)^2=(-1.0333+1)^2\approx0.0011

answers at a glance: (a) 1.375. (b) $\varepsilon\approx-1.0333$ , loss $\approx0.0011$ .

Remember

A small error in $\hat\varepsilon$ becomes an error in $\hat x_0$ multiplied by $\sqrt{1-\bar\alpha}/\sqrt{\bar\alpha}$ : here 0.75 times as big.

Problem 10mediumthe compass

(a) For $N(-1,\ 0.5)$ (variance 0.5), find the score at $x=0$ and at $x=-2$ . (b) A single clean point $x_0=3$ is noised to $\bar\alpha=0.25$ , and you see $x_t=1.2$ . Find the score there, and the noise guess $\hat\varepsilon$ it corresponds to.

What this tests. The score of a bell, and score $\leftrightarrow\hat\varepsilon$ (§10). Plan. $-(x-\mu)/\sigma^2$ ; the noisy point is $N(\sqrt{\bar\alpha}x_0,\ 1-\bar\alpha)$ .

Show the full solution

Step 1 — (a).

x=0

-(0+1)/0.5=-2

(go left).

x=-2

-(-2+1)/0.5=+2

(go right).

Step 2 — (b). Centre

0.5\times3=1.5

, variance

0.75

: score

=-(1.2-1.5)/0.75=0.4

Step 3 — (b).

\hat\varepsilon=-\sqrt{1-\bar\alpha}\cdot\text{score}=-\sqrt{0.75}\times0.4\approx-0.3464

answers at a glance: (a) −2 and +2. (b) score 0.4, $\hat\varepsilon\approx-0.3464$ .

Remember

Score and noise guess always have opposite signs: the noise pushed the point away; the compass points back.

Problem 11hardtwo villages

Data: half the points at $-1$ , half at $+1$ , each with variance 0.04. It is noised to $\bar\alpha=0.36$ . (a) Where are the two noisy villages, and what is their variance? (b) At $x=0.2$ , find the two claims. (c) Find the score at $x=0.2$ . Which way does it point, and why?

What this tests. The mixture compass of §10. Plan. Centres $\pm\sqrt{\bar\alpha}$ , variance $\bar\alpha s^2+1-\bar\alpha$ ; claims from the two bells; score = claims × pulls.

Show the full solution

Step 1 — (a). Centres

\pm0.6

; variance

0.36(0.04)+0.64=0.6544

Step 2 — (b). Exponents:

-(0.2+0.6)^2/(2\times0.6544)\approx-0.4890

and

-(0.2-0.6)^2/(2\times0.6544)\approx-0.1222

e^{-0.4890}\approx0.6132

e^{-0.1222}\approx0.8849

. Claims

\approx(0.4093,\ 0.5907)

Step 3 — (c). Pulls

-(0.2+0.6)/0.6544\approx-1.2225

and

-(0.2-0.6)/0.6544\approx0.6112

. Score

\approx0.4093(-1.2225)+0.5907(0.6112)\approx-0.1394

Step 4 — why. It points gently left, towards the middle — even though the right village claims more. At this much noise the two villages overlap into one soft hill, and 0.2 is just to the right of its top.

answers at a glance: (a) $\pm0.6$ , 0.6544. (b) $(0.4093,\ 0.5907)$ . (c) $\approx-0.1394$ , gently to the left.

Remember

The nearer village does not always win the compass: a strong pull from a far village, weighted by its claim, can outweigh a weak pull from a near one.

Problem 12mediumone DDPM step

A DDPM step with $\beta_t=0.2$ , $\bar\alpha_t=0.64$ , $x_t=0.8$ , $\hat\varepsilon=0.3$ and fresh draw $z=0.5$ . Find (a) the mean of the step, (b) $\sigma_t$ , (c) $x_{t-1}$ .

What this tests. DDPM's step (§11). Plan. $\alpha_t=0.8$ , $\sqrt{1-\bar\alpha_t}=0.6$ , $\sigma_t=\sqrt{\beta_t}$ .

Show the full solution

Step 1 — (a).

\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\hat\varepsilon=\frac{0.2}{0.6}(0.3)=0.1

. Mean

=(0.8-0.1)/\sqrt{0.8}\approx0.7/0.8944\approx0.7826

Step 2 — (b).

\sigma_t=\sqrt{0.2}\approx0.4472

Step 3 — (c).

0.7826+0.4472(0.5)\approx1.0062

answers at a glance: (a) $\approx0.7826$ . (b) $\approx0.4472$ . (c) $\approx1.0062$ .

Remember

Remove only this step's share of the noise, undo this step's shrink, then jiggle. Never remove all of $\hat\varepsilon$ in one DDPM step.

Problem 13easyLangevin

Target $p=N(1,\ 0.25)$ . One Langevin step from $x=2$ with $\eta=0.05$ and jiggle $z=-1$ . Where do you land?

What this tests. $x\leftarrow x+\eta\,\text{score}+\sqrt{2\eta}\,z$ (§11). Plan. The score is $-(x-1)/0.25$ .

Show the full solution

Step 1 — score.

-(2-1)/0.25=-4

Step 2 — step.

2+0.05(-4)=1.8

; jiggle

\sqrt{0.1}(-1)\approx-0.3162

; land at

\approx1.4838

answer at a glance: $x\approx1.4838$ .

Remember

The downhill part pulls towards the peak at 1; the jiggle, of size $\sqrt{2\eta}$ , keeps the walkers spread with the right width.

Problem 14mediumone DDIM jump

At $\bar\alpha_t=0.36$ a walker is at $x_t=0.2$ , and the restorer guesses $\hat\varepsilon=-0.5$ . (a) Find $\hat x_0$ . (b) Jump with DDIM to $\bar\alpha_s=0.81$ . (c) Where would a jump to $\bar\alpha_s=1$ land?

What this tests. The two moves of DDIM (§12). Plan. $\sqrt{0.36}=0.6$ , $\sqrt{0.64}=0.8$ , $\sqrt{0.81}=0.9$ , $\sqrt{0.19}\approx0.4359$ .

Show the full solution

Step 1 — (a).

\hat x_0=(0.2-0.8(-0.5))/0.6=0.6/0.6=1

Step 2 — (b).

x_s=0.9(1)+0.4359(-0.5)\approx0.6821

Step 3 — (c). At

\bar\alpha_s=1

the noise weight is 0:

x_s=\hat x_0=1

answers at a glance: (a) 1. (b) $\approx0.6821$ . (c) 1.

Remember

DDIM re-uses the same $\hat\varepsilon$ : no new randomness, so the same start always gives the same picture.

Problem 15mediumguidance

For one pixel pair, the unconditional guess is $\varepsilon_u=(0.1,\ -0.2)$ and the conditional guess is $\varepsilon_c=(0.4,\ 0.2)$ . (a) Find the guided guess for $w=2.5$ . (b) Which $w$ makes its first entry 1.3?

What this tests. Classifier-free guidance (§13), forwards and backwards. Plan. $\varepsilon_u+w(\varepsilon_c-\varepsilon_u)$ , entry by entry.

Show the full solution

Step 1 — the difference.

\varepsilon_c-\varepsilon_u=(0.3,\ 0.4)

Step 2 — (a).

(0.1+2.5(0.3),\ -0.2+2.5(0.4))=(0.85,\ 0.8)

Step 3 — (b).

0.1+0.3w=1.3\Rightarrow w=4

answers at a glance: (a) $(0.85,\ 0.8)$ . (b) $w=4$ .

Remember

$w=1$ gives exactly $\varepsilon_c$ ; every extra unit of $w$ adds one more copy of the difference the prompt makes.

Problem 16easythe latent squeeze

A 1024 × 1024 colour picture is squeezed by an autoencoder into a 128 × 128 × 4 latent. (a) How many numbers before and after, and by what factor? (b) Show that squeezing each side by a factor $f$ into $c$ channels shrinks a colour picture by $3f^2/c$ , and apply it to $f=16$ , $c=16$ .

What this tests. Latent diffusion's saving (§13), as a general formula. Plan. Count numbers; each side shrinks by $f$ , channels go from 3 to $c$ .

Show the full solution

Step 1 — (a).

1024\times1024\times3=3\,145\,728

;

128\times128\times4=65\,536

; factor 48.

Step 2 — (b).

\dfrac{H\cdot W\cdot3}{(H/f)(W/f)\,c}=\dfrac{3f^2}{c}

. With

f=8,\ c=4

: 48, as in (a). With

f=16,\ c=16

3\times256/16=48

again.

answers at a glance: (a) 3 145 728 → 65 536, 48 times. (b) $3f^2/c$ ; 48.

Remember

The diffusion network runs on the small latent at every one of its steps; the big decoder runs only once, at the end.

The route is complete · 20 of 20

You have walked the whole way — from a line on a page to a machine that draws.

You started with equations that meet at a point. You learned to see matrices as machines, to find the directions they leave alone, to follow a slope downhill, to count with probabilities. Then those tools grew into networks that read, remember, pay attention, write — and now draw from noise.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

See the whole route →Big ideas: the course on one page

← Unit 19 · The Maths Inside an LLM · All units