Unit 15 · The Network, Whole — The Math Behind the Machine

A network is a stack of simple steps

Imagine this

Think of a line of stations in a factory. A part comes in. The first station does one small job and passes it on. The next station does another small job. No station is clever. But at the end of the line, a whole scooter rolls out.

A neural network is the same kind of line. Each station is simple. The power comes from putting many of them in a row.

Over fourteen units we collected the parts. A matrix is a machine that moves space (Unit 1). The chain rule multiplies slopes along a chain (Unit 6). Blame can flow backwards through a graph (Unit 7). A step downhill makes a guess better (Unit 9). Softmax turns scores into probabilities (Unit 14). Now we bolt them together.

Here is the one station a network is built from. We call it a layer, and it does three things in order:

Multiply by a matrix. Mix the incoming numbers, each with its own weight.
Add a shift. Add one extra number to each result. This is the bias.
Bend. Pass each result through a simple bent function, like "keep it if positive, else make it zero". This is the activation.

In symbols, layer number $l$ takes the list $\mathbf a^{(l-1)}$ that arrives and makes two new lists:

$\begin{gathered}\mathbf z^{(l)}=W^{(l)}\mathbf a^{(l-1)}+\mathbf b^{(l)},\\ \mathbf a^{(l)}=\operatorname{ReLU}\big(\mathbf z^{(l)}\big).\end{gathered}$

The list $\mathbf z$ is the mix before the bend. The list $\mathbf a$ is what comes out after the bend. The superscript $(l)$ just says which layer. That is all a layer is.

Meet the salary network. We will use one small network for the whole unit. It guesses a person's salary (in lakhs) from two facts: their age and their years of experience. The two facts go in. A layer of 3 neurons works on them. A layer of 2 neurons works on that. One neuron at the end gives the guess, $\hat y$ . We write this shape as 2 → 3 → 2 → 1.

How big is each matrix? There is one simple rule. A weight joins one neuron in this layer to one neuron in the layer before. So the matrix has one row for every neuron here and one column for every neuron before:

$W^{(l)}\ \text{is}\ (\text{neurons in layer } l)\times(\text{neurons in layer } l-1).$

So $W^{(1)}$ is $3\times2$ , $W^{(2)}$ is $2\times3$ and $W^{(3)}$ is $1\times2$ . Each bias has one entry per neuron: $3\times1$ , $2\times1$ , $1\times1$ . Add them up: $6+3+6+2+2+1=20$ numbers. Those 20 numbers are everything the network knows. Training means changing them.

Try: Point at the middle bundle of wires in the graph and watch $W^{(2)}$ light up. Then switch to the chain and point at $\mathbf z^{(1)}$ . Last, turn on the shape quiz and click each "?" until the shape is right.

Point at a wire, a neuron or a box.

The realization

$\begin{gathered}\mathbf z^{(l)}=W^{(l)}\mathbf a^{(l-1)}+\mathbf b^{(l)}\\ \mathbf a^{(l)}=\operatorname{ReLU}\big(\mathbf z^{(l)}\big)\end{gathered}$

A network is not a mystery box. It is a chain of simple steps: multiply, shift, bend — again and again. The graph, the matrices and the chain are three drawings of the same thing. The rows of each matrix belong to where the wires go; the columns to where they come from.

Pause & predict

A network has shape 4 → 5 → 3. What is the shape of $W^{(2)}$ , and how many numbers (weights and biases) does the whole network have?

The road ahead. The unit has four acts.

What a network is made of (§1–§4): one neuron is a line and a bend; without the bend, layers collapse; with it, enough folds can draw any shape.
The forward pass (§5–§6): push age 30 and experience 10 through the salary network by hand, then score the answer.
The backward pass (§7–§9): two rules carry the blame back to every one of the 20 numbers; then one step makes the guess better.
Build your own (§10–§12): when blame fades or dies, and a playground where you build a network and watch it learn.

In one sentence: A network is a line of simple layers, and each layer only multiplies by a matrix, adds a shift, and bends — the matrix having one row per neuron here and one column per neuron before.

One neuron is a line and a bend

Imagine this

Take a flat sheet of paper and fold it once along a straight crease. One side stays flat on the table. The other side lifts up like a ramp.

That fold is exactly what one neuron does to its inputs. The crease is a straight line. The bend happens along it.

Look at a single neuron with two inputs $x_1, x_2$ . First it mixes them: $z=w_1x_1+w_2x_2+b$ . Where is $z$ exactly zero? On the straight line $w_1x_1+w_2x_2+b=0$ . On one side of that line $z$ is positive, on the other side negative. This is the same line you met in Unit 13 — a support vector machine's line, just without the street around it.

A tiny example. Take $w_1=1$ , $w_2=1$ , $b=-2$ . At the point $(3,1)$ : $z=3+1-2=2$ . At $(0,0)$ : $z=-2$ . At $(1,1)$ : $z=0$ — right on the line.

Then the neuron bends. It passes $z$ through an activation. There are four common choices:

ReLU, $\max(0,z)$ : keep a positive number, turn a negative one into zero. It is a hinge. With it, the flat plane folds along the line: flat zero on one side, a straight ramp on the other. Our example gives $\operatorname{ReLU}(2)=2$ and $\operatorname{ReLU}(-2)=0$ .
Sigmoid, $1/(1+e^{-z})$ : a smooth step from 0 to 1. It reads like a probability (Unit 14). $\sigma(0)=0.5$ on the line.
tanh: a smooth step from −1 to 1.
None: leave $z$ as it is. No bend at all — just a tilted flat sheet.

The bend has a slope, and the slope will matter a lot on the way back. ReLU's slope is simple: 1 on the awake side, 0 on the sleeping side. Sigmoid's slope is never more than 0.25. Keep those two numbers in mind.

Try: With ReLU, turn $b$ and watch the crease slide without turning. Turn $w_1$ and watch it swing. Then switch to sigmoid and to "none". Last, press show the slope: the sheet is coloured by how steep the bend is — 0 or 1 for ReLU, never more than 0.25 for sigmoid.

drag the picture to orbit

w_1

w_2

b

−2

Here are the four bends side by side, each with its slope underneath. Slide the probe along $z$ and read both numbers.

The realization

$a=\operatorname{act}(w_1x_1+w_2x_2+b)$

One neuron draws one straight line across its inputs — the place where $z=0$ — and then bends the flat sheet along it. ReLU folds it like paper. Sigmoid and tanh make a smooth step. "None" does not bend at all. The weights turn the line; the bias slides it.

Pause & predict

A ReLU neuron has $w=(2,-1)$ and $b=1$ . What does it output at the input $(1,4)$ , and what is its slope $\operatorname{ReLU}'(z)$ there?

Pause & predict

You double both weights and the bias of a neuron: $w\to2w$ , $b\to2b$ . What happens to its crease line $z=0$ ?

If you want the algebra · 1 proof, step by step

Prove it · sigmoid's slope is never more than ¼

Claim. For $\sigma(z)=1/(1+e^{-z})$ , the slope is $\sigma'(z)=\sigma(z)\,(1-\sigma(z))$ , and it is at most $\tfrac14$ , reached only at $z=0$ .

Write

\sigma=(1+e^{-z})^{-1}

and use the chain rule:

\begin{aligned}\sigma'(z)&=\frac{e^{-z}}{(1+e^{-z})^2}\\ &=\frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}}.\end{aligned}

The derivative of

u^{-1}

-u^{-2}u'

, and

u'=-e^{-z}

The second factor is

1-\sigma

, because

\dfrac{e^{-z}}{1+e^{-z}}=1-\dfrac{1}{1+e^{-z}}

. So

\sigma'=\sigma(1-\sigma)

. A handy form: the slope comes straight from the value you already stored on the way forward.

Put

s=\sigma\in(0,1)

. Then

s(1-s)=\tfrac14-(s-\tfrac12)^2\le\tfrac14

, with equality only at

s=\tfrac12

, that is

z=0

. ∎ Example: at

z=2

\sigma\approx0.881

and the slope is

0.881\times0.119\approx0.105

In one sentence: A neuron draws one straight line $w\cdot x+b=0$ and bends the plane along it — a hinge with slope 0 or 1 for ReLU, a smooth step with slope at most 0.25 for sigmoid.

Why the bend matters: straight layers collapse

Imagine this

You photocopy a photo at 150%. Then you photocopy the copy at 80%. You could have done it in one go at 120%.

Two straight "stretch and shift" steps in a row are always one straight step. Stack a hundred of them and you still have just one. Depth without bends buys you nothing.

Let us check it with small numbers. Take two layers with no bend:

$\begin{gathered}W_1=\begin{pmatrix}1&2\\ 0&1\end{pmatrix},\ \mathbf b_1=\begin{pmatrix}1\\ 0\end{pmatrix},\\ W_2=\begin{pmatrix}1&-1\\ 2&1\end{pmatrix},\ \mathbf b_2=\begin{pmatrix}0\\ 1\end{pmatrix}.\end{gathered}$

Push $\mathbf x=(1,1)$ through. Layer 1 gives $W_1\mathbf x+\mathbf b_1=(1+2+1,\ 0+1+0)=(4,1)$ . Layer 2 gives $W_2(4,1)+\mathbf b_2=(4-1+0,\ 8+1+1)=(3,10)$ .

Now do it in one step. Multiply the matrices once: $W_2W_1=\begin{pmatrix}1&1\\ 2&5\end{pmatrix}$ . Push the first shift through the second matrix and add the second shift: $W_2\mathbf b_1+\mathbf b_2=(1,3)$ . So

$\begin{aligned}&W_2(W_1\mathbf x+\mathbf b_1)+\mathbf b_2\\ &\quad=\begin{pmatrix}1&1\\ 2&5\end{pmatrix}\mathbf x+\begin{pmatrix}1\\ 3\end{pmatrix}.\end{aligned}$

Check at $(1,1)$ : $(1+1+1,\ 2+5+3)=(3,10)$ . The same answer, from one layer. In Unit 1 words: a matrix moves space so that straight grid lines stay straight and parallel. Two such moves in a row are still such a move.

So why do deep networks work? Because of the bend. Put a ReLU between the two layers, and the grid gets folded. A fold is something no single matrix can do. Try it below.

Try: On the worked example, pick none and compare the last page with the ghost page "one matrix": they match exactly. Pick ReLU and watch the grid fold. Then load XOR: with none no straight line can split the blue and orange groups; with ReLU the fold stacks two blue groups on top of each other and one straight cut does it.

drag the picture to orbit

—

The realization

$\begin{aligned}&W_2(W_1\mathbf x+\mathbf b_1)+\mathbf b_2\\ &\quad=(W_2W_1)\,\mathbf x+(W_2\mathbf b_1+\mathbf b_2)\end{aligned}$

A straight step after a straight step is one straight step. Without bends, a network of any depth is just one matrix and one shift, and can only ever draw straight boundaries. The bend between layers is what lets depth fold space — and folding is what untangles data.

Pause & predict

A network has 50 layers, each 10 → 10, and no activation anywhere. What single thing can replace the whole network?

If you want the algebra · 2 proofs, step by step

Prove it · a straight step after a straight step is one straight step

Claim. If $f(\mathbf x)=W_1\mathbf x+\mathbf b_1$ and $g(\mathbf u)=W_2\mathbf u+\mathbf b_2$ , then $g(f(\mathbf x))=W\mathbf x+\mathbf b$ with $W=W_2W_1$ and $\mathbf b=W_2\mathbf b_1+\mathbf b_2$ .

Substitute

f

into

g

\begin{aligned}&g(f(\mathbf x))\\ &=W_2(W_1\mathbf x+\mathbf b_1)+\mathbf b_2.\end{aligned}

The output of the first layer is the input of the second.

Multiplying by a matrix spreads over a sum (it is a linear machine, Unit 1):

\begin{aligned}&W_2(W_1\mathbf x+\mathbf b_1)+\mathbf b_2\\ &=W_2W_1\mathbf x+W_2\mathbf b_1+\mathbf b_2\\ &=(W_2W_1)\mathbf x\\ &\quad+(W_2\mathbf b_1+\mathbf b_2).\end{aligned}

∎ With the numbers of §3:

W_2W_1=\begin{pmatrix}1&1\\ 2&5\end{pmatrix}

and

W_2\mathbf b_1+\mathbf b_2=(1,3)

Prove it · a hundred straight layers are still one

Claim. Any stack of $n$ layers without activations, $\mathbf x\mapsto W_n(\cdots(W_1\mathbf x+\mathbf b_1)\cdots)+\mathbf b_n$ , equals one layer $\mathbf x\mapsto M\mathbf x+\mathbf c$ .

For

n=1

it is already one layer. Suppose the first

n-1

layers equal

M'\mathbf x+\mathbf c'

. This is a proof by walking up one layer at a time (induction).

Add layer

n

on top and use the previous proof:

\begin{aligned}&W_n(M'\mathbf x+\mathbf c')+\mathbf b_n\\ &=(W_nM')\,\mathbf x\\ &\quad+(W_n\mathbf c'+\mathbf b_n).\end{aligned}

M=W_n\cdots W_2W_1

and the shifts collect into one vector

\mathbf c

. ∎ The consequence: without bends, every boundary the network can draw is a straight line (a flat plane, in more dimensions) — exactly what one layer could draw.

In one sentence: Two straight layers are one straight layer — $W_2W_1$ and $W_2\mathbf b_1+\mathbf b_2$ — so without a bend depth is wasted, and the bend is what lets a network fold space.

Fold by fold: a network can draw any shape

Imagine this

You want to bend a straight piece of wire into the shape of a hill. One bend makes a corner. Three bends make a little tent. With enough bends, placed well, the wire can follow any curve you like — a hill, a valley, a wave.

Each ReLU neuron is one bend in the wire.

Work in one dimension, with one input $x$ . One ReLU neuron, $\operatorname{ReLU}(x-c)$ , is flat until $x=c$ and then rises. It is one crease at $c$ .

Now add three of them with weights $1,-2,1$ :

$\begin{aligned}h(x)=\ &\operatorname{ReLU}(x)-2\operatorname{ReLU}(x-1)\\ &+\operatorname{ReLU}(x-2).\end{aligned}$

Read it off at a few points. $h(0)=0$ . $h(1)=1-0+0=1$ . $h(2)=2-2+0=0$ . And for $x\ge2$ : $x-2(x-1)+(x-2)=0$ . So $h$ is a hat (a tent): flat, up to 1, down to 0, flat again. Three neurons, one bump.

Now the big idea. Any curve can be traced by little straight pieces. Every straight piece needs a crease at each end. So enough hats, added up, can trace any curve as closely as you like. This is the universal approximation result, said plainly: one hidden layer with enough neurons can get as close as you want to any smooth shape.

Be honest about what it does not say. It does not say how many neurons you need. It does not say how to find the weights. Finding them is the job of training (§7–§9).

Try: Start with $N=1$ on the sine and add neurons one by one — watch the error fall. Pick the step: it needs many creases near the jump. Pick draw your own and drag across the plot to sketch any curve.

neurons

N

Wide or deep? One layer adds creases side by side: $N$ neurons give about $N$ straight pieces. A second layer does something stronger. It takes the already folded paper and folds it again. Fold a strip in half: 2 pieces. Fold the folded strip in half: 4. Again: 8. After $k$ folds you have $2^k$ pieces. Depth multiplies; width only adds.

Try: Press ▶ fold again and count the pieces on each new ribbon: 2, 4, 8, 16. Then compare the two neuron counts in the readout.

drag the picture to orbit

layers

k

The realization

$\begin{aligned}&\operatorname{ReLU}(x)-2\operatorname{ReLU}(x-1)\\ &\quad+\operatorname{ReLU}(x-2)=\text{a hat}\end{aligned}$

One ReLU is one crease. Three make a hat. Enough hats trace any shape — that is universal approximation, and it promises closeness, not a recipe. Width adds creases one at a time; depth folds the folds, so $k$ layers can make $2^k$ pieces.

Pause & predict

For the hat $h(x)=\operatorname{ReLU}(x)-2\operatorname{ReLU}(x-1)+\operatorname{ReLU}(x-2)$ , what is $h(1.5)$ ?

Pause & predict

A strip is folded by 6 layers of tent folds, one after another. How many straight pieces does it have?

If you want the algebra · 2 proofs, step by step

Prove it · three ReLUs make a hat

Claim. $h(x)=\operatorname{ReLU}(x)-2\operatorname{ReLU}(x-1)+\operatorname{ReLU}(x-2)$ equals $0$ for $x\le0$ , $x$ on $[0,1]$ , $2-x$ on $[1,2]$ , and $0$ for $x\ge2$ .

Each ReLU is

0

to the left of its crease and the straight line

x-c

to the right. So just count which creases are behind you. The creases sit at 0, 1 and 2.

Go piece by piece:

\begin{aligned}x\le0:&\quad 0\\ 0\le x\le1:&\quad x\\ 1\le x\le2:&\quad 2-x\\ x\ge2:&\quad 0\end{aligned}

∎ For example, on

[1,2]

x-2(x-1)=2-x

; and for

x\ge2

x-2(x-1)+(x-2)=0

. The weights

1,-2,1

are the changes of slope at each crease: slope goes

0\to1

, then

1\to-1

(a change of

-2

), then

-1\to0

Prove it · k tent folds make 2ᵏ pieces

Claim. Let $t(x)=1-|2x-1|$ on $[0,1]$ (one tent, made from two ReLUs: $t(x)=2\operatorname{ReLU}(x)-4\operatorname{ReLU}(x-\tfrac12)$ ). Then $t$ applied $k$ times, $t^{(k)}$ , has exactly $2^k$ straight pieces, and each piece runs all the way from 0 to 1 or from 1 to 0.

For

k=1

t

rises from 0 to 1 on

[0,\tfrac12]

and falls back on

[\tfrac12,1]

. Two pieces, each covering all of

[0,1]

. Check the ReLU form: for

x\le\tfrac12

it is

2x

; for

x\ge\tfrac12

it is

2x-4(x-\tfrac12)=2-2x

Suppose

t^{(k-1)}

has

2^{k-1}

pieces, each sweeping across all of

[0,1]

. Applying

t

on top of one such sweep folds it once: the sweep passes

\tfrac12

exactly once, so it becomes 2 pieces, each again sweeping all of

[0,1]

. A tent turns any full sweep into an up-and-down.

So every piece doubles:

t^{(k)}

has

2\cdot2^{k-1}=2^k

pieces. ∎ Cost:

2k

ReLUs in

k

layers. A single layer makes at most one new crease per neuron, so it needs about

2^k

neurons for the same zig-zag. Depth is exponentially cheaper here.

In one sentence: Each ReLU adds one crease, three make a hat, enough hats trace any curve — and stacking layers folds the folds, doubling the pieces with every layer.

The forward pass, by hand

Imagine this

A dabbawala relay in Mumbai. The tiffin is picked up at home, handed to a runner, put on a train, passed to another runner, and delivered at the office. At every hand-over, someone writes a note in the register: "received at 10:05, passed on at 10:12".

The forward pass is that relay. Numbers are handed from layer to layer. And, just like the register, we write down every hand-over. We will need those notes on the way back.

Here are the starting numbers of the salary network. Everything is in lakhs.

$\begin{aligned}W^{(1)}&=\begin{pmatrix}0.1&0.3\\ 0.2&-0.1\\ -0.1&0.2\end{pmatrix},\ \mathbf b^{(1)}=\begin{pmatrix}1\\ -1\\ -1\end{pmatrix},\\ W^{(2)}&=\begin{pmatrix}1&-0.5&1\\ 1&0.5&-1\end{pmatrix},\ \mathbf b^{(2)}=\begin{pmatrix}1\\ -2\end{pmatrix},\\ W^{(3)}&=\begin{pmatrix}4&2\end{pmatrix},\ b^{(3)}=2.\end{aligned}$

The input is $\mathbf a^{(0)}=(30,10)$ : age 30, experience 10. The true salary is $y=50$ . Now walk the chain from left to right. Each row of a matrix times the input column is a dot product (Unit 3): multiply matching entries and add.

F1 · into hidden layer 1 ( $\mathbf z^{(1)}=W^{(1)}\mathbf a^{(0)}+\mathbf b^{(1)}$ ). One row per neuron:

$\begin{aligned}\text{row 1: }&(0.1)(30)+(0.3)(10)+1=3+3+1=7\\ \text{row 2: }&(0.2)(30)+(-0.1)(10)-1=6-1-1=4\\ \text{row 3: }&(-0.1)(30)+(0.2)(10)-1=-3+2-1=-2\end{aligned}$

So $\mathbf z^{(1)}=(7,4,-2)$ . Its shape is $3\times1$ ✓ — one number per neuron.

F2 · the bend. $\mathbf a^{(1)}=\operatorname{ReLU}(7,4,-2)=(7,4,0)$ . The third neuron got a negative number, so ReLU switched it off. It is asleep for this person. Keep an eye on it.

F3 · into hidden layer 2.

$\begin{aligned}\text{row 1: }&(1)(7)+(-0.5)(4)+(1)(0)+1=7-2+0+1=6\\ \text{row 2: }&(1)(7)+(0.5)(4)+(-1)(0)-2=7+2-0-2=7\end{aligned}$

F4 · the bend. Both are positive, so $\mathbf a^{(2)}=(6,7)$ . ReLU changes nothing here.

F5 · into the output. $z^{(3)}=(4)(6)+(2)(7)+2=24+14+2=40$ .

F6 · the last bend. $\hat y=\operatorname{ReLU}(40)=40$ . The network predicts 40 lakhs.

F7 · the score. $L=\tfrac12(\hat y-y)^2=\tfrac12(40-50)^2=\tfrac12(100)=50$ . The error is $\hat y-y=-10$ : the guess is 10 lakhs too low.

Keep all of these

The backward pass will ask for $\mathbf a^{(0)},\mathbf a^{(1)},\mathbf a^{(2)}$ and for $\mathbf z^{(1)},\mathbf z^{(2)},\mathbf z^{(3)}$ . Write them down now — this list is called the cache — and nothing has to be computed twice.

$\mathbf a^{(0)}$	$\mathbf z^{(1)}$	$\mathbf a^{(1)}$	$\mathbf z^{(2)}$	$\mathbf a^{(2)}$	$z^{(3)}$	$\hat y$	$L$
(30, 10)	(7, 4, −2)	(7, 4, 0)	(6, 7)	(6, 7)	40	40	50

Try: Press ▶ play the pass and follow the gold row across the input. Then slide the experience up to 25: the third neuron wakes up (its light turns gold). Last, set age 18 and experience 30: now the second neuron falls asleep.

age30

experience10

The realization

$\begin{gathered}\mathbf a^{(0)}\to\mathbf z^{(1)}\to\mathbf a^{(1)}\to\mathbf z^{(2)}\\ \to\mathbf a^{(2)}\to z^{(3)}\to\hat y\to L\end{gathered}$

The forward pass is just row times column, add the shift, bend — layer after layer, left to right. Each row is one neuron's dot product. A neuron that gets a negative total is switched off by ReLU. And every value along the way goes into the cache, because the backward pass will read it.

Pause & predict

Keep age 30, but make the experience 20 instead of 10. What is $\mathbf z^{(1)}$ now, and which hidden-1 neuron is asleep?

Pause & predict

Why do we keep $\mathbf z^{(1)},\mathbf a^{(1)},\dots$ after the forward pass is done?

In one sentence: The forward pass walks the chain left to right — row times column, add the shift, bend — and keeps every z and a in a cache for the way back.

What comes out, and how we score it

Imagine this

Two kinds of guessing game. In the first, you guess a number — "this flat rents for 25 thousand" — and you are told how far off you were. In the second, you pick from a list — "this photo is a cat, a dog or a cow" — and you are told how sure you were of the right one.

A network ends with a head made for its game. The two heads look different. But the blame they send back has the same simple shape.

Regression head (a number). The output is the number $\hat y$ . We score it with $L=\tfrac12(\hat y-y)^2$ . Why a square? Unit 14 showed it: if the truth is the guess plus bell-shaped noise, then the squared error is exactly "how unlikely the data is". And why the $\tfrac12$ ? Differentiate:

$\frac{\partial L}{\partial\hat y}=\tfrac12\cdot2(\hat y-y)=\hat y-y.$

The $\tfrac12$ is there only to cancel the 2. For our salary: $40-50=-10$ .

Classification head (a choice). The last layer gives one score per class, $\mathbf z$ . Softmax turns scores into probabilities (Unit 14): $q_i=e^{z_i}/\sum_je^{z_j}$ . The loss is the cross-entropy: minus the log of the probability you gave the right class (Unit 14). Take scores $\mathbf z=(2,1,0)$ with class 1 correct:

$\mathbf q=(0.665,\,0.245,\,0.090),\qquad L=-\ln0.665\approx0.408.$

And the blame at the scores? Write the truth as a one-hot list $\mathbf y=(1,0,0)$ . Then

$\frac{\partial L}{\partial\mathbf z}=\mathbf q-\mathbf y=(-0.335,\,0.245,\,0.090).$

Look at the two results side by side: $\hat y-y$ and $\mathbf q-\mathbf y$ . Both say prediction minus truth. The right class's score gets pushed up (negative blame means "increase me"). The wrong classes get pushed down, each in proportion to how much probability it wrongly took.

From here the blame enters the network the same way in both cases. If the layer below fed $\mathbf a=(1,2)$ into these three scores, the weight blame is the outer product (Rule B, coming in §7):

$\frac{\partial L}{\partial W}=(\mathbf q-\mathbf y)\,\mathbf a^{\mathsf T}=\begin{pmatrix}-0.335&-0.670\\ 0.245&0.489\\ 0.090&0.180\end{pmatrix}.$

Try: Drag $\hat y$ above 50: the arrow flips and points down. On the right, raise the score of class 1 until $q_1$ is near 1: all three arrows shrink to nothing. Then press same story to line the two arrows up.

prediction

\hat y

score

z_1

score

z_2

score

z_3

The realization

$\frac{\partial L}{\partial\hat y}=\hat y-y\quad\text{and}\quad\frac{\partial L}{\partial\mathbf z}=\mathbf q-\mathbf y$

At the output, the whole blame is simply prediction minus truth — for a number with squared error, and for a choice with softmax and cross-entropy. The head changes; the shape of the answer does not. Everything before the head is the same backward walk.

Pause & predict

A three-class head gives $\mathbf q=(0.2,\,0.7,\,0.1)$ and the true class is 3. What is $\partial L/\partial\mathbf z$ ?

Pause & predict

We dropped the $\tfrac12$ and used $L=(\hat y-y)^2$ instead. What changes in training?

If you want the algebra · 2 proofs, step by step

Prove it · the ½ is there to cancel the 2

Claim. For $L=\tfrac12(\hat y-y)^2$ , the output gradient is $\partial L/\partial\hat y=\hat y-y$ . And minimising $L$ is maximising a Gaussian likelihood.

The chain rule on a square:

\dfrac{d}{d\hat y}\,\tfrac12(\hat y-y)^2=\tfrac12\cdot2(\hat y-y)\cdot1=\hat y-y.

Without the ½ every gradient in the network would simply be twice as big — the same direction, a different step size.

If the truth is

y=\hat y+

Gaussian noise of spread

\sigma

, the likelihood of one example is

\frac{1}{\sqrt{2\pi}\sigma}e^{-(y-\hat y)^2/2\sigma^2}

, so

\begin{aligned}&-\log(\text{likelihood})\\ &=\frac{1}{\sigma^2}\cdot\tfrac12(\hat y-y)^2+\text{const}.\end{aligned}

∎ So squared error is the negative log-likelihood of Gaussian noise, exactly as Unit 14 showed.

Prove it · softmax + cross-entropy gives q − y

Claim. With $q_i=e^{z_i}/\sum_je^{z_j}$ and true class $c$ , the loss $L=-\log q_c$ has $\partial L/\partial z_i=q_i-y_i$ , where $y$ is the one-hot label.

Open the log:

\begin{aligned}L&=-\log\frac{e^{z_c}}{\sum_je^{z_j}}\\ &=-z_c+\log\sum_je^{z_j}.\end{aligned}

The log of a fraction is the top minus the bottom.

Differentiate by

z_i

. The first term gives

-1

i=c

and

0

otherwise — that is

-y_i

. The second gives

e^{z_i}/\sum_je^{z_j}=q_i

\frac{\partial L}{\partial z_i}=q_i-y_i.

∎ Scores

(2,1,0)

, true class 1:

q-y=(-0.335,\,0.245,\,0.090)

. The entries add to 0 because both

q

and

y

add to 1. Unit 14 has the full story.

In one sentence: Whether the head predicts a number (squared error) or a class (softmax + cross-entropy), the blame it sends back is prediction minus truth — $\hat y-y$ or $\mathbf q-\mathbf y$ .

Two kinds of arrow, two rules

Imagine this

India has lost a close match by 10 runs. In the review, the coach does not blame everyone equally. The bowler who gave away the most runs gets the most blame. A player who was not on the field that day gets none at all.

The backward pass is that review. The mistake at the end, −10, is shared out along the wires — more to whoever contributed more, nothing to whoever was switched off.

Look again at the chain of eight nodes. The arrows between them come in only two kinds:

a linear arrow, $\mathbf z=W\mathbf a+\mathbf b$ — three of them: $\mathbf a^{(0)}\to\mathbf z^{(1)}$ , $\mathbf a^{(1)}\to\mathbf z^{(2)}$ , $\mathbf a^{(2)}\to z^{(3)}$ ;
a ReLU arrow, $\mathbf a=\operatorname{ReLU}(\mathbf z)$ — three of them: $\mathbf z^{(1)}\to\mathbf a^{(1)}$ , $\mathbf z^{(2)}\to\mathbf a^{(2)}$ , $z^{(3)}\to\hat y$ .

The whole network alternates: linear, ReLU, linear, ReLU, linear, ReLU — then the loss. So to walk backwards we need only two rules, one for each kind of arrow. We walk back carrying one thing: the blame on the node we stand on, $\partial L/\partial(\text{node})$ . Each arrow we cross turns one blame into the next. This is exactly Unit 7's "local slope × incoming blame" (Unit 7, §3), now done with whole lists and matrices (Unit 7, §6).

Rule A · crossing a ReLU arrow backwards

Each $a_j$ depends only on its own $z_j$ . So the blame passes straight through, times the local slope — and ReLU's slope is 1 or 0:

$\frac{\partial L}{\partial\mathbf z}=\frac{\partial L}{\partial\mathbf a}\odot\operatorname{ReLU}'(\mathbf z).$

The sign $\odot$ means "multiply entry by entry". Tiny example: blame $(-60,10,-20)$ arriving at $\mathbf z=(7,4,-2)$ . The slopes are $(1,1,0)$ . So the blame that gets through is $(-60,10,0)$ . Rule A is a switch: open where the neuron was awake, shut where it was asleep.

Rule B · crossing a linear arrow backwards

Three things fed into $\mathbf z=W\mathbf a+\mathbf b$ , so crossing it back gives three answers:

$\frac{\partial L}{\partial W}=\frac{\partial L}{\partial\mathbf z}\,\mathbf a^{\mathsf T},\quad \frac{\partial L}{\partial\mathbf b}=\frac{\partial L}{\partial\mathbf z},\quad \frac{\partial L}{\partial\mathbf a}=W^{\mathsf T}\frac{\partial L}{\partial\mathbf z}.$

The weights: each weight's blame = the blame at its output × what flowed in through it. As a grid that is an outer product: a column times a row.
The shifts: a shift adds straight into its $z$ , so it gets exactly the blame of that $z$ .
The layer below: each input fed every neuron, so its blame adds up the blame of all of them, weighted by the wires. Forward you multiply by $W$ . Backward you multiply by $W^{\mathsf T}$ . Same wires, opposite direction.

Tiny example (layer 3 of the salary network): blame $-10$ at $z^{(3)}$ , stored $\mathbf a^{(2)}=(6,7)$ , weights $W^{(3)}=(4,2)$ . Then $\partial L/\partial W^{(3)}=(-10)(6,7)=(-60,-70)$ , $\partial L/\partial b^{(3)}=-10$ , and $\partial L/\partial\mathbf a^{(2)}=(4,2)^{\mathsf T}(-10)=(-40,-20)$ .

Where does each transpose go? Use one rule that never fails: the blame of anything has the same shape as the thing. $\partial L/\partial W$ must be shaped like $W$ , so it can only be (column)(row).

Try: On the left, click a $z$ to flip its sign and watch its switch open or shut. On the right, press ▶ build ∂L/∂W and watch the grid fill cell by cell, then ▶ build Wᵀ·blame and watch the blame travel back along the same wires.

Rule A · the ReLU switch

Rule B · the linear arrow

The realization

$\begin{gathered}\text{Rule A:}\\ \frac{\partial L}{\partial\mathbf z}=\frac{\partial L}{\partial\mathbf a}\odot\operatorname{ReLU}'(\mathbf z)\\ \text{Rule B:}\\ \frac{\partial L}{\partial W}=\frac{\partial L}{\partial\mathbf z}\mathbf a^{\mathsf T},\quad \frac{\partial L}{\partial\mathbf b}=\frac{\partial L}{\partial\mathbf z},\\ \frac{\partial L}{\partial\mathbf a}=W^{\mathsf T}\frac{\partial L}{\partial\mathbf z}\end{gathered}$

Crossing a ReLU arrow: pass the blame where the neuron was awake, block it where it slept. Crossing a linear arrow: a weight's blame is its output's blame × its input, and the layer below gets the blame sent back through $W^{\mathsf T}$ — the same wires, walked the other way. Two rules, any depth.

Pause & predict

Blame $\partial L/\partial\mathbf a=(5,-3,2)$ arrives at a ReLU layer whose stored $\mathbf z=(-1,\,2,\,0.5)$ . What is $\partial L/\partial\mathbf z$ ?

Pause & predict

A layer has $W=\begin{pmatrix}1&2\\ 3&0\end{pmatrix}$ , stored input $\mathbf a=(2,1)$ , and the blame arriving at its output is $\partial L/\partial\mathbf z=(1,-1)$ . What is $\partial L/\partial\mathbf a$ ?

If you want the algebra · 3 proofs, step by step

Prove it · Rule A, the switch

Claim. Across $\mathbf a=\operatorname{ReLU}(\mathbf z)$ (entry by entry), $\dfrac{\partial L}{\partial\mathbf z}=\dfrac{\partial L}{\partial\mathbf a}\odot\operatorname{ReLU}'(\mathbf z)$ .

a_j=\operatorname{ReLU}(z_j)

depends on

z_j

and on nothing else. So

z_j

reaches the loss only through

a_j

— one path, one term in the chain rule (Unit 7):

\begin{aligned}\frac{\partial L}{\partial z_j}&=\frac{\partial L}{\partial a_j}\cdot\frac{\partial a_j}{\partial z_j}\\ &=\frac{\partial L}{\partial a_j}\cdot\operatorname{ReLU}'(z_j).\end{aligned}

No sum, because

z_j

does not feed any other

a_k

Do it for every

j

at once: multiply the two lists entry by entry. That is what

\odot

means. ∎

\operatorname{ReLU}'(z)

is 1 for

z>0

and 0 for

z<0

: a switch. (At exactly

z=0

we pick 0, a common convention.)

Prove it · Rule B, parts (i) and (ii): the weights and the shift

Claim. Across $\mathbf z=W\mathbf a+\mathbf b$ : $\dfrac{\partial L}{\partial W_{jk}}=\dfrac{\partial L}{\partial z_j}\,a_k$ , so $\dfrac{\partial L}{\partial W}=\dfrac{\partial L}{\partial\mathbf z}\,\mathbf a^{\mathsf T}$ ; and $\dfrac{\partial L}{\partial\mathbf b}=\dfrac{\partial L}{\partial\mathbf z}$ .

Row

j

of the arrow says

z_j=\sum_kW_{jk}a_k+b_j

. The number

W_{jk}

appears in

z_j

only, multiplied by

a_k

. So

\partial z_j/\partial W_{jk}=a_k

, and one chain-rule term:

\frac{\partial L}{\partial W_{jk}}=\frac{\partial L}{\partial z_j}\,a_k.

"A weight's blame = the blame at its output × what flowed in through it."

Entry

(j,k)

is (entry

j

of a column) × (entry

k

of a row). A grid built like that is an outer product:

\frac{\partial L}{\partial\mathbf z}\mathbf a^{\mathsf T}

. Shapes:

(n\times1)(1\times m)=n\times m

, the shape of

W

b_j

also sits only in

z_j

, with coefficient 1:

\partial L/\partial b_j=\partial L/\partial z_j

. ∎ Salary example, layer 3:

(-10)(6,7)=(-60,-70)

, and

\partial L/\partial b^{(3)}=-10

Prove it · Rule B, part (iii): blame for the layer below is Wᵀ times blame

Claim. $\dfrac{\partial L}{\partial\mathbf a}=W^{\mathsf T}\dfrac{\partial L}{\partial\mathbf z}$ .

Now

a_k

sits inside every

z_j

(each row uses every input). So

a_k

reaches the loss along many paths, and the chain rule adds them up:

\begin{aligned}\frac{\partial L}{\partial a_k}&=\sum_j\frac{\partial L}{\partial z_j}\,\frac{\partial z_j}{\partial a_k}\\ &=\sum_j\frac{\partial L}{\partial z_j}\,W_{jk}.\end{aligned}

Many paths means a sum — this is the "add the blame from every road" rule of Unit 7.

The sum runs down column $k$ of

W

. Column

k

W

is row

k

W^{\mathsf T}

. So the whole list is

W^{\mathsf T}\,\partial L/\partial\mathbf z

. ∎ Salary example, layer 2:

W^{(2)\mathsf T}(-40,-20)=(-60,\,10,\,-20)

. Forward you multiply by

W

; backward by

W^{\mathsf T}

— the same wires, walked the other way.

In one sentence: Walking back through a network needs only two rules — a ReLU arrow is a switch that passes or blocks the blame, and a linear arrow gives the weights (blame)(input)ᵀ, the shifts the blame itself, and the layer below Wᵀ·blame.

The backward pass, by hand

Imagine this

The dabbawala relay again — but now a tiffin reached the wrong office. The complaint travels back along the same route: office to last runner, last runner to the train, the train to the first runner. At each hand-over, the register says who touched it and how much.

That is the backward pass: the complaint walks the chain in reverse, and the register (the cache) tells each step what it needs.

Walk the salary network from right to left, one arrow at a time.

B1 · start at the loss. $\partial L/\partial\hat y=\hat y-y=40-50=-10$ .

B2 · cross $z^{(3)}\to\hat y$ (Rule A). We stored $z^{(3)}=40$ , which is positive, so the switch is on: $\partial L/\partial z^{(3)}=-10\times1=-10$ .

B3 · cross $\mathbf a^{(2)}\to z^{(3)}$ (Rule B). Stored $\mathbf a^{(2)}=(6,7)$ , and $W^{(3)}=(4,2)$ :

$\begin{aligned}\frac{\partial L}{\partial W^{(3)}}&=(-10)\begin{pmatrix}6&7\end{pmatrix}=\begin{pmatrix}-60&-70\end{pmatrix},\\ \frac{\partial L}{\partial b^{(3)}}&=-10,\\ \frac{\partial L}{\partial\mathbf a^{(2)}}&=\begin{pmatrix}4\\ 2\end{pmatrix}(-10)=\begin{pmatrix}-40\\ -20\end{pmatrix}.\end{aligned}$

B4 · cross $\mathbf z^{(2)}\to\mathbf a^{(2)}$ (Rule A). Stored $\mathbf z^{(2)}=(6,7)$ : both awake, so $\partial L/\partial\mathbf z^{(2)}=(-40,-20)$ .

B5 · cross $\mathbf a^{(1)}\to\mathbf z^{(2)}$ (Rule B). Stored $\mathbf a^{(1)}=(7,4,0)$ :

$\begin{aligned}\frac{\partial L}{\partial W^{(2)}}&=\begin{pmatrix}-40\\ -20\end{pmatrix}\begin{pmatrix}7&4&0\end{pmatrix}\\ &=\begin{pmatrix}-280&-160&0\\ -140&-80&0\end{pmatrix},\\ \frac{\partial L}{\partial\mathbf b^{(2)}}&=\begin{pmatrix}-40\\ -20\end{pmatrix}.\end{aligned}$

And for the layer below, $W^{(2)\mathsf T}$ times the blame, row by row:

$\begin{aligned}\text{row 1: }&(1)(-40)+(1)(-20)=-60\\ \text{row 2: }&(-0.5)(-40)+(0.5)(-20)=20-10=10\\ \text{row 3: }&(1)(-40)+(-1)(-20)=-40+20=-20\end{aligned}$

B6 · cross $\mathbf z^{(1)}\to\mathbf a^{(1)}$ (Rule A — the switch closes). Stored $\mathbf z^{(1)}=(7,4,-2)$ . The third is negative:

$\frac{\partial L}{\partial\mathbf z^{(1)}}=\begin{pmatrix}-60\\ 10\\ -20\end{pmatrix}\odot\begin{pmatrix}1\\ 1\\ 0\end{pmatrix}=\begin{pmatrix}-60\\ 10\\ 0\end{pmatrix}.$

B7 · cross $\mathbf a^{(0)}\to\mathbf z^{(1)}$ (Rule B, the last step). Stored $\mathbf a^{(0)}=(30,10)$ :

$\begin{aligned}\frac{\partial L}{\partial W^{(1)}}&=\begin{pmatrix}-60\\ 10\\ 0\end{pmatrix}\begin{pmatrix}30&10\end{pmatrix}=\begin{pmatrix}-1800&-600\\ 300&100\\ 0&0\end{pmatrix},\\ \frac{\partial L}{\partial\mathbf b^{(1)}}&=\begin{pmatrix}-60\\ 10\\ 0\end{pmatrix}.\end{aligned}$

That is all 20 gradients. Three moments are worth saying out loud:

The bigger weight gets more blame. In B3, the wire with weight 4 sends back −40 and the wire with weight 2 sends back −20. Twice the weight, twice the blame. That is all $W^{\mathsf T}$ is doing.
A weight that carried nothing gets no blame. The third column of $\partial L/\partial W^{(2)}$ is all zero, because it multiplied $a^{(1)}_3=0$ .
A neuron that slept on the way forward learns nothing on the way back. The −20 was blocked in B6, so the whole third row of $\partial L/\partial W^{(1)}$ is zero. Its weights will not move this step.

Try: Press ▶ play and watch the blue wave go right, then the red wave come back. Pause at B6 and look at the third neuron: the red −20 stops there. Click any box of the chain to see its value, its local slope and its blame.

Click a box of the chain.

step 1 / 17

The realization

$\begin{gathered}\frac{\partial L}{\partial W^{(l)}}=\frac{\partial L}{\partial\mathbf z^{(l)}}\,\mathbf a^{(l-1)\mathsf T}\\ \frac{\partial L}{\partial\mathbf a^{(l-1)}}=W^{(l)\mathsf T}\frac{\partial L}{\partial\mathbf z^{(l)}}\end{gathered}$

One backward sweep gives every gradient. It starts from prediction minus truth, crosses each arrow with Rule A or Rule B, and reads the stored z's and a's in reverse order. The red numbers are just the blue ones' story told backwards — except where a sleeping ReLU blocks the blame.

Pause & predict

In the salary network, why is the whole third row of $\partial L/\partial W^{(1)}$ equal to zero?

Pause & predict

Suppose the guess had been too high: the blame arriving at $z^{(3)}$ is $+4$ instead of $-10$ . With $W^{(3)}=(4,\,2)$ as before, what is $\partial L/\partial\mathbf a^{(2)}$ ?

If you want the algebra · 2 proofs, step by step

Prove it · the whole backward pass, as four formulas with shapes

Claim. For a network $\mathbf z^{(l)}=W^{(l)}\mathbf a^{(l-1)}+\mathbf b^{(l)}$ , $\mathbf a^{(l)}=\operatorname{ReLU}(\mathbf z^{(l)})$ , with $L=\tfrac12(\hat y-y)^2$ , the gradients of all parameters come from one backward sweep:

Start:

\dfrac{\partial L}{\partial\hat y}=\hat y-y

1\times1

). The seed. For a softmax head it is

\mathbf q-\mathbf y

instead, with the same meaning.

Then, for

l=L,L-1,\dots,1

\begin{aligned}\frac{\partial L}{\partial\mathbf z^{(l)}}&=\frac{\partial L}{\partial\mathbf a^{(l)}}\\ &\quad\odot\operatorname{ReLU}'(\mathbf z^{(l)})\\ \frac{\partial L}{\partial W^{(l)}}&=\frac{\partial L}{\partial\mathbf z^{(l)}}\,\mathbf a^{(l-1)\mathsf T}\\ \frac{\partial L}{\partial\mathbf b^{(l)}}&=\frac{\partial L}{\partial\mathbf z^{(l)}}\\ \frac{\partial L}{\partial\mathbf a^{(l-1)}}&=W^{(l)\mathsf T}\frac{\partial L}{\partial\mathbf z^{(l)}}\end{aligned}

Shapes, line by line:

n_l\times1

n_l\times n_{l-1}

n_l\times1

n_{l-1}\times1

(

n_l

= neurons in layer

l

). Rule A, then Rule B's three results. The top of the output layer uses

\partial L/\partial\mathbf a^{(L)}=\partial L/\partial\hat y

Each line needs only things already known: the stored

\mathbf z^{(l)}

and

\mathbf a^{(l-1)}

from the forward pass,

W^{(l)}

, and the blame just computed one layer up. So one backward sweep gives every gradient. ∎ This is why the forward pass must keep its notes (the cache): the backward pass reads them in reverse order.

Prove it · the shapes place every transpose

Claim. "The gradient of anything has the shape of that thing" leaves only one way to write each formula.

\partial L/\partial W^{(2)}

must be

2\times3

, like

W^{(2)}

. The two ingredients are

\partial L/\partial\mathbf z^{(2)}

(

2\times1

) and

\mathbf a^{(1)}

(

3\times1

). The only product of these that is

2\times3

(2\times1)(1\times3)

\frac{\partial L}{\partial W^{(2)}}=\frac{\partial L}{\partial\mathbf z^{(2)}}\,\mathbf a^{(1)\mathsf T}.

\mathbf a^{(1)}\,\partial L/\partial\mathbf z^{(2)\mathsf T}

would be

3\times2

— the wrong shape.

\partial L/\partial\mathbf a^{(1)}

must be

3\times1

. From

W^{(2)}

(

2\times3

) and

\partial L/\partial\mathbf z^{(2)}

(

2\times1

) the only

3\times1

product is

W^{(2)\mathsf T}\,\partial L/\partial\mathbf z^{(2)}

(3\times2)(2\times1)

. ∎ A shape check is not a proof of the formula — the proofs in §7 are — but it catches almost every slip.

In one sentence: Starting from $\hat y-y=-10$ and crossing the seven arrows backwards with Rule A or Rule B gives all 20 gradients — big where a weight carried a lot, zero where a neuron slept.

Update, check, and why the step had to be tiny

Imagine this

You are tuning an old radio with two knobs. One knob is very sensitive: a tiny turn jumps across three stations. The other barely changes anything. If you turn both by the same amount, you will overshoot with the first and hardly move the second.

Our salary network has exactly this problem. And the cause is simple: the age, 30, is a big number.

The gradient points uphill on the loss, so we step the other way (Unit 9). For every weight and every shift:

$\begin{gathered}W^{(l)}\leftarrow W^{(l)}-\eta\,\frac{\partial L}{\partial W^{(l)}},\\ \mathbf b^{(l)}\leftarrow\mathbf b^{(l)}-\eta\,\frac{\partial L}{\partial\mathbf b^{(l)}}.\end{gathered}$

Take $\eta=10^{-5}$ . For example $W^{(1)}_{11}$ becomes $0.1-10^{-5}(-1800)=0.1+0.018=0.118$ . All the new numbers:

$\begin{aligned}W^{(1)}&=\begin{pmatrix}0.118&0.306\\ 0.197&-0.101\\ -0.1&0.2\end{pmatrix},\ \mathbf b^{(1)}=\begin{pmatrix}1.0006\\ -1.0001\\ -1\end{pmatrix},\\ W^{(2)}&=\begin{pmatrix}1.0028&-0.4984&1\\ 1.0014&0.5008&-1\end{pmatrix},\ \mathbf b^{(2)}=\begin{pmatrix}1.0004\\ -1.9998\end{pmatrix},\\ W^{(3)}&=\begin{pmatrix}4.0006&2.0007\end{pmatrix},\ b^{(3)}=2.0001.\end{aligned}$

The third row of $W^{(1)}$ and the third column of $W^{(2)}$ did not move: their gradients were zero. Now run the chain forward again: $\mathbf z^{(1)}=(7.6006,\ 3.8999,\ -2)$ , $\mathbf a^{(2)}=(6.6786,\ 7.5645)$ , and

$\hat y=43.8527,\qquad L=\tfrac12(43.8527-50)^2=18.89.$

One forward pass and one backward pass moved the guess from 40 to 43.85, and the loss from 50 to 18.89. Repeat the pair a few hundred times and the error goes to zero.

Why was $\eta$ so tiny? Look at the gradients: $\partial L/\partial W^{(1)}_{11}=-1800$ , but $\partial L/\partial b^{(3)}=-10$ . That is 180 times bigger. Rule B explains it: a first-layer weight's blame is multiplied by its input, and the input is age = 30. So the first layer has huge gradients, and a step size that suits them is far too timid for the rest. Try $\eta=10^{-4}$ : the first step overshoots to $\hat y\approx79.3$ and the loss jumps up to about 429.

The real fix is not a smaller $\eta$ . It is to scale the inputs before training — for example age/100 and experience/10 — so that no input is huge (Unit 10). Then one $\eta$ suits every weight.

How do we know the 20 gradients are right? Check them the slow, honest way. Nudge one weight up by a tiny $h$ , run the forward pass, nudge it down by $h$ , run it again, and compare:

$\frac{L(\theta+h)-L(\theta-h)}{2h}\ \approx\ \frac{\partial L}{\partial\theta}.$

For $W^{(1)}_{11}$ with $h=0.001$ : $L(0.101)=48.2162$ , $L(0.099)=51.8162$ , so the slope is $(48.2162-51.8162)/0.002=-1800$ . Backprop said −1800. Do this for all 20 numbers and they agree to many decimal places. This is called a gradient check, and every serious implementation runs one.

Try: Press ▶ run at $\eta=10^{-5}$ and watch $\hat y$ climb to 50. Reset, set $\eta=10^{-4}$ : the first step overshoots. Push to $2\times10^{-3}$ : the network dies — $\hat y$ is stuck at 0. Now turn on scale the inputs and try $\eta=5\times10^{-4}$ .

step size

\eta

10⁻⁵

backprop against

(L(\theta+h)-L(\theta-h))/2h

h=10^{-5}

, at the current weights

Why does one input's size matter so much? Here is the loss seen through a tiny window: freeze 18 of the 20 numbers and move only two — the very first weight $W^{(1)}_{11}$ and the very last shift $b^{(3)}$ , each by the same amount. The loss becomes a landscape over a floor, and gradient descent is a ball rolling on it.

Try: With the raw inputs, press $\eta=6\times10^{-5}$ : the ball bounces from wall to wall of a narrow canyon. Turn on scale the inputs: the same network, the same starting guess — and the canyon opens into a gentle valley, where even $\eta=0.2$ rolls smoothly to the floor.

drag the picture to orbit

The realization

$\begin{gathered}\theta\leftarrow\theta-\eta\,\frac{\partial L}{\partial\theta}\\ \frac{\partial L}{\partial W^{(1)}_{jk}}=\frac{\partial L}{\partial z^{(1)}_j}\,x_k\end{gathered}$

One forward pass, one backward pass, one small step: the loss fell from 50 to 18.89. The step had to be tiny because a first-layer gradient is multiplied by the raw input, and age = 30 is big. Scale the inputs and a much bigger step becomes safe. And check your gradients against finite differences — they must agree.

Pause & predict

A weight has $w=0.5$ and gradient $\partial L/\partial w=-200$ . With $\eta=10^{-3}$ , what is the new weight?

Pause & predict

A gradient check nudges $\theta$ by $h=0.01$ and finds $L(\theta+h)=3.02$ , $L(\theta-h)=2.98$ . Backprop says $\partial L/\partial\theta=20$ . What do you conclude?

If you want the algebra · 2 proofs, step by step

Prove it · why a central difference is so accurate

Claim. $\dfrac{L(\theta+h)-L(\theta-h)}{2h}=L'(\theta)+O(h^2)$ , while the one-sided $\dfrac{L(\theta+h)-L(\theta)}{h}=L'(\theta)+O(h)$ .

Taylor (Unit 8) on both sides:

\begin{aligned}L(\theta\pm h)&=L\pm hL'\\ &\quad+\tfrac{h^2}{2}L''\\ &\quad\pm\tfrac{h^3}{6}L'''+\cdots\end{aligned}

Everything evaluated at

\theta

Subtract: the even terms cancel, so

L(\theta+h)-L(\theta-h)=2hL'+\tfrac{h^3}{3}L'''+\cdots

. Divide by

2h

: the error is

\tfrac{h^2}{6}L'''

, tiny when

h

is small. ∎ For the salary network with

h=0.001

W^{(1)}_{11}

(48.2162-51.8162)/0.002=-1800

, matching backprop. When a check disagrees, the backward pass has a bug.

Prove it · big inputs make big first-layer gradients

Claim. Scaling input $x_k$ by a factor $s$ (and $W^{(1)}_{jk}$ by $1/s$ , so the network computes the same thing) scales $\partial L/\partial W^{(1)}_{jk}$ by $s$ — and the safe step size for that weight by about $1/s^2$ .

Rule B:

\partial L/\partial W^{(1)}_{jk}=\frac{\partial L}{\partial z^{(1)}_j}\,x_k

. The first factor does not change (the network's outputs are the same), so the gradient is proportional to

x_k

. Age = 30 is why

\partial L/\partial W^{(1)}_{11}=-60\times30=-1800

A step

\Delta W_{jk}=-\eta\,\partial L/\partial W_{jk}

changes

z_j

\Delta W_{jk}\,x_k=-\eta\,\frac{\partial L}{\partial z_j}x_k^2

. So the effect of one step grows like

x_k^2

, and a step size that is safe must shrink like

1/x_k^2

. ∎ That is the canyon of Unit 10: one very steep direction forces a tiny

\eta

on all of them. The cure is to scale the inputs (Unit 10, §5).

In one sentence: Step every number against its gradient — loss 50 → 18.89 — but a first-layer gradient carries the raw input inside it, so scale the inputs, and check the gradients against $(L(\theta+h)-L(\theta-h))/2h$ .

When the blame fades, explodes, or dies

Imagine this

A message is whispered down a line of twenty children. If each child passes on only a quarter of what they heard, the last child hears almost nothing. If each child shouts it twice as loud, the last one is deafened. And if one child has fallen asleep, the message stops right there.

The backward pass is that line of children. Every layer multiplies the blame by its slope and its weights.

Walking back through one layer multiplies the blame by two things: the slope of the bend (Rule A) and the weights (Rule B). Through many layers these multiply up. Three things can go wrong.

Fading (vanishing) blame. Sigmoid's slope is at most 0.25. So through 5 sigmoid layers the blame shrinks by at least $0.25^5\approx0.001$ , and through 10 layers by $0.25^{10}\approx10^{-6}$ — unless the weights are big enough to make up for it. The first layers then hear almost nothing and barely learn. tanh is better (its slope reaches 1 at $z=0$ ). ReLU's slope is exactly 1 on every awake neuron, so nothing fades there.

Exploding blame. If the weights are big, each layer multiplies the blame up. Twenty layers that each double it give about a million. Steps become wild. (Unit 17 will meet this again, as an eigenvalue story, when the same matrix is used over and over in time.)

Dead neurons. A ReLU neuron whose $z$ is negative for every input never passes any blame back — Rule A blocks it every time — so its incoming weights never change again. One sleeping neuron is normal; our salary network has one. Too many at once is the "dying ReLU" problem, and a big negative bias or one huge step (as in §9) can cause it.

The usual fixes, named here and used in the playground: ReLU-family bends; careful starting weights, about $\sqrt{2/\text{fan-in}}$ in size for ReLU (fan-in = the number of wires coming into a neuron); scaled inputs (Unit 10); and adaptive steps like Adam (Unit 11).

Try: Pick sigmoid and 12 layers: the bars near the bottom fade to almost nothing. Switch to ReLU. Set the weight scale to 2×: the bars grow as the blame goes down — exploding. Last, drag the bias slider to the left and watch neurons fall asleep and whole layers go dark.

drag the picture to orbit

layers12

bias0

weights × the careful start √(2/fan-in):

The realization

$\begin{gathered}\Big|\frac{\partial L}{\partial\mathbf a^{(0)}}\Big|\ \text{is roughly}\\ \Big|\frac{\partial L}{\partial\hat y}\Big|\times\prod_{\text{layers}}(\text{slope}\times\text{weight size})\end{gathered}$

The blame reaching the first layer is a product of one factor per layer. Factors below 1 make it fade (sigmoid: at most 0.25 each). Factors above 1 make it explode. A factor of exactly 0 — a neuron asleep for every input — kills it. Good bends, good starting weights and scaled inputs keep the factors near 1.

Pause & predict

A chain of 8 sigmoid layers has all weights equal to 1. At best, how much of the blame at the top reaches the bottom?

Pause & predict

Across a whole batch of 100 inputs, one ReLU neuron always has a negative $z$ . What are the gradients of its incoming weights for this batch?

If you want the algebra · 1 proof, step by step

Prove it · how fast blame can fade through sigmoid layers

Claim. In a chain of $k$ one-neuron sigmoid layers $a_l=\sigma(w_la_{l-1}+b_l)$ , $\left|\dfrac{\partial L}{\partial a_0}\right|\le\left|\dfrac{\partial L}{\partial a_k}\right|\prod_{l=1}^k\frac{|w_l|}{4}$ . With $|w_l|\le1$ the blame shrinks at least by $4^{-k}$ .

Crossing layer

l

backwards multiplies the blame by the local slope: Rule A gives

\sigma'(z_l)\le\tfrac14

, Rule B gives

w_l

. So

\frac{\partial L}{\partial a_{l-1}}=\frac{\partial L}{\partial a_l}\,\sigma'(z_l)\,w_l.

The §2 drawer proved

\sigma'\le\tfrac14

Multiply the

k

crossings together and bound each slope by

\tfrac14

. ∎

0.25^5\approx0.00098

and

0.25^{10}\approx9.5\times10^{-7}

. With ReLU the slope is exactly 1 on awake neurons, so nothing shrinks there; with weights bigger than 4 the product can grow instead — exploding blame.

In one sentence: The blame reaching an early layer is a product of one factor per layer, so it fades with sigmoid (≤ 0.25 each), explodes with big weights, and dies at neurons that are always asleep — good bends, starting weights and scaled inputs keep it alive.

The playground: build a network

Imagine this

You have read the recipe, watched the chef, and tasted the dish. Now the kitchen is yours. Pick the ingredients, cook, taste, change one thing, and cook again.

Everything in this unit now runs in one place. You choose how many layers, how many neurons in each, and which bend each layer uses. You choose the head: a number (squared error) or a class (softmax and cross-entropy). Then you watch the network three ways at once — as a graph of neurons and wires, as matrices, and as the space it carves up — and you step through the forward and backward pass one node at a time, blue going right and red coming back, with the rule (A or B) and the numbers of each step written out.

Five experiments are built in as buttons. Try them in order:

Salary 2-3-2-1. The worked example of this unit. Step it node by node and check every number against §5 and §8: $\mathbf z^{(1)}=(7,4,-2)$ , $\hat y=40$ , $L=50$ , $\partial L/\partial\mathbf z^{(1)}=(-60,10,0)$ . One step at $\eta=10^{-5}$ must give $L=18.89$ .
No bend = one matrix. The same data with no activation: the boundary stays a straight line however many layers you add (§3). Switch to ReLU and watch it bend around the data.
Dead ReLU. Start with a big negative bias. Most neurons sleep, the graph greys them out, and their gradients are exactly 0 (§10).
Vanishing sigmoid. Six sigmoid layers. The gradient bars fade towards the input. Swap to ReLU and they come back (§10).
Fold by fold. Fit a sine with one hidden layer. Add neurons one at a time and watch a new hinge appear with each (§4).

Try: Start with experiment 1. Press ↺ start over, then ▶ one node again and again: the salary network is computed one neuron at a time, then the blame comes back. At the end, press ▶ twice more to take one small step downhill and see the loss fall from 50 to 18.89. Then open experiments 2–5, press ▶ train, and click any neuron or wire to look inside it.

Experiments

Data noise add x₁², x₂² as inputs scale inputs (age/100, exp/10)

neuron pictures numbers on wires

value going forwardblame coming backw > 0w < 0awakeasleep

Space

look throughoutput

Train

step size η1e−5

optimizer

batch

weight scale1×

bias start0

New weights are drawn with spread (weight scale)/√(inputs to the neuron); every bias starts at the bias value. Same seed, same weights.

Blame reaching each layer typical size of ∂L/∂z, log scale

Gradient check

Nudge every weight up and down by a hair, measure the loss both times, and compare the slope with what backprop said.

The realization

$\begin{gathered}\text{forward: store every }\mathbf z,\mathbf a\\ \downarrow\\ \text{backward: Rule A, Rule B}\\ \downarrow\\ \theta\leftarrow\theta-\eta\,\partial L/\partial\theta\end{gathered}$

A network of any shape runs on the same three moves you did by hand. The playground is not a new idea — it is the salary network's forward pass, backward pass and update, done for whatever network you build, as many times as you like.

Try it, then answer

In the playground, load No bend = one matrix, keep the activation at "none", and add two more hidden layers. What shape is the boundary between the classes?

Try it, then answer

Load Salary 2-3-2-1 and inspect the wire from input "age" to the third neuron of hidden layer 1. What gradient does backprop show for it, and what does the finite-difference check show?

In one sentence: The playground runs the same forward pass, two backward rules and update on any network you build, so you can test every claim of this unit — straight collapse, folds, dead and fading blame — with your own hands.

What to carry forward

The whole unit fits on one card. Here it is, piece by piece.

A layer

Multiply by a matrix, add a shift, bend:

\mathbf a=\operatorname{ReLU}(W\mathbf a_{\text{prev}}+\mathbf b)

W

is (neurons here) × (neurons before).

No bend ⇒ collapse

Two straight layers are one:

W_2W_1

W_2\mathbf b_1+\mathbf b_2

. Depth needs bends.

Folds ⇒ any shape

One ReLU is one crease; three make a hat; enough hats trace any curve. Depth folds the folds:

2^k

pieces.

Forward

Row times column, shift, bend, layer by layer — and store every z and a.

The seed

At the output, the blame is prediction minus truth:

\hat y-y

, or

\mathbf q-\mathbf y

with softmax.

Rule A

Across a ReLU:

\partial L/\partial\mathbf z=\partial L/\partial\mathbf a\odot\operatorname{ReLU}'(\mathbf z)

. A switch.

Rule B

Across

W\mathbf a+\mathbf b

\partial L/\partial W=\partial L/\partial\mathbf z\,\mathbf a^{\mathsf T}

\partial L/\partial\mathbf b=\partial L/\partial\mathbf z

\partial L/\partial\mathbf a=W^{\mathsf T}\partial L/\partial\mathbf z

Update and check

\theta\leftarrow\theta-\eta\,\partial L/\partial\theta

. Scale the inputs. Check gradients with

(L(\theta+h)-L(\theta-h))/2h

Watch the blame

It is a product over layers: it fades (sigmoid), explodes (big weights) or dies (sleeping ReLUs).

Where this goes next.

Unit 16 · Words as Vectors. The network's first real job: reading. Each word becomes a list of numbers, and a softmax over the whole vocabulary — thousands of classes — predicts the next word. The blame at that softmax is $\mathbf q-\mathbf y$ again.
Unit 17 · Machines with Memory. The same layer is used again and again, once per word, so the backward pass runs back through time — and the fading and exploding of §10 become an eigenvalue story.
Unit 18 · Attention and Transformers. Fixed wires are replaced by wires the network chooses for itself, using dot products (Unit 3) and softmax.

The realization

$\begin{gathered}\frac{\partial L}{\partial\hat y}=\hat y-y,\ \text{then, layer by layer:}\\ \text{local slope}\times\text{incoming blame}\end{gathered}$

At the output, the whole gradient is prediction minus truth. Everything before it is Unit 7's "local slope times incoming blame", done with matrices — Rule A at every bend, Rule B at every matrix. That is backpropagation, whole.

In one sentence: A network is multiply–shift–bend repeated; training is a forward pass that stores everything, a backward pass that starts from prediction minus truth and uses two rules, and a small step downhill.

Practice arena — fourteen problems, solved in full

Fourteen problems, easy to hard: shapes and parameter counts, a forward pass by hand, collapsing two straight layers, building a shape out of ReLUs, a softmax head, both backward rules on their own, then a complete backward pass through a new network, one update, a gradient check, fading blame, dead neurons, wide against deep, and a training run to diagnose. Every number here was checked by machine.

Three habits do most of the work. Write the shapes first: every gradient has the shape of the thing it belongs to. Keep a cache: write every $\mathbf z$ and $\mathbf a$ of the forward pass in a table before you start going back. And check each ReLU's switch before you use Rule A — a negative $z$ means a zero.

Problem 1easyshapes

A network has shape 3 → 4 → 4 → 2. (a) Give the shape of every weight matrix and bias vector. (b) How many numbers does the network learn in total?

What this tests. The shape rule: $W^{(l)}$ is (neurons in layer $l$ ) × (neurons in layer $l-1$ ). Plan. Walk the arrows one at a time; count weights and biases separately, then add.

Show the full solution

Step 1 — layer 1 (3 → 4). Rows = 4 neurons here, columns = 3 before:

W^{(1)}

4\times3

(12 numbers),

\mathbf b^{(1)}

4\times1

(4).

Step 2 — layer 2 (4 → 4).

W^{(2)}

4\times4

(16),

\mathbf b^{(2)}

4\times1

(4).

Step 3 — layer 3 (4 → 2).

W^{(3)}

2\times4

(8),

\mathbf b^{(3)}

2\times1

(2).

Step 4 — add up (b).

12+4+16+4+8+2=46

answers at a glance: (a) $W^{(1)}:4\times3$ , $\mathbf b^{(1)}:4\times1$ , $W^{(2)}:4\times4$ , $\mathbf b^{(2)}:4\times1$ , $W^{(3)}:2\times4$ , $\mathbf b^{(3)}:2\times1$ . (b) 46 numbers.

Remember

Rows belong to where the wires go, columns to where they come from. Each layer adds (out × in) weights plus (out) biases.

Problem 2easyforward pass

A 2 → 2 → 1 network has a ReLU hidden layer and a plain (no bend) output: $W^{(1)}=\begin{pmatrix}1&-1\\ 2&1\end{pmatrix},\ \mathbf b^{(1)}=\begin{pmatrix}0\\ -3\end{pmatrix},\qquad W^{(2)}=\begin{pmatrix}3&-2\end{pmatrix},\ b^{(2)}=4.$ Find $\mathbf z^{(1)}$ , $\mathbf a^{(1)}$ and $\hat y$ for (a) $\mathbf x=(2,1)$ and (b) $\mathbf x=(1,3)$ .

What this tests. Row-times-column, the shift, and the ReLU switch. Plan. One row per hidden neuron; check each sign before bending.

Show the full solution

Step 1 — (a), the mix.

\begin{aligned}\text{row 1: }&(1)(2)+(-1)(1)+0=1\\ \text{row 2: }&(2)(2)+(1)(1)-3=2\end{aligned}

\mathbf z^{(1)}=(1,2)

Step 2 — (a), bend and output. Both positive, so

\mathbf a^{(1)}=(1,2)

. Then

\hat y=(3)(1)+(-2)(2)+4=3-4+4=3

Step 3 — (b), the mix. Row 1:

1-3+0=-2

. Row 2:

2+3-3=2

. So

\mathbf z^{(1)}=(-2,2)

Step 4 — (b), bend and output. The first neuron is asleep:

\mathbf a^{(1)}=(0,2)

. Then

\hat y=(3)(0)+(-2)(2)+4=0

answers at a glance: (a) $\mathbf z^{(1)}=(1,2)$ , $\mathbf a^{(1)}=(1,2)$ , $\hat y=3$ . (b) $\mathbf z^{(1)}=(-2,2)$ , $\mathbf a^{(1)}=(0,2)$ , $\hat y=0$ .

Remember

Which neurons are awake depends on the input. The same network uses different "active wires" for different people.

Problem 3easycollapse

Two layers with no activation: $W_1=\begin{pmatrix}2&0\\ 1&1\end{pmatrix}$ , $\mathbf b_1=\begin{pmatrix}1\\ -1\end{pmatrix}$ , $W_2=\begin{pmatrix}1&1\\ 0&3\end{pmatrix}$ , $\mathbf b_2=\begin{pmatrix}0\\ 2\end{pmatrix}$ . (a) Write the single layer $M\mathbf x+\mathbf c$ that does the same job. (b) Check both versions at $\mathbf x=(1,2)$ .

What this tests. $W_2(W_1\mathbf x+\mathbf b_1)+\mathbf b_2=(W_2W_1)\mathbf x+(W_2\mathbf b_1+\mathbf b_2)$ . Plan. One matrix product, one matrix–vector product, one check.

Show the full solution

Step 1 — the matrix.

M=W_2W_1=\begin{pmatrix}2+1&0+1\\ 0+3&0+3\end{pmatrix}=\begin{pmatrix}3&1\\ 3&3\end{pmatrix}

Step 2 — the shift.

\mathbf c=W_2\mathbf b_1+\mathbf b_2=(1-1,\ 0-3)+(0,2)=(0,-1)

Step 3 — two layers at (1, 2). Layer 1:

(2+0+1,\ 1+2-1)=(3,2)

. Layer 2:

(3+2+0,\ 0+6+2)=(5,8)

Step 4 — one layer at (1, 2).

(3+2+0,\ 3+6-1)=(5,8)

. The same.

answers at a glance: (a) $M=\begin{pmatrix}3&1\\ 3&3\end{pmatrix}$ , $\mathbf c=(0,-1)$ . (b) both give $(5,8)$ .

Remember

Without a bend, depth adds nothing: every stack of straight layers is one matrix and one shift.

Problem 4mediumbuilding shapes

Build this function from ReLUs of the form $c\cdot\operatorname{ReLU}(x-k)$ : it is 0 for $x\le1$ , rises in a straight line to 3 at $x=2$ , falls in a straight line to 0 at $x=5$ , and stays 0 after that. Give the creases and their weights, and check the value at $x=3$ .

What this tests. The hat idea: each crease's weight is the change of slope there. Plan. Find the slope of each straight piece, then the jumps in slope.

Show the full solution

Step 1 — slopes of the pieces. Before 1: slope 0. From 1 to 2: rise 3 over 1, slope 3. From 2 to 5: fall 3 over 3, slope −1. After 5: slope 0.

Step 2 — changes of slope. At

x=1

0\to3

, change

+3

. At

x=2

3\to-1

, change

-4

. At

x=5

-1\to0

, change

+1

Step 3 — the function.

f(x)=3\operatorname{ReLU}(x-1)-4\operatorname{ReLU}(x-2)+\operatorname{ReLU}(x-5).

Step 4 — checks.

f(2)=3

f(3)=3(2)-4(1)+0=2

(on the falling piece:

3-1=2

✓).

f(6)=15-16+1=0

answers at a glance: creases at 1, 2, 5 with weights $+3,-4,+1$ ; $f(3)=2$ .

Remember

A ReLU's weight is how much the slope turns at its crease. Read slopes off the picture, take differences, done.

Problem 5easyclassification head

A three-class head outputs scores $\mathbf z=(1,3,0)$ . The true class is 2. Find (a) the softmax $\mathbf q$ , (b) the cross-entropy loss, (c) $\partial L/\partial\mathbf z$ .

What this tests. Softmax, $-\ln q_{\text{true}}$ , and prediction minus truth. Plan. Exponentiate, add, divide; then subtract the one-hot label.

Show the full solution

Step 1 — exponentials.

e^1\approx2.718

e^3\approx20.086

e^0=1

. Total

\approx23.804

Step 2 — softmax (a).

\mathbf q\approx(2.718,\,20.086,\,1)/23.804=(0.114,\,0.844,\,0.042)

Step 3 — loss (b).

L=-\ln0.844\approx0.170

Step 4 — blame (c).

\mathbf y=(0,1,0)

, so

\mathbf q-\mathbf y\approx(0.114,\,-0.156,\,0.042)

. The entries add to 0.

answers at a glance: (a) $\mathbf q\approx(0.114,0.844,0.042)$ . (b) $L\approx0.170$ . (c) $(0.114,-0.156,0.042)$ .

Remember

At a softmax head the blame is $\mathbf q-\mathbf y$ : the true class is told "go up" by how much probability it is missing; the others "go down" by how much they took.

Problem 6easyRule A

Blame $\partial L/\partial\mathbf a=(4,-2,6,-1)$ arrives at a ReLU layer whose stored pre-activations are $\mathbf z=(3,-1,0.5,-4)$ . Find $\partial L/\partial\mathbf z$ and say which neurons pass no blame.

What this tests. Rule A as a switch. Plan. Slopes from the signs of $\mathbf z$ , then multiply entry by entry.

Show the full solution

Step 1 — the switches.

\operatorname{ReLU}'(\mathbf z)=(1,0,1,0)

: neurons 2 and 4 were asleep.

Step 2 — multiply.

(4,-2,6,-1)\odot(1,0,1,0)=(4,0,6,0)

answers at a glance: $\partial L/\partial\mathbf z=(4,0,6,0)$ ; neurons 2 and 4 pass nothing.

Remember

Rule A looks at the sign of the stored $z$ , never at the sign of the blame.

Problem 7mediumRule B

A linear layer has $W=\begin{pmatrix}2&-1&0\\ 1&3&-2\end{pmatrix}$ , stored input $\mathbf a=(1,2,3)$ , and incoming blame $\partial L/\partial\mathbf z=(0.5,-1)$ . Find $\partial L/\partial W$ , $\partial L/\partial\mathbf b$ and $\partial L/\partial\mathbf a$ .

What this tests. All three results of Rule B, and where the transpose goes. Plan. Shapes first: $\partial L/\partial W$ is $2\times3$ , $\partial L/\partial\mathbf a$ is $3\times1$ .

Show the full solution

Step 1 — the weights (outer product).

\frac{\partial L}{\partial W}=\begin{pmatrix}0.5\\ -1\end{pmatrix}\begin{pmatrix}1&2&3\end{pmatrix}=\begin{pmatrix}0.5&1&1.5\\ -1&-2&-3\end{pmatrix}.

Step 2 — the shifts.

\partial L/\partial\mathbf b=(0.5,-1)

Step 3 — the layer below ( $W^{\mathsf T}$ ).

\begin{aligned}\text{row 1: }&(2)(0.5)+(1)(-1)=0\\ \text{row 2: }&(-1)(0.5)+(3)(-1)=-3.5\\ \text{row 3: }&(0)(0.5)+(-2)(-1)=2\end{aligned}

answers at a glance: $\partial L/\partial W=\begin{pmatrix}0.5&1&1.5\\ -1&-2&-3\end{pmatrix}$ , $\partial L/\partial\mathbf b=(0.5,-1)$ , $\partial L/\partial\mathbf a=(0,-3.5,2)$ .

Remember

Weights: (blame column)(input row). Layer below: $W^{\mathsf T}$ ·blame — each input collects the blame of every neuron it fed.

Problem 8hardfull backward pass

Take Problem 2's network at $\mathbf x=(2,1)$ , with target $y=5$ and loss $L=\tfrac12(\hat y-y)^2$ . Find the loss and all six gradients: $\partial L/\partial W^{(2)}$ , $\partial L/\partial b^{(2)}$ , $\partial L/\partial\mathbf a^{(1)}$ , $\partial L/\partial\mathbf z^{(1)}$ , $\partial L/\partial W^{(1)}$ , $\partial L/\partial\mathbf b^{(1)}$ .

What this tests. The whole backward sweep on a fresh network. Plan. Cache first; the output has no bend, so the seed goes straight into Rule B.

Show the full solution

Step 1 — the cache. From Problem 2:

\mathbf a^{(0)}=(2,1)

\mathbf z^{(1)}=(1,2)

\mathbf a^{(1)}=(1,2)

\hat y=3

. Loss

L=\tfrac12(3-5)^2=2

Step 2 — the seed.

\partial L/\partial\hat y=3-5=-2

. No bend at the output, so this is also

\partial L/\partial z^{(2)}

Step 3 — Rule B at layer 2.

\partial L/\partial W^{(2)}=(-2)(1,2)=(-2,-4)

\partial L/\partial b^{(2)}=-2

\partial L/\partial\mathbf a^{(1)}=(3,-2)^{\mathsf T}(-2)=(-6,4)

Step 4 — Rule A at layer 1.

\mathbf z^{(1)}=(1,2)

: both awake, so

\partial L/\partial\mathbf z^{(1)}=(-6,4)

Step 5 — Rule B at layer 1.

\frac{\partial L}{\partial W^{(1)}}=\begin{pmatrix}-6\\ 4\end{pmatrix}\begin{pmatrix}2&1\end{pmatrix}=\begin{pmatrix}-12&-6\\ 8&4\end{pmatrix},\qquad\frac{\partial L}{\partial\mathbf b^{(1)}}=\begin{pmatrix}-6\\ 4\end{pmatrix}.

answers at a glance: $L=2$ ; $\partial L/\partial W^{(2)}=(-2,-4)$ , $\partial L/\partial b^{(2)}=-2$ , $\partial L/\partial\mathbf a^{(1)}=(-6,4)$ , $\partial L/\partial\mathbf z^{(1)}=(-6,4)$ , $\partial L/\partial W^{(1)}=\begin{pmatrix}-12&-6\\ 8&4\end{pmatrix}$ , $\partial L/\partial\mathbf b^{(1)}=(-6,4)$ .

Remember

Seed, then alternate: Rule B, Rule A, Rule B… Every step reads one thing from the cache.

Problem 9mediumone update

Continue Problem 8. Take one gradient-descent step with $\eta=0.01$ . Write the new weights and shifts, run the forward pass again at $\mathbf x=(2,1)$ , and give the new $\hat y$ and loss.

What this tests. $\theta\leftarrow\theta-\eta\,\partial L/\partial\theta$ for every number, then a fresh forward pass. Plan. Each change is $-0.01\times$ gradient.

Show the full solution

Step 1 — the new numbers.

\begin{aligned}W^{(1)}&=\begin{pmatrix}1.12&-0.94\\ 1.92&0.96\end{pmatrix},\ \mathbf b^{(1)}=\begin{pmatrix}0.06\\ -3.04\end{pmatrix},\\ W^{(2)}&=\begin{pmatrix}3.02&-1.96\end{pmatrix},\ b^{(2)}=4.02.\end{aligned}

For example

1-0.01(-12)=1.12

, and

-3-0.01(4)=-3.04

Step 2 — forward again. Row 1:

2.24-0.94+0.06=1.36

. Row 2:

3.84+0.96-3.04=1.76

. Both awake.

Step 3 — output and loss.

\hat y=(3.02)(1.36)+(-1.96)(1.76)+4.02=4.1072-3.4496+4.02=4.6776

L=\tfrac12(4.6776-5)^2\approx0.052

answers at a glance: $\hat y=4.6776$ , loss $2\to0.052$ .

Remember

One forward pass, one backward pass, one small step — and the loss here fell by a factor of almost 40.

Problem 10mediumgradient check

In Problem 8's setting, check $\partial L/\partial W^{(2)}_1$ (the weight 3) by finite differences with $h=0.1$ . (a) Compute $L(3.1)$ and $L(2.9)$ and the central difference. (b) Compute the one-sided difference $(L(3.1)-L(3))/0.1$ . Compare both with backprop.

What this tests. Doing a gradient check by hand, and why the central version is better. Plan. Only $\hat y$ changes: $\hat y=w\cdot1-2\cdot2+4=w$ .

Show the full solution

Step 1 — the loss as a function of this weight. With

\mathbf a^{(1)}=(1,2)

fixed,

\hat y=w-4+4=w

, so

L(w)=\tfrac12(w-5)^2

Step 2 — central (a).

L(3.1)=\tfrac12(1.9)^2=1.805

L(2.9)=\tfrac12(2.1)^2=2.205

. Central difference

(1.805-2.205)/0.2=-2

. Backprop said

-2

: exact agreement.

Step 3 — one-sided (b).

L(3)=2

(1.805-2)/0.1=-1.95

: off by 0.05, which is

h/2

times the curvature 1.

answers at a glance: (a) $1.805,\ 2.205,\ -2$ (matches backprop). (b) $-1.95$ .

Remember

Use the central difference $(L(\theta+h)-L(\theta-h))/2h$ : its error shrinks like $h^2$ , the one-sided one only like $h$ .

Problem 11easyfading blame

A chain of 6 one-neuron sigmoid layers. (a) If every $|w|\le2$ , what is the most the blame can keep, as a fraction, from top to bottom? (b) And if every $|w|\le1$ ? (c) What is the factor through 6 awake ReLU layers with every $w=1$ ?

What this tests. The blame is a product of (slope × weight) per layer. Plan. Sigmoid's slope ≤ 0.25; ReLU's slope = 1 when awake.

Show the full solution

Step 1 — (a). Each layer: at most

0.25\times2=0.5

. Six layers:

0.5^6=1/64\approx0.0156

Step 2 — (b). Each layer at most

0.25

0.25^6=1/4096\approx0.000244

Step 3 — (c). Each layer exactly

1\times1=1

, so the factor is 1: nothing fades.

answers at a glance: (a) $1/64\approx0.0156$ . (b) $1/4096\approx0.000244$ . (c) 1.

Remember

Blame through $k$ layers is a product of $k$ factors. Keep each factor near 1.

Problem 12mediumdead neurons

A ReLU layer of 3 neurons sees a batch of 4 inputs $\mathbf x_1=(1,0)$ , $\mathbf x_2=(2,1)$ , $\mathbf x_3=(0,1)$ , $\mathbf x_4=(1,1)$ . Its pre-activations (rows = neurons, columns = inputs) and the incoming blames $\partial L/\partial a$ are $Z=\begin{pmatrix}2&-1&0.5&3\\ -3&-0.2&-1&-4\\ -1&-2&1&-0.5\end{pmatrix},\quad G=\begin{pmatrix}1&1&1&1\\ 0.5&-0.5&2&1\\ 0.4&-0.2&0.6&1\end{pmatrix}.$ (a) Which neuron is dead on this batch? (b) Find each neuron's bias gradient, summed over the batch. (c) Find each neuron's weight gradients, summed over the batch.

What this tests. Rule A per input, then Rule B summed over a batch. Plan. Mask $G$ by the signs of $Z$ ; sum rows for the biases; for the weights add (masked blame) × input.

Show the full solution

Step 1 — the switches. Neuron 1 is awake on inputs 1, 3, 4. Neuron 2 is asleep on every input — dead. Neuron 3 is awake only on input 3.

Step 2 — masked blame. Neuron 1:

(1,0,1,1)

. Neuron 2:

(0,0,0,0)

. Neuron 3:

(0,0,0.6,0)

Step 3 — biases (b). Sum each row:

3,\ 0,\ 0.6

Step 4 — weights (c). Neuron 1:

1\cdot(1,0)+1\cdot(0,1)+1\cdot(1,1)=(2,2)

. Neuron 2:

(0,0)

. Neuron 3:

0.6\cdot(0,1)=(0,0.6)

answers at a glance: (a) neuron 2. (b) $(3,\ 0,\ 0.6)$ . (c) $(2,2)$ , $(0,0)$ , $(0,0.6)$ .

Remember

A neuron asleep on the whole batch gets exactly zero gradient, so plain gradient descent can never wake it. That is a dead ReLU.

Problem 13mediumwide or deep

Two ReLU networks with one input and one output. A: one hidden layer of 16 neurons (1 → 16 → 1). B: five hidden layers of 2 neurons (1 → 2 → 2 → 2 → 2 → 2 → 1). (a) Count each network's numbers. (b) At most how many straight pieces can A make? (c) Using the tent fold, how many can B make?

What this tests. Parameter counting, and "width adds creases, depth folds the folds". Plan. Count per layer; A has one crease per neuron; B doubles per layer.

Show the full solution

Step 1 — count A.

16

weights +

16

biases in,

16

weights +

1

bias out:

49

Step 2 — count B. First layer

2+2=4

. Four

2\to2

layers,

4+2=6

each:

24

. Output

2+1=3

. Total

31

Step 3 — pieces of A. 16 creases cut the line into at most

17

straight pieces.

Step 4 — pieces of B. Each layer of 2 ReLUs makes one tent,

t(x)=2\operatorname{ReLU}(x)-4\operatorname{ReLU}(x-\tfrac12)

, and each tent doubles the pieces:

2^5=32

answers at a glance: (a) A: 49, B: 31. (b) 17. (c) 32.

Remember

Width adds pieces one crease at a time. Depth multiplies them. That is one reason deep networks are worth their trouble.

Problem 14harddiagnose the run

The salary network of this unit was trained three times from the same start with three step sizes. The recorded losses:

$\eta$	step 0	step 1	step 2	step 3	step 10
$10^{-5}$	50	18.89	7.07	2.62	0.0024
$10^{-4}$	50	428.81	1078.76	841.74	0.0157
$2\times10^{-3}$	50	624412	1250	1250	1250

(a) Which run would you trust, and why? (b) What is

\hat y

in the third run from step 2 on? (c) Why can the third run never recover? (d) What is the real fix?

What this tests. Reading a loss curve like a doctor (Unit 10), and connecting it to dead ReLUs. Plan. Look for monotone falls, jumps, and a loss that freezes at a suspicious value.

Show the full solution

Step 1 — (a).

\eta=10^{-5}

: the loss falls every step, by a factor of about 2.7 each time. The

10^{-4}

run jumps up to 429 and then 1079 — it overshoots and bounces; it happens to settle by step 10, but you would not bet on it. Trust

10^{-5}

Step 2 — (b). A loss of exactly 1250 means

\tfrac12(\hat y-50)^2=1250

, so

\hat y=0

\hat y=100

. If it were 100, the blame

\hat y-y=+50

would move the weights and the loss would change. It is frozen, so nothing is moving:

\hat y=0

Step 3 — (c). The giant first step (loss 624 412,

\hat y\approx1168

) pushed the first-layer weights so far that every hidden-1 neuron now has a negative

z

: all asleep. Rule A blocks all blame, every gradient before them is 0, and

\hat y

stays 0. The network is dead.

Step 4 — (d). Scale the inputs (age/100, experience/10). Then the first-layer gradients are no longer 30 times inflated, and a much larger

\eta

(like

5\times10^{-4}

) trains smoothly.

answers at a glance: (a) $\eta=10^{-5}$ (steady fall). (b) $\hat y=0$ . (c) every first-layer neuron is asleep, so all gradients are 0. (d) scale the inputs.

Remember

A loss frozen at a round number is a clue. $\tfrac12y^2$ means the network is outputting 0 — often because its ReLUs have died.

Next up

Unit 16 · Words as Vectors — upcoming

The network gets its first real job: reading. Each word becomes a list of numbers — a vector — and a softmax over the whole vocabulary predicts the next word. The blame at that softmax is prediction minus truth again. And meaning turns out to be a direction: the arrow from "man" to "woman" points the same way as the arrow from "king" to "queen".

← Unit 14 · Thinking in Probabilities · All units