Unit 7 · Backpropagation & Automatic Differentiation

1

The million-knob problem

Imagine this

You are playing darts. Your first throw misses. Your brain notes how far off you were and in which direction — that is the error — and quietly adjusts your stance, grip, and force for the next throw. You learn from the miss. A neural network learns the same way: it makes a prediction (the throw), a loss function measures the miss (the error), and then every parameter gets adjusted to shrink that miss. The question this unit answers: with a million knobs, how does each one find out its own share of the blame — and how can that possibly be cheap?

From Unit 6 you already know what each knob needs: the partial derivative of the loss with respect to that knob. Gradient descent does the rest. So the whole problem is computing a million derivatives, fast.

Try the honest way first, on one small function from the lecture:

f(x)=\sqrt{x^2+e^{x^2}}+\cos\!\big(x^2+e^{x^2}\big)

Differentiating by hand, the chain rule sprays copies of $2x+2xe^{x^2}$ everywhere:

\begin{aligned}\frac{df}{dx}&=\frac{2x+2xe^{x^2}}{2\sqrt{x^2+e^{x^2}}}\\[.3em]&\quad-\sin\!\big(x^2+e^{x^2}\big)\big(2x+2xe^{x^2}\big)\end{aligned}

One input, and the expression is already a mouthful — long to write, slow to evaluate, easy to get wrong. Now scale the ambition: a modern network has billions of parameters, and we need the derivative with respect to every one of them, at every training step. Writing formulas by hand is not just impractical — it is impossible.

Backpropagation is the way out, and its two ideas fit in one breath: break the function into baby steps, and send the error backwards through them. Everything else in this unit is those two ideas, carefully.

Pause & predict

A network has a million parameters. Estimating each gradient separately by nudging one parameter and re-running the network would cost about how much?

In one sentence: Training needs the derivative of the loss with respect to every knob, every step — and computing them one at a time would cost a forward pass per knob.

2

Baby steps — the computation graph

Here is the monster function again — but this time we refuse to differentiate it. Instead we just compute it, one baby step at a time, giving each intermediate result a name:

a=x^2,\quad b=e^{a},\quad c=a+b,\quad d=\sqrt{c},\quad e=\cos(c),\quad f=d+e

Each step is trivial — a square, an exponential, an addition. Drawn with arrows showing who feeds whom, this is a computation graph, and it is the single most important picture in modern machine learning. Push a number through it, then watch the error walk home:

Try: press ▶ forward, then use step once to walk the blame back one node at a time — and pause at $c$ , where two pieces of blame add.

drag the graph to orbit it

input x1

—

Forward, every node computes one baby step and remembers its value. Backward, every node receives its blame $\bar v = \partial f/\partial v$ — starting from $\bar f = 1$ — and passes blame to its parents by multiplying with a local derivative. Watch node $c$ : it feeds two children, so its blame arrives in two pieces that add. At the far left, $\bar x$ is the answer $df/dx$ — and the chip checks it against a numerical nudge.

Errata — check this against your companion doc

The companion's worked run of this graph at $x=2$ reports $e=\cos(58.6)\approx 0.83$ and ends with $df/dx\approx-2.224$ . But $58.6$ here is in radians: $\cos(58.6)\approx-0.46$ and $\sin(58.6)\approx 0.89$ , so $\bar c=\tfrac{1}{2\sqrt c}-\sin(c)\approx 0.065-0.888=-0.822$ (not $-0.01$ ), and the true answer is $df/dx\big|_{x=2}\approx\mathbf{-182.87}$ — the $e^{x^2}$ term makes it huge. The method in the companion is exactly right; only the arithmetic slipped. Every number on this page is machine-verified.

Pause & predict

In the graph, node $a=x^2$ feeds both $b=e^a$ and $c=a+b$ . When blame flows backward, what does $a$ receive?

In one sentence: Rewrite any function as baby steps in a graph; then the forward pass computes values, and the backward pass hands every node its blame

\bar v=\partial f/\partial v

.

3

The whole algorithm is two rules

In plain words

Rule 1 — along a path, multiply. Each edge of the graph has a local derivative (how much the child moves per unit of parent). Following a path from $x$ to $f$ , the sensitivities multiply — Unit 6's amplifiers in series. Rule 2 — where paths merge, add. If $x$ can reach $f$ by several routes, each route contributes its product, and the contributions add. That is all backpropagation is: multiply along paths, add across paths — organised so no path is ever walked twice.

See both rules in the smallest graph that has two paths — the practice set's own example: $f=uv$ with $u=x^2+y$ and $v=x-y$ . The input $x$ reaches $f$ through $u$ and through $v$ ; click each path and watch its product, then their sum:

Try: click route via u, then route via v, then both — and slide $x$ to $0$ to see one whole route switch off.

x1

y1

—

At $(1,1)$ : via $u$ : $\;v\cdot 2x = 0\cdot 2 = 0$ ; via $v$ : $\;u\cdot 1 = 2$ . Sum: $\partial f/\partial x = 2$ — which matches expanding $f=x^3-x^2y+xy-y^2$ and differentiating directly. Two tiny products beat one messy expansion, and the gap only grows with depth.

Pause & predict

Still at $(x,y)=(1,1)$ : what is $\partial f/\partial y$ ? (Routes: $y\to u\to f$ with $\partial u/\partial y=1$ , and $y\to v\to f$ with $\partial v/\partial y=-1$ .)

In one sentence: Backpropagation is exactly two rules — multiply local derivatives along each path, add the paths — applied without ever walking a path twice.

4

Why you can trust the rules — the chain rule, proved

"Multiply along paths, add across paths" has been doing a lot of work, so it had better be true — a theorem, not a slogan. The proof needs exactly one idea, and it is the idea this course keeps returning to: a derivative is a promise about a straight line. Saying " $g$ has slope $g'(x_0)$ at $x_0$ " means precisely this:

g(x_0+h)=g(x_0)+g'(x_0)\,h+r(h),\qquad\text{where }\ \frac{r(h)}{h}\to 0\ \text{ as }h\to 0.

In words: near $x_0$ the function is a straight line plus a leftover $r(h)$ so small that even after dividing by the tiny number $h$ , it still shrinks to nothing. That last clause is the entire content of differentiability. Hold on to it — both rules fall out of it in a few lines each.

Line plus dying leftover. Near

x_0

the curve

g

(blue) and its tangent (orange) differ by the red gap

r(h)

. Halve the nudge and the gap does not merely shrink — it shrinks faster than $h$ (here

r(h)=0.35\,h^2

), so

r(h)/h

itself goes to zero. That is the whole content of "has a derivative".

Pause & predict

The promise demands $r(h)/h\to0$ , not merely $r(h)\to0$ . What would go wrong if we only asked for $r(h)\to0$ ?

Prove it · Rule 1 — along a path, multiply

Claim. If $u=g(x)$ and $y=f(u)$ , then at $x_0$ (writing $u_0=g(x_0)$ ): $\ \dfrac{dy}{dx}=f'(u_0)\,g'(x_0)$ — the two slopes multiply.

1

Nudge the input by

h

and ask how much

u

moves. That is

g

's own promise:

\Delta u \;=\; g(x_0+h)-g(x_0)\;=\;g'(x_0)\,h+r_g(h)

Line plus leftover, with

r_g(h)/h\to0

. Nothing new — just the definition, applied to

g

.

2

Feed that movement to

f

. That is

f

's promise, written at

u_0

with nudge

\Delta u

:

f(u_0+\Delta u)\;=\;f(u_0)+f'(u_0)\,\Delta u+r_f(\Delta u)

Same definition, applied to

f

. Note the nudge

f

receives is

\Delta u

, not

h

.

3

Substitute step 1 into step 2 and multiply out:

f\big(g(x_0+h)\big)=f(u_0)+\underbrace{f'(u_0)\,g'(x_0)}_{\text{the clean slope}}\,h\;+\;\underbrace{f'(u_0)\,r_g(h)+r_f(\Delta u)}_{\text{call it }R(h)}

One term proportional to

h

, and everything else swept into a single bracket

R(h)

.

4

Check that

R(h)

is a legal leftover — that

R(h)/h\to0

. Piece by piece:

\frac{f'(u_0)\,r_g(h)}{h}\to f'(u_0)\cdot 0=0,\qquad \frac{r_f(\Delta u)}{h}=\frac{r_f(\Delta u)}{\Delta u}\cdot\frac{\Delta u}{h}\to 0\cdot g'(x_0)=0

The first piece is a constant times something dying. The second:

\Delta u/h\to g'(x_0)

by step 1, while

r_f(\Delta u)/\Delta u\to0

because

\Delta u\to0

. (Fine point: if

\Delta u

happens to be exactly

0

, then

r_f(\Delta u)=r_f(0)=0

and the term is already zero — no division needed.)

5

Read off the conclusion. We have shown

f\big(g(x_0+h)\big)=f\big(g(x_0)\big)+\big[f'(u_0)\,g'(x_0)\big]\,h+R(h),\qquad \frac{R(h)}{h}\to0

— one clean slope, one dying leftover. That is exactly the shape of a derivative's promise, so the number in the bracket is the derivative of the composite. ∎ Two promises composed make a third promise, and slopes multiply while the leftovers stay harmless. "Multiply along a path" is now a theorem.

Now the second rule. Suppose $f$ reaches the output through several intermediates at once — $f$ is a function of $u_1,\dots,u_m$ , and each $u_k$ is a function of $x$ . This is precisely the fork situation in the graph: one input, many routes.

Prove it · Rule 2 — across paths, add

Claim. $\ \dfrac{df}{dx}=\displaystyle\sum_{k=1}^{m}\frac{\partial f}{\partial u_k}\,\frac{du_k}{dx}$ — each route contributes its product, and the contributions add.

1

The many-input promise (Unit 6's total derivative): nudging all inputs at once,

f(u_1+\Delta u_1,\dots,u_m+\Delta u_m)=f(\mathbf u)+\sum_{k}\frac{\partial f}{\partial u_k}\,\Delta u_k+r,\qquad \frac{r}{\|\Delta \mathbf u\|}\to0

A multivariable derivative is a plane-promise: one slope per input, one shared leftover that dies faster than the total nudge. This is what "differentiable" means in several variables.

2

Each intermediate keeps its own one-variable promise:

\Delta u_k=\frac{du_k}{dx}\,h+r_k(h),\qquad \frac{r_k(h)}{h}\to0

Every route's first leg is an honest derivative —

m

copies of step 1 from Rule 1.

3

Substitute step 2 into step 1 and collect the terms proportional to

h

:

f(\cdots)=f(\mathbf u)+\Big[\sum_{k}\frac{\partial f}{\partial u_k}\frac{du_k}{dx}\Big]h+R(h),\qquad R(h)=\sum_k\frac{\partial f}{\partial u_k}\,r_k(h)+r

The bracket is the claimed slope. Everything not proportional to

h

— one leftover per route, plus the shared plane-leftover

r

— is swept into

R(h)

. Nothing has been dropped yet.

4

The route leftovers die: for each

k

,

\frac{1}{h}\cdot\frac{\partial f}{\partial u_k}\,r_k(h)=\frac{\partial f}{\partial u_k}\cdot\frac{r_k(h)}{h}\to\frac{\partial f}{\partial u_k}\cdot 0=0

A fixed number times something that dies — the same move as step 4 of Rule 1, once per route.

5

The shared leftover dies too. By step 2,

\|\Delta\mathbf u\|/h\to\|\mathbf u'(x)\|

, where

\mathbf u'=(du_1/dx,\dots,du_m/dx)

. So

\frac{r}{h}=\frac{r}{\|\Delta\mathbf u\|}\cdot\frac{\|\Delta\mathbf u\|}{h}\to 0\cdot\|\mathbf u'(x)\|=0

The constant is

\|\mathbf u'(x)\|

, the size of the combined nudge per unit

h

. Fine point: if

\Delta\mathbf u

is exactly

\mathbf 0

then

r=r(\mathbf 0)=0

and the term is already gone — no division needed. Hence

R(h)/h\to0

.

6

Same shape, same conclusion: the composite keeps a promise whose slope is the bracket — the sum over routes of (slope out) × (slope in). ∎ "Add across paths" is not a new fact about graphs. It is what a multi-input derivative means, read one route at a time.

The bridge

The two rules are matrix multiplication in disguise. For vector maps $\mathbf x\xrightarrow{F}\mathbf u\xrightarrow{G}\mathbf y$ , the chain rule reads $J_{G\circ F}=J_G\,J_F$ — Jacobians multiply. Now look at one entry of that product, using the row-times-column rule:

\big(J_G J_F\big)_{ij}=\sum_{k}\big(J_G\big)_{ik}\big(J_F\big)_{kj}

Each term is one path: input $j$ nudges intermediate $k$ (that is $(J_F)_{kj}$ ), which nudges output $i$ (that is $(J_G)_{ik}$ ) — multiply along the path. The $\sum_k$ adds one product per intermediate stop — add across paths. The row-times-column rule you have used since Unit 1 is literally the two rules of backpropagation, written at once. Unit 6 promised "the chain rule becomes matrix multiplication"; you have now watched the promise come true, entry by entry.

One entry of a Jacobian product, drawn. The sensitivity of output

y_1

to input

x_2

is

\sum_k (J_G)_{1k}(J_F)_{k2}

. Each orange route

x_2\to u_k\to y_1

is one term: multiply its two edge slopes (Rule 1); the three routes add (Rule 2). The row-times-column rule is those two rules at once.

Pause & predict

In $\big(J_G J_F\big)_{ij}=\sum_k (J_G)_{ik}(J_F)_{kj}$ , what is one single term $(J_G)_{ik}(J_F)_{kj}$ of the sum, in graph language?

In one sentence: A derivative is a "line plus dying leftover" promise; composing promises multiplies slopes (Rule 1), parallel routes add their products (Rule 2), and matrix multiplication's row-times-column formula is both rules written at once.

5

The recipe, written once and for all

The lecture states the two rules formally, and it is worth reading slowly because this half-line is automatic differentiation. Number every quantity in the graph: $x_1,\dots,x_d$ are the inputs, $x_{d+1},\dots,x_{D-1}$ the intermediate baby steps, $x_D$ the output. Each is computed from its parents by an elementary function:

x_i = g_i\big(x_{\mathrm{Pa}(x_i)}\big)

Now walk backwards. The output blames itself fully, and every other node collects blame from the nodes it feeds — its children:

\frac{\partial f}{\partial x_D}=1,\qquad \frac{\partial f}{\partial x_i}=\sum_{j:\,x_i\in \mathrm{Pa}(x_j)}\frac{\partial f}{\partial x_j}\,\frac{\partial g_j}{\partial x_i}

Read it like a postal system: node $j$ already knows its own blame $\partial f/\partial x_j$ . It mails each parent $i$ a copy, scaled by the local derivative $\partial g_j/\partial x_i$ of its own baby step. A parent with several children just adds up its mail. One sweep from right to left and every node in the graph — every input, every weight — holds its derivative.

Two things make this cheap. Each blame $\partial f/\partial x_j$ is computed once and reused by all of $j$ 's parents. And each local derivative $\partial g_j/\partial x_i$ is trivial, because each baby step is trivial — the derivative of $e^a$ or $\sqrt c$ or $a+b$ , never of the whole monster. This is why the lecture can conclude: computing all the derivatives costs about the same as computing the function itself.

Prove it · the recipe is correct

Claim. Process the nodes from right to left; then the recipe's $\bar x_i$ equals the true $\partial f/\partial x_i$ for every node in the graph.

1

Base case. The rightmost node:

\bar x_D=1=\partial f/\partial f

— the output's sensitivity to itself. Nudge

f

by

h

and

f

changes by

h

. Slope 1, no leftover at all.

2

The order does the bookkeeping. Working right to left means: by the time we reach node

x_i

, every child

x_j

(every node that uses

x_i

) has already been processed and — by induction — already holds its correct blame

\bar x_j=\partial f/\partial x_j

. Children sit to the right of their parents in the graph, so a right-to-left sweep always finds the mail ready before it must be sent.

3

Apply Rule 2 at node $x_i$ , with its children as the intermediates. Any nudge of

x_i

can reach

f

only through the children, so §4's theorem says

\frac{\partial f}{\partial x_i}=\sum_{j:\,x_i\in\mathrm{Pa}(x_j)}\frac{\partial f}{\partial x_j}\,\frac{\partial g_j}{\partial x_i}

— and every factor on the right is one the recipe possesses: the child's (already correct) blame, times the local slope of the child's baby step. So the value the recipe writes into

\bar x_i

is the true derivative. Right-to-left induction covers the whole graph. ∎ The recipe is not a new algorithm to take on trust — it is Rule 2, applied once per node, in an order that guarantees the mail is always ready.

Pause & predict

In the recipe, the sum for $\partial f/\partial x_i$ runs over the children of $x_i$ — the nodes that use it. Why children, not parents?

In one sentence: Set

\bar f=1

, then let every node mail each parent its own blame times a local derivative — one right-to-left sweep prices every knob.

6

One neuron, trained by your own hands

Time to point this machinery at something that learns. The smallest possible network: one neuron with one weight $w$ and one bias $b$ , one input $x$ , one target $y$ :

z=wx+b,\qquad a=\tanh(z),\qquad L=\tfrac12\,(a-y)^2

Forward: three baby steps to the loss. Backward: the blame walks home through the same three steps. The key arrival is $\delta=\partial L/\partial z$ — the blame reaching the pre-activation — because every parameter feeding $z$ then reads its gradient straight off $\delta$ : $\partial L/\partial w=\delta\,x$ , $\partial L/\partial b=\delta$ , $\partial L/\partial x=\delta\,w$ . Compute $\delta$ once, hand out copies. Then take an actual gradient step and watch the loss fall:

Try: predict the sign of $\delta$ before each step; then set the target to $y=-1$ and watch the same machinery push $w$ the other way.

drag the landscape to orbit it

input x1

target y1

—

Starting point is the practice set's own numbers: $w=0.5,\ b=0,\ x=1,\ y=1$ , so $z=0.5$ , $a=\tanh 0.5=0.4621$ , $L=0.1447$ , and the backward pass gives $\delta=(a-y)(1-a^2)=-0.4230$ , hence $\partial L/\partial w=-0.4230$ , $\partial L/\partial b=-0.4230$ . Both gradients are negative — the loss wants $w$ and $b$ bigger. Press the step button and watch it happen; press train and watch the loss meter dive.

Prove it · every number in the widget, from scratch

Claim. $\delta=(a-y)(1-a^2)$ . Nothing to memorise — derive each local slope, then chain them with Rule 1.

1

The loss.

L(a)=\tfrac12(a-y)^2

. Nudge

a

by

\Delta

and expand the square:

\tfrac12(a+\Delta-y)^2=\tfrac12(a-y)^2+(a-y)\,\Delta+\tfrac12\Delta^2\quad\Rightarrow\quad \frac{\partial L}{\partial a}=a-y

The

\tfrac12\Delta^2

is the dying leftover. And now you can see why the

\tfrac12

was planted in the loss: it exists purely to eat the 2 that squaring brings down.

2

The activation.

a=\tanh z=\dfrac{s}{c}

with

s=e^z-e^{-z}

,

c=e^z+e^{-z}

. Notice the pair swaps under differentiation:

s'=c

and

c'=s

. Quotient rule:

\Big(\frac{s}{c}\Big)'=\frac{s'c-sc'}{c^2}=\frac{c^2-s^2}{c^2}=1-\Big(\frac{s}{c}\Big)^2=1-\tanh^2 z=1-a^2

The derivative comes out written in terms of the output

a

— which the forward pass already computed. So the backward pass buys

1-a^2

for one multiplication, no new function calls.

3

The pre-activation.

z=wx+b

is linear, and a linear function's slopes are just its coefficients:

\frac{\partial z}{\partial w}=x,\qquad \frac{\partial z}{\partial b}=1,\qquad \frac{\partial z}{\partial x}=w

Nudge

w

by

\Delta

and

z

moves by

x\Delta

exactly — not even a leftover this time.

4

Chain them (Rule 1, twice).

\delta=\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\cdot\frac{da}{dz}=(a-y)(1-a^2)

With the widget's starting numbers,

a-y=0.4621-1=-0.5379

and

1-a^2=0.7864

:

\delta=(-0.5379)(0.7864)=-0.4230

then

\ \partial L/\partial w=\delta\,x,\ \ \partial L/\partial b=\delta,\ \ \partial L/\partial x=\delta\,w

. ∎ Every figure the widget displays is these four lines evaluated — press the buttons above and check any of them by hand.

Pause & predict

At the starting numbers, $\partial L/\partial w$ and $\partial L/\partial b$ are both exactly $\delta\cdot x$ and $\delta\cdot 1$ with $\delta=-0.423$ . What made them so cheap to get?

Pause & predict

Set the target to $y=-1$ instead. Before pressing anything: which way will the gradient step now push $w$ ?

In one sentence: Compute the blame

\delta=\partial L/\partial z

once, and every parameter feeding

z

reads its gradient off it — then step downhill and the loss actually falls.

7

A whole layer at once — backprop in matrix form

A real layer is the same neuron, vectorised: $m$ neurons share the input vector $\mathbf{x}\in\mathbb{R}^n$ ,

Z=A\mathbf{x}+\mathbf{b},\qquad \mathbf{a}=\sigma(Z)

with $A\in\mathbb{R}^{m\times n}$ the weights, $\mathbf b\in\mathbb{R}^m$ the biases, and the sigmoid $\sigma(z)=\tfrac{1}{1+e^{-z}}$ applied entry by entry. The blame arriving from above is $\boldsymbol\delta^{(a)}=\partial\mathrm{Loss}/\partial \mathbf a$ , one entry per neuron.

Read the answer as people first, then as symbols. Passing blame back through the layer gives four lines. Through the sigmoid: each neuron's blame is scaled by its own slope — no mixing between neurons. A weight: gets the blame of its neuron times the activity of its input — big only when both are big. A bias: gets its neuron's blame straight. The input: gets blame collected back through the weights, and that becomes the mail for the layer below.

Now the symbols. Step one, through the sigmoid: $a_i=\sigma(Z_i)$ touches only its own $Z_i$ , so the Jacobian $\partial\mathbf a/\partial Z$ is diagonal — $\operatorname{diag}\!\big(\mathbf a\odot(1-\mathbf a)\big)$ , using $\sigma'=\sigma(1-\sigma)$ — and the blame is scaled entrywise ( $\odot$ is the Hadamard, entrywise, product):

\boldsymbol\delta^{(Z)}=\boldsymbol\delta^{(a)}\odot \mathbf a\odot(1-\mathbf a)

Step two, through $Z=A\mathbf x+\mathbf b$ . Since $Z_i=\sum_j A_{ij}x_j+b_i$ , each weight sees $\partial Z_i/\partial A_{ij}=x_j$ , and the chain rule assembles into three matrix formulas:

\begin{gathered}\frac{\partial\mathrm{Loss}}{\partial A}=\boldsymbol\delta^{(Z)}\mathbf{x}^{\top},\qquad \frac{\partial\mathrm{Loss}}{\partial \mathbf b}=\boldsymbol\delta^{(Z)},\\[.4em] \frac{\partial\mathrm{Loss}}{\partial \mathbf x}=A^{\top}\boldsymbol\delta^{(Z)}\end{gathered}

The weights: $\boldsymbol\delta^{(Z)}\mathbf x^\top$ is an outer product — blame of my row, times activity of my column. The biases: the blame itself. The input: $A^\top\boldsymbol\delta^{(Z)}$ re-mixes the blame back through the transposed weights — and this vector is precisely the $\boldsymbol\delta^{(a)}$ the previous layer receives as its mail. Layer after layer, the same four lines: that is deep learning's inner loop.

Prove it · where σ(1−σ) comes from

Claim. $\sigma(z)=\dfrac{1}{1+e^{-z}}$ satisfies $\sigma'=\sigma(1-\sigma)$ — so the backward pass needs no new function evaluations.

1

Differentiate with Rule 1:

\sigma=(1+e^{-z})^{-1}

, outer slope

-(1+e^{-z})^{-2}

, inner slope

-e^{-z}\!:

\sigma'=-(1+e^{-z})^{-2}\cdot(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^{2}}

The two minus signs cancel — sigmoid always rises.

2

Split the fraction into two factors you can recognise:

\frac{e^{-z}}{(1+e^{-z})^{2}}=\underbrace{\frac{1}{1+e^{-z}}}_{=\ \sigma}\cdot\underbrace{\frac{e^{-z}}{1+e^{-z}}}_{=\ 1-\sigma} \qquad\text{since}\quad 1-\sigma=\frac{(1+e^{-z})-1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}

So

\sigma'=\sigma(1-\sigma)

. ∎ Output in, derivative out:

\mathbf a\odot(1-\mathbf a)

uses only the

\mathbf a

the forward pass cached. This is why frameworks store activations — the way back is pure arithmetic.

Prove it · why the sigmoid step is entrywise — the Jacobian is diagonal

Claim. $\dfrac{\partial\mathbf a}{\partial Z}=\operatorname{diag}\!\big(\mathbf a\odot(1-\mathbf a)\big)$ , and therefore $\boldsymbol\delta^{(Z)}=\boldsymbol\delta^{(a)}\odot\mathbf a\odot(1-\mathbf a)$ — a multiplication per neuron, no matrix at all.

1

Who depends on whom.

a_i=\sigma(Z_i)

reads exactly one input,

Z_i

. Nudge a different

Z_j

and

a_i

does not move:

\frac{\partial a_i}{\partial Z_j}=0\quad(i\neq j)

Every off-diagonal entry of the Jacobian is zero because the activation never mixes neurons — it is applied one entry at a time.

2

The diagonal. On its own input,

a_i

is the one-variable function

\sigma

, and the box above gives its slope:

\frac{\partial a_i}{\partial Z_i}=\sigma'(Z_i)=\sigma(Z_i)\big(1-\sigma(Z_i)\big)=a_i(1-a_i)

Steps 1 and 2 together: a matrix with

a_i(1-a_i)

down the diagonal and

0

everywhere else —

\operatorname{diag}(\mathbf a\odot(1-\mathbf a))

.

3

Chain it (Rule 1 in matrix form). The blame on

Z

is the blame on

\mathbf a

times this Jacobian; multiplying a row by a diagonal matrix scales each entry:

\delta^{(Z)}_i=\sum_j \delta^{(a)}_j\,\frac{\partial a_j}{\partial Z_i}=\delta^{(a)}_i\,a_i(1-a_i)

The sum over

j

is Rule 2, but only the

j=i

term survives (step 1) — so the "sum over routes" has one route per neuron. Written for all

i

at once, that is the

\odot

formula.

4

Check on the widget's numbers.

\mathbf a=(0.622,\,0.953)

gives

\mathbf a\odot(1-\mathbf a)=(0.235,\,0.045)

; with

\boldsymbol\delta^{(a)}=(0.6,\,-0.4)

,

\boldsymbol\delta^{(Z)}=(0.6\cdot0.235,\ \ -0.4\cdot0.045)=(0.141,\,-0.018)

— exactly what stage 3 below displays. ∎ Neuron 2 sits at

a=0.953

, nearly saturated, so its factor

0.045

throttles almost all of its blame. That is the vanishing-gradient mechanism, in one entry of a diagonal matrix.

Try: reach stage 4, then slide $\delta^{(a)}_1$ — the top row of the outer product and the mail $A^{\top}\boldsymbol\delta^{(Z)}$ react; row 2 does not.

δ⁽ᵃ⁾₁ from above0.6

—

The layer is $2\times2$ so every number fits on screen: $A=$ 1−120, $\mathbf b=(0,\,1)$ , $\mathbf x=(1,\,0.5)$ , and the loss above sends down $\boldsymbol\delta^{(a)}=(0.6,\,-0.4)$ . Watch stage 4: $\partial L/\partial A$ is literally the outer-product grid of $\boldsymbol\delta^{(Z)}$ against $\mathbf x$ — and $A^{\top}\boldsymbol\delta^{(Z)}$ leaves the widget as the mail for the layer below.

Prove it · the three matrix formulas, entry by entry

Claim. The matrix formulas hide no new laws — they are Rule 1 and Rule 2 aimed at each entry, then reassembled into matrix shape. Write $\boldsymbol\delta=\boldsymbol\delta^{(Z)}$ for short.

1

A weight $A_{ij}$ . Scan the forward pass

Z_k=\sum_p A_{kp}x_p+b_k

: the entry

A_{ij}

appears in exactly one place — the row

k=i

, multiplied by

x_j

. One appearance means one path, so Rule 1 alone:

\frac{\partial L}{\partial A_{ij}}=\delta_i\cdot\frac{\partial Z_i}{\partial A_{ij}}=\delta_i\,x_j

"Blame of my row, times activity of my column" — now proved, not just recited.

2

Assemble. The matrix whose

(i,j)

entry is

\delta_i x_j

is, by definition, the outer product:

\frac{\partial L}{\partial A}=\boldsymbol\delta\,\mathbf x^{\top}

Same shape as

A

, one honest entry per weight, ready for

A\leftarrow A-\eta\,\partial L/\partial A

.

3

A bias $b_i$ . It appears only in

Z_i

, with slope

1

:

\ \partial L/\partial b_i=\delta_i\cdot1

, so

\partial L/\partial\mathbf b=\boldsymbol\delta

. The bias rides its own row and nothing else — blame arrives undiluted.

4

An input $x_j$ — the fork.

x_j

appears in every row:

Z_1,\dots,Z_m

all use it.

m

appearances,

m

paths — Rule 2:

\frac{\partial L}{\partial x_j}=\sum_{i=1}^{m}\delta_i\,\frac{\partial Z_i}{\partial x_j}=\sum_{i=1}^{m}\delta_i\,A_{ij}=\big(A^{\top}\boldsymbol\delta\big)_j

Summing

\delta_i A_{ij}

over

i

walks down column

j

of

A

— and reading columns is exactly what the transpose does. The mysterious

A^{\top}

is Rule 2 wearing matrix clothes.

5

All three formulas recovered from two rules and one scan of where each variable appears. ∎ This "find every appearance, add a term per appearance" habit is the entire skill of differentiating matrix expressions. The very next section runs on it.

Pause & predict

A layer has $A\in\mathbb{R}^{100\times 300}$ . What is the shape of $\partial\mathrm{Loss}/\partial A=\boldsymbol\delta^{(Z)}\mathbf{x}^{\top}$ , and why was that inevitable?

Why the transpose. Forward, neuron

i

reads along row

i

— that is what

A\mathbf x

does. Backward, input

j

must collect blame from every neuron it fed, which means reading down column

j

— and reading columns of

A

is reading rows of

A^{\top}

. Same numbers, opposite direction.

Pause & predict

Why does the blame leave through $A^{\top}$ — the transpose — rather than through $A$ itself?

In one sentence:

\boldsymbol\delta^{(Z)}=\boldsymbol\delta^{(a)}\!\odot\mathbf a\odot(1-\mathbf a)

, then weights get

\boldsymbol\delta^{(Z)}\mathbf x^{\top}

, biases get

\boldsymbol\delta^{(Z)}

, and the layer below gets

A^{\top}\boldsymbol\delta^{(Z)}

— four lines, all of deep learning's inner loop.

8

The gradient cookbook — five identities you'll use forever

Backprop needs local derivatives, and in matrix land a handful of them cover almost everything. The lecture lists five; learn them as sentences, not just symbols. (Gradients here are rows, Unit 6's convention.)

Identity	Say it in words
$\dfrac{\partial\, \mathbf{x}^{\top}\mathbf{a}}{\partial \mathbf{x}}=\mathbf{a}^{\top}$	a dot product's sensitivity to $\mathbf x$ is just the other vector
$\dfrac{\partial\, \mathbf{a}^{\top}\mathbf{x}}{\partial \mathbf{x}}=\mathbf{a}^{\top}$	same thing — a dot product doesn't care about the order
$\dfrac{\partial\, \mathbf{a}^{\top}X\mathbf{b}}{\partial X}=\mathbf{a}\mathbf{b}^{\top}$	each entry $X_{ij}$ is multiplied by $a_i b_j$ — so the gradient is that outer product
$\dfrac{\partial\, \mathbf{x}^{\top}B\mathbf{x}}{\partial \mathbf{x}}=\mathbf{x}^{\top}\!\big(B+B^{\top}\big)$	$\mathbf x$ appears twice, so you get two copies — $B$ once from each side
$\dfrac{\partial}{\partial \mathbf{s}}(\mathbf x-A\mathbf s)^{\top}W(\mathbf x-A\mathbf s)=-2(\mathbf x-A\mathbf s)^{\top}WA$	for symmetric $W$ : least-squares' own gradient — chain rule through the residual

Notice the family resemblance to Section 7: the third identity is why $\partial\mathrm{Loss}/\partial A$ came out as an outer product, and the last one is the loss $\|\mathbf y-A\mathbf s\|^2$ that trains linear models — you will meet it again the moment optimisation starts in Unit 9. Below, every identity is derived in full — the whole cookbook takes four short arguments. The arena's Problem 7 re-checks two of them with concrete numbers, and Problem 10 puts the cookbook to work on a real model.

Prove it · the whole cookbook, four short derivations

Claim. Every identity in the table is entry-wise differentiation, organised. The method is always the same: find every appearance of the variable, add one term per appearance.

1

Dot product.

\mathbf x^{\top}\mathbf a=\sum_i x_i a_i

. Differentiate with respect to one coordinate

x_k\!:

every term is constant except

x_k a_k

, whose slope is

a_k

. Collect the coordinates into a row:

\frac{\partial(\mathbf x^{\top}\mathbf a)}{\partial\mathbf x}=\mathbf a^{\top}

And

\mathbf a^{\top}\mathbf x

is the same scalar, so the table's second line comes free.

2

The sandwich.

\mathbf a^{\top}X\mathbf b=\sum_{i,j}a_i X_{ij}b_j

. The entry

X_{ij}

appears once, with coefficient

a_ib_j\!:

\frac{\partial(\mathbf a^{\top}X\mathbf b)}{\partial X_{ij}}=a_ib_j\quad\Rightarrow\quad \frac{\partial(\mathbf a^{\top}X\mathbf b)}{\partial X}=\mathbf a\mathbf b^{\top}

The same one-appearance argument that made

\partial L/\partial A

an outer product in the layer proof — it is literally the same fact.

3

The quadratic — where $B+B^{\top}$ is born.

\mathbf x^{\top}B\mathbf x=\sum_{i,j}B_{ij}\,x_i\,x_j

, and

\mathbf x

appears twice in each term. Product rule on

x_ix_j

, with respect to

x_k

: the left copy fires when

i=k

, contributing

\sum_j B_{kj}x_j=(B\mathbf x)_k

; the right copy fires when

j=k

, contributing

\sum_i B_{ik}x_i=(B^{\top}\mathbf x)_k

. Add the two:

\frac{\partial(\mathbf x^{\top}B\mathbf x)}{\partial x_k}=(B\mathbf x)_k+(B^{\top}\mathbf x)_k\quad\Rightarrow\quad \frac{\partial(\mathbf x^{\top}B\mathbf x)}{\partial\mathbf x}=\mathbf x^{\top}\big(B+B^{\top}\big)

Two appearances of

\mathbf x

, two donated terms — the "mysterious"

+B^{\top}

is Rule 2's fork, hiding in plain algebra. For symmetric

B

it collapses to

2\mathbf x^{\top}B

.

4

Least squares — a chain of the previous lines. Set the residual

\mathbf r=\mathbf x-A\mathbf s

and

E=\mathbf r^{\top}W\mathbf r

with

W

symmetric. Rule 1 with

\mathbf r

as the intermediate:

\begin{aligned}\frac{\partial E}{\partial\mathbf s}&=\frac{\partial E}{\partial\mathbf r}\,\frac{\partial\mathbf r}{\partial\mathbf s} =\underbrace{\mathbf r^{\top}(W+W^{\top})}_{\text{line 3}}\cdot\underbrace{(-A)}_{\mathbf r\text{ linear in }\mathbf s}\\[.3em] &=2\,\mathbf r^{\top}W\cdot(-A)=-2(\mathbf x-A\mathbf s)^{\top}WA\end{aligned}

The

2

came from

W+W^{\top}

under symmetry; the minus came from the

-A\mathbf s

inside the residual. Nothing to memorise once you can rebuild it in two lines. ∎

Pause & predict

With $B=$ 1234 (not symmetric), what is $\dfrac{\partial\,\mathbf x^{\top}B\mathbf x}{\partial \mathbf x}$ at $\mathbf x=(x_1,x_2)$ ?

Pause & predict

In the least-squares gradient $-2(\mathbf x-A\mathbf s)^{\top}WA$ , where exactly did the factor $2$ come from?

In one sentence: Five identities — dot product, outer product, the

B+B^\top

doubling, and least squares — supply nearly every local derivative a real model needs.

9

Why it's so cheap — one sweep, everything reused

Stack $K$ layers, $f_i=\sigma_i(A_{i-1}f_{i-1}+b_{i-1})$ , and put a loss at the end: $L=\|\mathbf y-f_K\|^2$ . Every layer's parameters $\theta_i=\{A_i,b_i\}$ need a gradient, and the chain rule writes each one as a product that reaches from the loss back to that layer:

\begin{aligned}\frac{\partial L}{\partial\theta_{K-1}}&=\frac{\partial L}{\partial f_K}\frac{\partial f_K}{\partial\theta_{K-1}}\\[.35em] \frac{\partial L}{\partial\theta_{K-2}}&=\frac{\partial L}{\partial f_K}\frac{\partial f_K}{\partial f_{K-1}}\frac{\partial f_{K-1}}{\partial\theta_{K-2}}\\[.35em] \frac{\partial L}{\partial\theta_{i}}&=\frac{\partial L}{\partial f_K}\frac{\partial f_K}{\partial f_{K-1}}\cdots\frac{\partial f_{i+2}}{\partial f_{i+1}}\frac{\partial f_{i+1}}{\partial\theta_{i}}\end{aligned}

Look at the shape of those formulas: each one is the previous one's product, extended by one factor. Computed naively, left to right every time, you would rebuild the same long prefix over and over. Computed once from the loss end — keeping the running product as the $\boldsymbol\delta$ that Section 7 handed backwards — every factor in the whole triangle is touched exactly once:

Prove it · the stacked formula is the recipe of §5, unrolled

Claim. $\dfrac{\partial L}{\partial\theta_i}=\dfrac{\partial L}{\partial f_K}\,\dfrac{\partial f_K}{\partial f_{K-1}}\cdots\dfrac{\partial f_{i+2}}{\partial f_{i+1}}\,\dfrac{\partial f_{i+1}}{\partial\theta_i}$ — no new chain rule, just the postal recipe on a graph that happens to be a line.

1

Draw the graph.

f_{i+1}=\sigma_{i+1}(A_if_i+b_i)

means node

f_{i+1}

has exactly two parents,

f_i

and

\theta_i=\{A_i,b_i\}

, and node

f_i

has exactly one child,

f_{i+1}

. The loss

L

is the single child of

f_K

. A stack of layers is a chain: each activation feeds only the next. Forks exist only where a parameter joins in.

2

Recipe at an activation node. §5's rule sums over children;

f_j

has one, so the sum has one term:

\bar f_K=\frac{\partial L}{\partial f_K},\qquad \bar f_j=\bar f_{j+1}\,\frac{\partial f_{j+1}}{\partial f_j}\quad(j<K)

One child, one piece of mail. Rule 2's sum collapses to a single Rule 1 product.

3

Unroll. Apply step 2 repeatedly, from

K

down to

i+1

:

\bar f_{i+1}=\frac{\partial L}{\partial f_K}\,\frac{\partial f_K}{\partial f_{K-1}}\cdots\frac{\partial f_{i+2}}{\partial f_{i+1}}

Each application appends one more Jacobian on the right. This running product is the

\boldsymbol\delta^{(a)}

that §7 handed down to each layer.

4

Recipe at the parameter node.

\theta_i

has one child,

f_{i+1}

, so

\frac{\partial L}{\partial\theta_i}=\bar\theta_i=\bar f_{i+1}\,\frac{\partial f_{i+1}}{\partial\theta_i}

and substituting step 3 gives the claim. ∎ Every layer's gradient is the same prefix

\bar f_{i+1}

times one local factor — which is exactly why computing the prefixes once, from the loss end, prices the whole stack.

Try: read one row left to right — it is the stacked formula above with real indices; then slide K to 8 and count grey versus blue.

layers K4

—

Grey blocks: factors a naive left-to-right evaluation would recompute. Blue: the factors backprop actually computes — each row just extends the running blame $\boldsymbol\delta$ by one new factor. Naive cost grows like $K^2/2$ ; backprop stays at $K$ . At depth 100 that is a 50× saving on top of the million-× saving over per-knob nudging.

Prove it · why the sweep must start at the loss

Claim. The chain $\dfrac{\partial L}{\partial f_1}=\dfrac{\partial L}{\partial f_K}\,\dfrac{\partial f_K}{\partial f_{K-1}}\cdots\dfrac{\partial f_2}{\partial f_1}$ can be multiplied in either order. One order is about $m$ times cheaper — count it.

1

Shapes first (Unit 6's rule: outputs × inputs).

L

is a scalar, so

\partial L/\partial f_K

is a

1\times m

row. Each layer-to-layer Jacobian

\partial f_{i}/\partial f_{i-1}

is a full

m\times m

matrix, for layers of width

m

. One skinny row at the loss end; big square blocks everywhere else. That asymmetry is the whole story.

2

Multiply from the loss end.

(1\times m)\cdot(m\times m)

costs about

m^2

multiplications — and returns another

1\times m

row. Repeat through all

K

layers:

\text{cost}\approx K\,m^{2},\qquad\text{always carrying just a row}

That carried row is exactly the blame

\boldsymbol\delta

of the layer section — reverse mode never holds anything bigger than one layer's worth of numbers.

3

Multiply from the input end. The first product is

(m\times m)\cdot(m\times m)

: cost

m^3

, and it returns another

m\times m

matrix that must be carried the whole way:

\text{cost}\approx K\,m^{3}

For

m=1000

, that is a thousand times more work — from nothing but multiplying the same factors in the wrong order.

4

Same factors, same answer, wildly different bill. Keeping the scalar-loss end first means every step is a cheap row-times-matrix product. That choice of order is reverse-mode automatic differentiation. ∎ And it explains the fine print to come: reverse mode wins because training has ONE output (the loss) and millions of inputs. Flip that ratio and the other order — forward mode — wins instead.

Pause & predict

Rule of thumb: one full backward pass costs about as much as the forward pass itself. What single fact makes that possible?

In one sentence: All the gradient formulas share their long prefix — carry it once, from the loss end, and the whole triangle of factors costs one sweep.

10

Linearization — the gradient as a stand-in for the function

The companion closes with the other reason gradients matter. Once you own $\nabla f(\mathbf x_0)$ , you own the best linear stand-in for $f$ near $\mathbf x_0$ :

L(\mathbf x)\;\approx\;f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}(\mathbf x-\mathbf x_0)

This is Unit 6's tangent line in vector form — the first two terms of the Taylor series (derived below). It is accurate near $\mathbf x_0$ , drifts as you leave, and it is the reason gradient descent works at all: each step trusts the linear stand-in for one small hop. Try the companion's own example — $f(x)=\sqrt{x^2+9}$ around $x_0=-4$ , where $f(-4)=5$ and $f'(-4)=-\tfrac45$ :

Try: put the probe at twice the distance from the anchor and check that the error roughly quadruples.

✋ drag near the dot to move the anchor · elsewhere to move the probe

—

At anchor $-4$ : $L(x)=5-\tfrac45(x+4)$ , the companion's answer. Slide the probe to $-3.5$ and the stand-in guesses $4.6$ against a true $4.61$ — near the anchor the line is almost the function. Push the probe far away and watch the error column grow: a linearization is a local promise, exactly like Unit 6's Taylor windows.

…and in two variables: the stand-in planef(x, y) = eˣ cos y — the companion's second example, with its tangent plane at the anchor

drag the surface to orbit it

anchor x₀0

anchor y₀0

probe radius0.5

At the origin the gradient is $(1,0)$ , so the plane is $L(x,y)=1+x$ — tilted along $x$ , flat along $y$ , exactly the check below. Drag the anchor away from the origin and the plane re-tilts to whatever the gradient says there. Shrink the probe radius and the red gap collapses like the radius squared.

Prove it · the stand-in is the first two Taylor terms

Claim. The linearization $L(x)=f(x_0)+f'(x_0)(x-x_0)$ is what is left of the Taylor series when every term with $(x-x_0)^2$ or higher is dropped — and the first dropped term tells you the error.

1

Taylor at $x_0$ (Unit 8's subject, used once here): with

h=x-x_0

,

f(x_0+h)=f(x_0)+f'(x_0)\,h+\tfrac12 f''(x_0)\,h^2+\tfrac16 f'''(x_0)\,h^3+\cdots

Keep the first two terms and you are holding

L(x)

exactly. Everything after them is §4's leftover

r(h)

, now written out.

2

The error is led by the $h^2$ term. For small

h

the

h^3

and later terms are smaller still, so

f(x)-L(x)\approx\tfrac12 f''(x_0)\,(x-x_0)^2

Twice the distance, four times the error — the same "leftover dies faster than

h

" you saw drawn in §4.

3

Check on the widget.

f=\sqrt{x^2+9}

, so

f''(x)=\dfrac{9}{(x^2+9)^{3/2}}

and

f''(-4)=\dfrac{9}{125}=0.072

. Probe at

x=-3.5

, i.e.

h=0.5

:

\tfrac12\cdot0.072\cdot0.5^2=0.009\qquad\text{vs. the widget's}\quad 4.6098-4.6=0.0098

The prediction is off only by the

h^3

crumbs. So a gradient does more than point downhill — with

f''

it also prices how far the linear stand-in can be trusted. ∎

Pause & predict

The companion also linearizes $f(x,y)=e^x\cos y$ at $(0,0)$ : value $1$ , gradient $(1,0)$ . What is the stand-in plane?

In one sentence:

f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}(\mathbf x-\mathbf x_0)

is the best linear stand-in near

\mathbf x_0

— the first two Taylor terms, and the promise each gradient-descent step relies on.

11

Fine print worth knowing

Three ways to differentiate, only one scales. Symbolic differentiation (what you do on paper, what algebra systems do) produces exact formulas — which explode in size for deep compositions. Numerical differentiation nudges the input and divides — one pass per parameter, plus a precision trap you can see below. Automatic differentiation — this unit — is a third thing: exact like symbolic, cheap like a function call, because it differentiates the program, step by step, instead of the formula.

The precision trap. A finite difference $\tfrac{f(x+h)-f(x-h)}{2h}$ has two enemies. Big $h$ : the secant misses the tangent (Unit 6's curvature gap). Tiny $h$ : the computer subtracts two nearly equal numbers and the difference drowns in rounding error. Squeeze $h$ yourself:

Try: slide $h$ down one decade at a time and watch the error fall 100× per decade — until, near $10^{-6}$ , it stops and turns.

step h = 10^−3

—

The function is the monster from §1, at $x=1$ , where autodiff says $df/dx=5.98308$ exactly. Slide $h$ down the valley: the error falls like $h^2$ (curvature fading), bottoms out near $h\approx10^{-6}$ for this $f$ (nearer $10^{-5}$ for a tamer one), then rises again as rounding noise takes over. Frameworks still use this as a gradient checker — a sanity test for hand-written backward passes — but never for training.

Prove it · the V-shape has a formula

Claim. The checker's error falls like $h^2$ , rises like $1/h$ , and bottoms out at a best step $h_*$ of order $10^{-6}$ – $10^{-5}$ — all three facts derived, not observed.

1

Taylor both evaluations around

x

(Unit 6's tool, Unit 8's whole subject):

f(x\pm h)=f(x)\pm f'(x)h+\tfrac{f''(x)}{2}h^{2}\pm\tfrac{f'''(x)}{6}h^{3}+\cdots

Same expansion, two signs — writing them together is what makes the next line click.

2

Subtract — the even terms cancel.

f(x+h)-f(x-h)=2f'(x)h+\tfrac{f'''(x)}{3}h^{3}+\cdots

. Divide by

2h

:

\frac{f(x+h)-f(x-h)}{2h}=f'(x)+\frac{f'''(x)}{6}\,h^{2}+\cdots

The

h^2

term is the left wall of the V — slope 2 on log–log axes. (The one-sided version

\frac{f(x+h)-f(x)}{h}

keeps an

\frac{f''}{2}h

term — slope only 1. That is why the checker uses the centred difference.)

3

Rounding. The computer stores

f(x\pm h)

only to relative precision

u\approx10^{-16}

, so the subtraction of two nearly equal numbers carries an absolute error of about

u|f(x)|

; dividing by

2h

turns it into

\text{rounding error}\approx\frac{u\,|f(x)|}{2h}

The

1/h

is the right wall: shrink

h

and you divide the same noise by an ever smaller number.

4

Add the walls and set the slope to zero.

E(h)\approx\frac{|f'''|}{6}h^{2}+\frac{u|f|}{2h}

. Differentiate in

h

:

E'(h)=\frac{|f'''|}{3}\,h-\frac{u\,|f|}{2h^{2}}=0\quad\Longleftrightarrow\quad h^{3}=\frac{3\,u\,|f|}{2\,|f'''|}

A rising wall plus a falling wall has one lowest point, where the two slopes cancel. Multiplying through by

h^2

isolates

h^3

.

5

Read off the best step.

h_{*}=\Big(\frac{3\,u\,|f|}{2\,|f'''|}\Big)^{1/3}\sim u^{1/3}\approx 5\times10^{-6}\quad\text{when }|f'''|\approx|f|

For the monster at

x=1

,

|f|=1.09

but

|f'''|\approx153

, so

h_*\approx1\times10^{-6}

— the huge third derivative pushes the floor left, which is what the widget shows. ∎ And the best possible error is only

\sim u^{2/3}\approx10^{-11}

(about

10^{-10}

here) — while autodiff is exact to full precision. That is the entire case for automatic differentiation, compressed into one minimisation.

$\sigma'$ is free. The identity $\sigma'=\sigma(1-\sigma)$ means the backward pass needs no new function calls — the forward pass already computed $\mathbf a$ , and $\mathbf a\odot(1-\mathbf a)$ is two cheap operations. The same trick works for $\tanh$ ( $1-a^2$ ). Frameworks cache activations on the way forward precisely so the way back is arithmetic only — that cache is the "memory cost of training".

Vanishing gradients, revisited. Each sigmoid layer multiplies the blame by $\mathbf a\odot(1-\mathbf a)\le\tfrac14$ . Fifty layers deep, the blame arriving at layer one has passed fifty such factors — Unit 6's gears compounding — which is why early deep networks would not train, and why ReLU (slope 1 where active) took over.

Reverse is for losses; forward exists too. Everything here swept blame backwards, which prices all inputs of one output — perfect for a scalar loss. There is a mirror-image forward mode that prices all outputs of one input; it wins only when outputs vastly outnumber inputs, which training never is. That is why every deep-learning framework is a reverse-mode machine.

Vanishing gradients, drawn. The loss sits to the right of layer 10. Every sigmoid the blame crosses scales it by

a(1-a)\le\tfrac14

, so by layer 1 the strongest possible signal is

4^-9\approx4\times10^-6

of what left the loss — and saturated neurons (

a\approx0.95

, factor

0.045

) make it far worse. The early layers barely learn. ReLU's slope of

1

where active is the fix.

Which sweep to run. A sweep costs about one pass either way; the question is what one sweep buys. Reverse mode starts at one output and prices every input — millions of weights for one loss. Forward mode starts at one input and prices every output. Training has one output and millions of inputs, so every framework runs reverse.

Pause & predict

In the gradient checker, why does making $h$ ever smaller eventually make the estimate worse?

Forward mode you can hold in your hand — dual numbers. Reverse mode swept blame backwards. Forward mode has an algebraic form so tidy you can run it with pencil and paper: carry every quantity as a pair — its value and its derivative — and teach ordinary arithmetic to update both at once.

Prove it · arithmetic that differentiates itself

Claim. Write each pair $(v,\dot v)$ as $v+\dot v\,\varepsilon$ , where $\varepsilon$ is a new symbol with one rule: $\varepsilon^{2}=0$ . Then ordinary algebra on these "dual numbers" performs exact differentiation.

1

Sums.

(a+\dot a\varepsilon)+(b+\dot b\varepsilon)=(a+b)+(\dot a+\dot b)\varepsilon

. Values add, derivatives add — the sum rule, with no one enforcing it.

2

Products. Multiply out and apply the one rule:

(a+\dot a\varepsilon)(b+\dot b\varepsilon)=ab+(a\dot b+\dot a b)\varepsilon+\dot a\dot b\,\varepsilon^{2}=ab+(a\dot b+\dot a b)\,\varepsilon

\varepsilon^2=0

killed the last term — it is the product of two tiny nudges, the very leftover that §4's promise says must die. What survives is exactly the product rule.

3

Any smooth function. Define

g(a+\dot a\varepsilon)=g(a)+g'(a)\,\dot a\,\varepsilon

— and this is forced, not chosen: Taylor gives

g(a+t)=g(a)+g'(a)t+\tfrac{g''(a)}{2}t^2+\cdots

, and with

t=\dot a\varepsilon

every term from

t^2

onwards contains

\varepsilon^2

and vanishes. The chain rule arrives baked in: whatever

\dot a

is carrying gets multiplied by the local slope

g'(a)

.

4

Run the program on pairs, seeding the input as

x=x_0+1\cdot\varepsilon

(its derivative with respect to itself is 1). Example —

f(x)=x\,e^{x}

at

x_0=1

:

x=(1,\,1)\ \xrightarrow{\ e^{x}\ }\ (e,\,e)\ \xrightarrow{\ \times\, x\ }\ \big(1\cdot e,\ \ 1\cdot e+1\cdot e\big)=(e,\,2e)

So

f(1)=e

and

f'(1)=2e\approx5.4366

— exact, and no derivative formula was ever written down. ∎ Check:

f'=e^{x}(1+x)

gives

2e

. ✓ This is forward-mode autodiff: one sweep prices all outputs of one input — the mirror image of backprop, and the winner when inputs are few and outputs many.

Pause & predict

The single rule $\varepsilon^{2}=0$ makes the product rule appear on its own. What is $\varepsilon^{2}=0$ really encoding?

In one sentence: Autodiff is exact like symbolic and cheap like numerical with neither one's flaw — and the finite-difference valley is why "just use tiny h" was never an option.

12

What to carry forward

One picture generated this entire unit: a function drawn as a graph of baby steps, with values flowing right and blame flowing left. Everything else was careful bookkeeping on that picture.

Idea	The one-line version	Where it returns
Computation graph	Any function as baby steps with arrows	every framework's autograd tape
Backward pass	$\bar f=1$ ; multiply along paths, add at forks	every training step you'll ever run
$\delta$ (the blame)	Computed once at each node, reused by every parameter feeding it	Unit 9 · gradient descent at scale
The four layer formulas	$\delta^{(Z)}=\delta^{(a)}\!\odot a\odot(1{-}a)$ ; $\ \partial_A=\delta^{(Z)}x^{\top}$ ; $\ \partial_b=\delta^{(Z)}$ ; $\ \partial_x=A^{\top}\delta^{(Z)}$	the inner loop of deep learning
Gradient cookbook	Dot → $\mathbf a^\top$ ; $\ \mathbf a^\top X\mathbf b\to \mathbf a\mathbf b^\top$ ; $\ \mathbf x^\top B\mathbf x\to \mathbf x^\top(B{+}B^\top)$	Units 9–15, constantly
Cost of a gradient	All of them, for the price of ≈ one forward pass	why billion-knob training is possible
Linearization	$f(\mathbf x_0)+\nabla f^{\top}(\mathbf x-\mathbf x_0)$ — the local stand-in	Unit 8 · Taylor; Unit 9 · each descent step

The one thing to remember

Blame flows backwards through the same graph that computed the value. Multiply along paths, add at forks, reuse every partial product — and a million derivatives cost one sweep. When a framework call like loss.backward() feels like magic, it is exactly this unit, executed by a for-loop.

13

Practice arena — the unit's problem set, solved in full

Eleven problems: three from Part C of Prof. Saurabh's practice set (Parts A and B — Taylor series and the Hessian — belong to Unit 8's territory and wait there), three from his companion guide, and five written for this arena — the identity drill (Problem 7), the complete numeric backward pass (Problem 8), and three proof workouts (Problems 9–11) that derive from scratch the rules this unit leaned on. Every solution below is machine-verified.

One habit does most of the work: draw the graph first, then walk it backwards — multiply along paths, add at forks, and check every shape before trusting any algebra.

Problem 1medium

A one-layer network computes $\mathbf z=W\mathbf x$ and the loss $g(\mathbf z)=\tfrac12\|\mathbf z-\mathbf t\|^2$ against a target $\mathbf t$ , with $W=\begin{bmatrix}1&2\\0&1\\-1&1\end{bmatrix},\qquad \mathbf x=\begin{bmatrix}1\\1\end{bmatrix},\qquad \mathbf t=\begin{bmatrix}1\\0\\2\end{bmatrix}.$ (a) Compute $\nabla_{\mathbf z}\,g$ (with its dimension) and $J=\partial\mathbf z/\partial\mathbf x$ (with its dimension). (b) Combine them to get $\nabla_{\mathbf x}\,g$ , and give the loss value.

What this tests. One honest backward step through a linear layer: the upstream gradient is the residual, and it returns to the input through $W^{\top}$ .

Show the full solution

Step 1 — forward pass.

\mathbf z=W\mathbf x=(1{+}2,\ 0{+}1,\ -1{+}1)^{\top}=(3,1,0)^{\top}

. Residual

\mathbf r=\mathbf z-\mathbf t=(2,1,-2)^{\top}

, and the loss is

g=\tfrac12(2^2+1^2+(-2)^2)=\tfrac92=4.5

.

(a) Step 2 — gradient at the output. For

g=\tfrac12\|\mathbf z-\mathbf t\|^2

,

\ \nabla_{\mathbf z}\,g=\mathbf z-\mathbf t=\mathbf r=(2,1,-2)^{\top}

— dimension

3\times1

, one entry per output. The Jacobian of a linear map is the matrix itself:

J=\partial(W\mathbf x)/\partial\mathbf x=W

, dimension

3\times2

(outputs × inputs).

(b) Step 3 — chain them.

\nabla_{\mathbf x}\,g=J^{\top}\nabla_{\mathbf z}\,g=W^{\top}\mathbf r=\begin{bmatrix}1&0&-1\\2&1&1\end{bmatrix}\begin{bmatrix}2\\1\\-2\end{bmatrix}=\begin{bmatrix}1\cdot2+0\cdot1+(-1)(-2)\\2\cdot2+1\cdot1+1\cdot(-2)\end{bmatrix}=\begin{bmatrix}4\\3\end{bmatrix}

Dimension check:

\nabla_{\mathbf z}g

is

3\times1

,

W^{\top}

is

2\times3

, product

2\times1

— the shape of

\mathbf x

, as a gradient must be. ✓

$\nabla_{\mathbf z}g=(2,1,-2)^{\top}$ (3×1), $J=W$ (3×2), $\nabla_{\mathbf x}g=(4,3)^{\top}$ (2×1), loss $=4.5$ .

Remember

"Backprop through a linear layer" = "multiply the incoming gradient by $W^{\top}$ ". You can also check by substituting first: $g(\mathbf x)=\tfrac12\|W\mathbf x-\mathbf t\|^2$ has gradient $W^{\top}(W\mathbf x-\mathbf t)=W^{\top}\mathbf r$ — same answer, one line.

Problem 2medium

A neuron computes $z=wx+b,\ a=\tanh(z)$ , and the loss $L=\tfrac12(a-y)^2$ . Using $\tanh'(z)=1-\tanh^2(z)$ and the values $w=0.5,\ x=1,\ b=0,\ y=1$ : (a) do the forward pass $(z,a,L)$ ; (b) backpropagate to obtain $\partial L/\partial w,\ \partial L/\partial b,\ \partial L/\partial x$ .

What this tests. The full forward–backward cycle on a single neuron, and the shared blame $\delta$ that Section 6 trains with.

Show the full solution

(a) Forward.

z=0.5\cdot1+0=0.5

;

\ a=\tanh(0.5)=0.4621

;

\ L=\tfrac12(0.4621-1)^2=\tfrac12(-0.5379)^2=0.1447

.

(b) Step 1 — the shared blame. Walk

z\to a\to L

backwards:

\frac{\partial L}{\partial a}=a-y=-0.5379,\qquad \frac{da}{dz}=1-a^2=1-0.2136=0.7864

\delta\;=\;\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\,\frac{da}{dz}=(-0.5379)(0.7864)=-0.4230

Step 2 — hand out copies.

z=wx+b

gives

\partial z/\partial w=x=1

,

\partial z/\partial b=1

,

\partial z/\partial x=w=0.5

:

\frac{\partial L}{\partial w}=\delta\cdot x=-0.4230,\qquad \frac{\partial L}{\partial b}=\delta=-0.4230,\qquad \frac{\partial L}{\partial x}=\delta\cdot w=-0.2115

$z=0.5,\ a=0.4621,\ L=0.1447$ ; $\ \partial L/\partial w=-0.4230,\ \partial L/\partial b=-0.4230,\ \partial L/\partial x=-0.2115$ .

Remember

$\delta$ is computed once and reused three times — the whole economy of backprop in miniature. Both parameter gradients are negative, so a descent step raises $w$ and $b$ , pushing $a$ toward the target $1$ . Section 6's widget runs exactly these numbers.

Problem 3easy

A scalar function is built from two intermediates: $f=uv$ , $\ u=x^2+y$ , $\ v=x-y$ . (a) Write $\partial f/\partial u,\ \partial f/\partial v$ and the four partials of $u,v$ with respect to $x,y$ . (b) Using the chain rule, assemble $\partial f/\partial x$ and $\partial f/\partial y$ , and evaluate both at $(x,y)=(1,1)$ .

What this tests. Multiply along paths, add across paths — the two rules of Section 3, by hand.

Show the full solution

(a) The local derivatives. Outer:

\partial f/\partial u=v

,

\ \partial f/\partial v=u

. Inner:

\partial u/\partial x=2x

,

\ \partial u/\partial y=1

,

\ \partial v/\partial x=1

,

\ \partial v/\partial y=-1

.

(b) Step 1 — two paths from each input.

\frac{\partial f}{\partial x}=\underbrace{\frac{\partial f}{\partial u}\frac{\partial u}{\partial x}}_{\text{via }u}+\underbrace{\frac{\partial f}{\partial v}\frac{\partial v}{\partial x}}_{\text{via }v}=v\,(2x)+u\,(1)=2x(x-y)+(x^2+y)

\frac{\partial f}{\partial y}=v\,(1)+u\,(-1)=(x-y)-(x^2+y)

Step 2 — evaluate at $(1,1)$ where

u=2,\ v=0

:

\frac{\partial f}{\partial x}\Big|_{(1,1)}=0\cdot2+2\cdot1=2,\qquad \frac{\partial f}{\partial y}\Big|_{(1,1)}=0\cdot1+2\cdot(-1)=-2

Step 3 — cross-check by expanding first.

f=(x^2+y)(x-y)=x^3-x^2y+xy-y^2

, so

\partial f/\partial x=3x^2-2xy+y=3-2+1=2

✓ and

\partial f/\partial y=-x^2+x-2y=-1+1-2=-2

✓.

$\partial f/\partial x\,(1,1)=2$ , $\ \partial f/\partial y\,(1,1)=-2$ — by paths and by expansion alike.

Problem 4hard

Define $g(\mathbf z,\boldsymbol\nu):=\log p(\mathbf x,\mathbf z)-\log q(\mathbf z,\boldsymbol\nu)$ with $\mathbf z:=t(\boldsymbol\epsilon,\boldsymbol\nu)$ , for differentiable functions $p,q,t$ and $\mathbf x\in\mathbb{R}^D,\ \mathbf z\in\mathbb{R}^E,\ \boldsymbol\nu\in\mathbb{R}^F,\ \boldsymbol\epsilon\in\mathbb{R}^G$ . Using the chain rule, compute the total derivative $\dfrac{d}{d\boldsymbol\nu}g(\mathbf z,\boldsymbol\nu)$ .

What this tests. Direct and indirect paths of influence — the multivariate chain rule when a variable appears both explicitly and through an intermediate. (This is the gradient inside variational autoencoders.)

Show the full solution

Step 1 — draw the graph.

\boldsymbol\nu

reaches

g

two ways: directly (it sits inside

\log q(\mathbf z,\boldsymbol\nu)

) and indirectly (it builds

\mathbf z=t(\boldsymbol\epsilon,\boldsymbol\nu)

, and

\mathbf z

feeds both

\log p

and

\log q

). Total derivative = sum over both routes.

Step 2 — the direct route. Hold

\mathbf z

fixed and differentiate

g

in the

\boldsymbol\nu

-slot only. The

\log p

term has no direct

\boldsymbol\nu

:

\frac{\partial g}{\partial\boldsymbol\nu}=-\frac{\partial \log q(\mathbf z,\boldsymbol\nu)}{\partial\boldsymbol\nu}\qquad(1\times F)

Step 3 — the indirect route, two links. First how

g

feels

\mathbf z

, then how

\mathbf z

feels

\boldsymbol\nu

:

\frac{\partial g}{\partial\mathbf z}=\frac{\partial \log p(\mathbf x,\mathbf z)}{\partial\mathbf z}-\frac{\partial \log q(\mathbf z,\boldsymbol\nu)}{\partial\mathbf z}\quad(1\times E),\qquad \frac{\partial\mathbf z}{\partial\boldsymbol\nu}=\frac{\partial t(\boldsymbol\epsilon,\boldsymbol\nu)}{\partial\boldsymbol\nu}\quad(E\times F)

Step 4 — add the routes.

\frac{d g}{d\boldsymbol\nu}=-\frac{\partial \log q}{\partial\boldsymbol\nu}\;+\;\left(\frac{\partial \log p}{\partial\mathbf z}-\frac{\partial \log q}{\partial\mathbf z}\right)\frac{\partial t}{\partial\boldsymbol\nu}

Dimension check:

(1\times F)+(1\times E)(E\times F)=(1\times F)

✓.

$\dfrac{dg}{d\boldsymbol\nu}=-\dfrac{\partial\log q}{\partial\boldsymbol\nu}+\Big(\dfrac{\partial\log p}{\partial\mathbf z}-\dfrac{\partial\log q}{\partial\mathbf z}\Big)\dfrac{\partial t}{\partial\boldsymbol\nu}$ .

Remember

Write $d/d\boldsymbol\nu$ (total) for "all routes", $\partial/\partial\boldsymbol\nu$ (partial) for "this slot only, everything else frozen". The whole problem is Section 3's fork rule wearing research-paper notation.

Problem 5medium

Compute $\dfrac{df}{d\mathbf x}$ for $f(z)=\log(1+z)$ , $\ z=\mathbf x^{\top}\mathbf x$ , $\ \mathbf x\in\mathbb{R}^D$ . State the dimension of every partial derivative involved.

What this tests. A two-step graph $\mathbf x\to z\to f$ where the middle is a scalar — plus the cookbook's dot-product identity applied to $\mathbf x^{\top}\mathbf x$ .

Show the full solution

Step 1 — the outer link.

\dfrac{df}{dz}=\dfrac{1}{1+z}

. Dimension

1\times1

.

Step 2 — the inner link.

z=\mathbf x^{\top}\mathbf x=\sum_i x_i^2

, so

\partial z/\partial x_k=2x_k

, assembled as the row

\frac{\partial z}{\partial\mathbf x}=2\mathbf x^{\top}\qquad(1\times D)

(Or by the cookbook:

\mathbf x^\top B\mathbf x

with

B=I

gives

\mathbf x^{\top}(I+I)=2\mathbf x^{\top}

.)

Step 3 — multiply the links.

\frac{df}{d\mathbf x}=\frac{df}{dz}\,\frac{\partial z}{\partial\mathbf x}=\frac{2\mathbf x^{\top}}{1+\mathbf x^{\top}\mathbf x}\qquad(1\times1)(1\times D)=(1\times D)\ \checkmark

Step 4 — sanity check with numbers. At

\mathbf x=(1,2)^{\top}

:

z=5

, so the formula gives

\tfrac{2}{6}(1,2)=(\tfrac13,\tfrac23)

— and differentiating

\log(1+x_1^2+x_2^2)

directly gives

\big(\tfrac{2x_1}{1+z},\tfrac{2x_2}{1+z}\big)=(\tfrac13,\tfrac23)

✓.

$\dfrac{df}{d\mathbf x}=\dfrac{2\mathbf x^{\top}}{1+\mathbf x^{\top}\mathbf x}$ , a $1\times D$ row.

Problem 6medium

Compute $\dfrac{d\mathbf f}{d\mathbf x}$ for $\mathbf f(\mathbf z)=\sin(\mathbf z)$ applied entrywise, $\ \mathbf z=A\mathbf x+\mathbf b$ , with $A\in\mathbb{R}^{E\times D},\ \mathbf x\in\mathbb{R}^{D},\ \mathbf b\in\mathbb{R}^{E}$ . State every dimension.

What this tests. The Jacobian chain rule with an entrywise nonlinearity — the exact structure of Section 7's layer, with $\sin$ in place of $\sigma$ .

Show the full solution

Step 1 — the outer Jacobian.

f_i=\sin(z_i)

touches only its own

z_i

, so off-diagonal partials vanish and

\frac{\partial\mathbf f}{\partial\mathbf z}=\operatorname{diag}\!\big(\cos(z_1),\dots,\cos(z_E)\big)=\operatorname{diag}\!\big(\cos(\mathbf z)\big)\qquad(E\times E)

— the same diagonal shape as Section 7's

\operatorname{diag}(\mathbf a\odot(1-\mathbf a))

, and for the same reason: entrywise functions have diagonal Jacobians.

Step 2 — the inner Jacobian.

\mathbf z=A\mathbf x+\mathbf b

is linear, so

\ \partial\mathbf z/\partial\mathbf x=A

(E\times D)

; the constant

\mathbf b

contributes nothing.

Step 3 — multiply, order matters.

\frac{d\mathbf f}{d\mathbf x}=\frac{\partial\mathbf f}{\partial\mathbf z}\,\frac{\partial\mathbf z}{\partial\mathbf x}=\operatorname{diag}\!\big(\cos(A\mathbf x+\mathbf b)\big)\,A\qquad(E\times E)(E\times D)=(E\times D)\ \checkmark

Concretely: row

i

of

A

, scaled by

\cos(z_i)

.

$\dfrac{d\mathbf f}{d\mathbf x}=\operatorname{diag}\!\big(\cos(A\mathbf x+\mathbf b)\big)A$ , an $E\times D$ Jacobian.

Problem 7medium

Verify two cookbook identities the honest way, entry by entry. (a) For $B=\begin{bmatrix}1&2\\3&4\end{bmatrix}$ , expand $\mathbf x^{\top}B\mathbf x$ and confirm $\partial(\mathbf x^{\top}B\mathbf x)/\partial\mathbf x=\mathbf x^{\top}(B+B^{\top})$ . (b) For $\mathbf a=(1,2)^{\top},\ \mathbf b=(3,1)^{\top}$ , expand $\mathbf a^{\top}X\mathbf b$ and confirm $\partial(\mathbf a^{\top}X\mathbf b)/\partial X=\mathbf a\mathbf b^{\top}$ .

What this tests. That the identities are nothing but organised entry-wise differentiation — once verified by hand, you may use them forever with a clear conscience.

Show the full solution

(a) Step 1 — expand the scalar.

\mathbf x^{\top}B\mathbf x=x_1^2+2x_1x_2+3x_2x_1+4x_2^2=x_1^2+5x_1x_2+4x_2^2

Step 2 — differentiate like any polynomial.

\partial/\partial x_1=2x_1+5x_2

,

\ \partial/\partial x_2=5x_1+8x_2

, so the gradient row is

[\,2x_1+5x_2,\ 5x_1+8x_2\,]

.

Step 3 — the identity's answer.

B+B^{\top}=\begin{bmatrix}2&5\\5&8\end{bmatrix}

, and

\mathbf x^{\top}(B+B^{\top})=[\,2x_1+5x_2,\ 5x_1+8x_2\,]

— identical. ✓ (Note

2B

would give

[2x_1{+}4x_2,\ 6x_1{+}8x_2]

— wrong unless

B

is symmetric.)

(b) Step 1 — expand. With

\mathbf a=(1,2)^{\top},\ \mathbf b=(3,1)^{\top}

:

\mathbf a^{\top}X\mathbf b=\sum_{i,j}a_i X_{ij} b_j=3X_{11}+X_{12}+6X_{21}+2X_{22}

Step 2 — differentiate per entry.

\partial/\partial X_{ij}=a_ib_j

: the four coefficients

3,1,6,2

arranged in

X

's own shape are

\frac{\partial(\mathbf a^{\top}X\mathbf b)}{\partial X}=\begin{bmatrix}3&1\\6&2\end{bmatrix}=\mathbf a\mathbf b^{\top}\ \checkmark

Both identities check entry by entry — and (b) is exactly why Section 7's $\partial\mathrm{Loss}/\partial A$ came out as the outer product $\boldsymbol\delta^{(Z)}\mathbf x^{\top}$ .

Problem 8hard

Run the full backward pass on the lecture's function $f(x)=\sqrt{x^2+e^{x^2}}+\cos(x^2+e^{x^2})$ at $x=1$ , using the graph $a=x^2,\ b=e^a,\ c=a+b,\ d=\sqrt c,\ e=\cos c,\ f=d+e$ . (a) Forward pass: all six values to 4 decimals. (b) Backward pass: all the blames $\bar f,\bar d,\bar e,\bar c,\bar b,\bar a,\bar x$ . (c) Confirm $\bar x$ against the closed-form derivative.

What this tests. The complete algorithm of Sections 2–5, executed by hand with real numbers — including the fork at $c$ and the fork at $a$ .

Show the full solution

(a) Forward, left to right.

a=1^2=1,\quad b=e^1=2.7183,\quad c=1+2.7183=3.7183

d=\sqrt{3.7183}=1.9283,\quad e=\cos(3.7183)=-0.8383,\quad f=1.9283-0.8383=1.0900

(Radians throughout —

c

is just a number, and

\cos

of it lives on the unit circle, not a protractor.)

(b) Backward, right to left. Start

\bar f=1

. Node

f=d+e

mails both parents a copy scaled by

1

:

\ \bar d=1,\ \bar e=1

.

The fork at $c$ .

c

feeds

d=\sqrt c

and

e=\cos c

, so its mail arrives in two pieces that add:

\bar c=\bar d\cdot\frac{1}{2\sqrt c}+\bar e\cdot(-\sin c)=\frac{1}{2(1.9283)}+\big(-\sin(3.7183)\big)=0.2593+0.5452=0.8045

(

\sin(3.7183)=-0.5452

— third quadrant, so the minus signs stack to a plus.)

Through $b$ , and the fork at $a$ .

\bar b=\bar c\cdot1=0.8045

. Then

a

feeds

b=e^a

and

c=a+b

:

\bar a=\bar b\cdot e^{a}+\bar c\cdot 1=0.8045\cdot2.7183+0.8045=2.1870+0.8045=2.9915

The last edge.

\ \bar x=\bar a\cdot 2x=2.9915\cdot2=5.9831

.

(c) The closed form agrees.

\frac{df}{dx}=\Big(\frac{1}{2\sqrt{x^2+e^{x^2}}}-\sin(x^2+e^{x^2})\Big)\big(2x+2xe^{x^2}\big)\Big|_{x=1}=0.8045\times7.4366=5.9831\ \checkmark

Notice the factor

0.8045

is

\bar c

, and

2x+2xe^{x^2}=7.4366

is what the graph assembled as

\bar a\cdot2x/\bar c

— the closed form and the graph are the same computation, differently organised.

Forward: $1,\ 2.7183,\ 3.7183,\ 1.9283,\ -0.8383,\ 1.0900$ . Backward: $\bar f=1,\ \bar d=\bar e=1,\ \bar c=\bar b=0.8045,\ \bar a=2.9915,\ \bar x=5.9831=df/dx$ . ✓

Watch out

Run the same pass at $x=2$ and $e^{x^2}=e^4=54.6$ dominates everything: $\bar c=-0.8223$ and $df/dx=-182.87$ . The companion doc's printed run at $x=2$ uses $\cos(58.6)\approx0.83$ , but the true value in radians is $-0.46$ — a calculator slip that flips the final answer completely (see the errata note in §2). The habit that catches such slips: always confirm a hand-computed derivative with a small numerical nudge.

Problem 9medium

The activation family, from first principles. (a) Show that the softplus $s(z)=\ln(1+e^{z})$ has derivative $s'(z)=\sigma(z)$ — the sigmoid is softplus's slope. (b) Prove the bridge $\tanh(z)=2\,\sigma(2z)-1$ . (c) Using (b) and $\sigma'=\sigma(1-\sigma)$ , derive $\tanh'=1-\tanh^{2}$ without a quotient rule.

What this tests. That the activation identities frameworks hard-code are a small connected family — one proof engine (find the appearances, chain the slopes) generates them all.

Show the full solution

(a) Chain rule through the logarithm. Outer slope

1/(1+e^{z})

, inner slope

e^{z}

:

s'(z)=\frac{e^{z}}{1+e^{z}}=\frac{1}{e^{-z}+1}=\sigma(z)\ \checkmark

(dividing top and bottom by

e^{z}

in the middle step). So the smooth ramp softplus steepens exactly at the sigmoid's rate — which is why

\sigma

is also called the logistic function.

(b) Build $2\sigma(2z)-1$ and simplify.

2\,\sigma(2z)-1=\frac{2}{1+e^{-2z}}-1=\frac{2-(1+e^{-2z})}{1+e^{-2z}}=\frac{1-e^{-2z}}{1+e^{-2z}}

Multiply top and bottom by

e^{z}

:

\frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}=\tanh(z)\ \checkmark

In words: tanh is a sigmoid, stretched to run twice as fast and re-centred to fill

(-1,1)

.

(c) Differentiate the bridge with Rule 1. Write

t=\tanh z

and note from (b) that

\sigma(2z)=\tfrac{1+t}{2}

, hence

1-\sigma(2z)=\tfrac{1-t}{2}

. Then

\tanh'(z)=2\cdot\sigma'(2z)\cdot 2=4\,\sigma(2z)\big(1-\sigma(2z)\big)=4\cdot\frac{1+t}{2}\cdot\frac{1-t}{2}=(1+t)(1-t)=1-t^{2}\ \checkmark

— the factor

2

outside is the bridge's scaling, the factor

2

inside is the chain rule through

2z

.

$s'=\sigma$ ; $\ \tanh z=2\sigma(2z)-1$ ; $\ \tanh'=1-\tanh^{2}$ , derived purely from $\sigma'=\sigma(1-\sigma)$ .

Remember

All three facts are one family portrait: softplus's slope is $\sigma$ , $\tanh$ is a rescaled $\sigma$ , and both backward-pass identities ( $\sigma(1-\sigma)$ and $1-a^2$ ) are the same identity seen through the bridge. Knowing one buys you the others.

Problem 10hard

The cookbook earns its keep: ridge regression solved by pure algebra. Consider the regularised least-squares loss $E(\mathbf s)=\|\mathbf x-A\mathbf s\|^{2}+\lambda\|\mathbf s\|^{2}$ (a) Using the cookbook identities, derive $\nabla_{\mathbf s}E$ . (b) Set it to zero and show the minimiser solves $(A^{\top}A+\lambda I)\,\mathbf s=A^{\top}\mathbf x$ . (c) Solve numerically for $A=\begin{bmatrix}1&0\\2&1\\0&3\end{bmatrix},\qquad \mathbf x=\begin{bmatrix}1\\2\\0\end{bmatrix},\qquad \lambda=1.$

What this tests. That the identities you proved are tools, not trivia: two of them combine into the closed-form solution of ridge regression — the first machine-learning model this course can now train end-to-end by algebra alone.

Show the full solution

(a) Differentiate each piece with the cookbook. The first piece is the least-squares identity with

W=I

:

\frac{\partial}{\partial\mathbf s}\|\mathbf x-A\mathbf s\|^{2}=-2(\mathbf x-A\mathbf s)^{\top}A

The second is the quadratic identity with

B=I

(so

B+B^{\top}=2I

):

\frac{\partial}{\partial\mathbf s}\lambda\,\mathbf s^{\top}\mathbf s=2\lambda\,\mathbf s^{\top}

Sum:

\ \nabla_{\mathbf s}E=-2(\mathbf x-A\mathbf s)^{\top}A+2\lambda\,\mathbf s^{\top}

.

(b) Set the gradient to zero (and transpose the row into a column to read it as an equation in

\mathbf s

):

-2A^{\top}(\mathbf x-A\mathbf s)+2\lambda\mathbf s=\mathbf 0 \;\;\Longrightarrow\;\; A^{\top}A\,\mathbf s+\lambda\mathbf s=A^{\top}\mathbf x \;\;\Longrightarrow\;\; \big(A^{\top}A+\lambda I\big)\mathbf s=A^{\top}\mathbf x\ \checkmark

This is a stationary point, and it is the minimum:

E

is a sum of squares whose quadratic part

\mathbf s^{\top}(A^{\top}A+\lambda I)\mathbf s

is positive for every

\mathbf s\neq\mathbf 0

once

\lambda>0

— the bowl opens upward in every direction, and adding

\lambda I

even guarantees the system is invertible when

A^{\top}A

alone is not (Unit 5's nearly-singular warning, answered).

(c) Numbers.

A^{\top}A=\begin{bmatrix}1{+}4{+}0&0{+}2{+}0\\0{+}2{+}0&0{+}1{+}9\end{bmatrix}=\begin{bmatrix}5&2\\2&10\end{bmatrix},\qquad A^{\top}A+I=\begin{bmatrix}6&2\\2&11\end{bmatrix},\qquad A^{\top}\mathbf x=\begin{bmatrix}1{+}4{+}0\\0{+}2{+}0\end{bmatrix}=\begin{bmatrix}5\\2\end{bmatrix}

Solve

\begin{bmatrix}6&2\\2&11\end{bmatrix}\mathbf s=\begin{bmatrix}5\\2\end{bmatrix}

: determinant

66-4=62

, so by Cramer (or elimination)

s_1=\frac{5\cdot11-2\cdot2}{62}=\frac{51}{62}\approx0.8226,\qquad s_2=\frac{6\cdot2-2\cdot5}{62}=\frac{2}{62}=\frac{1}{31}\approx0.0323

$\nabla_{\mathbf s}E=-2(\mathbf x-A\mathbf s)^{\top}A+2\lambda\mathbf s^{\top}$ ; minimiser $(A^{\top}A+\lambda I)\mathbf s=A^{\top}\mathbf x$ ; numerically $\mathbf s\approx(0.8226,\ 0.0323)^{\top}$ .

Remember

Gradient $\to$ zero $\to$ linear system: for quadratic losses, "training" is one solve. Unit 9 begins exactly where this stops — with losses whose gradients cannot be set to zero in closed form, so we must walk downhill instead.

Problem 11medium

Differentiate a program with dual numbers (Section 11), seeding $x=x_0+1\cdot\varepsilon$ . (a) $f(x)=x^{2}e^{x}$ at $x_0=1$ . (b) $f(x)=\sqrt{x^{2}+9}$ at $x_0=-4$ — then compare with Section 10's stand-in line. (c) In one sentence: why are these results exact, where the gradient checker's finite differences never are?

What this tests. Forward-mode autodiff executed by hand: the $\varepsilon^{2}=0$ algebra runs the chain rule mechanically — you only ever do arithmetic on pairs.

Show the full solution

(a) Run the pairs. Seed

x=(1,\,1)

.

x^{2}=(1\cdot1,\ 1\cdot1+1\cdot1)=(1,\,2)\qquad e^{x}=(e,\ e\cdot1)=(e,\,e)

x^{2}\cdot e^{x}=(1\cdot e,\ \ 1\cdot e+2\cdot e)=(e,\ 3e)

So

f(1)=e\approx2.7183

and

f'(1)=3e\approx8.1548

. Check against the formula the program never used:

f'=(2x+x^{2})e^{x}\big|_{1}=3e

✓.

(b) A chain with a square root. Seed

x=(-4,\,1)

.

x^{2}=\big((-4)^2,\ 2\cdot(-4)\cdot1\big)=(16,\,-8)\qquad x^{2}+9=(25,\,-8)

\sqrt{\,\cdot\,}:\ \big(\sqrt{25},\ \tfrac{-8}{2\sqrt{25}}\big)=(5,\ -0.8)

So

f(-4)=5

and

f'(-4)=-0.8=-\tfrac45

— precisely the anchor value and slope of Section 10's stand-in line. The widget's tangent line and this pencil-and-paper pair computation are the same mathematics arriving by two roads.

(c) Why exact. No step ever subtracts two nearly equal numbers or divides by a small

h

— each rule (sum, product,

g'(a)\dot a

) is the corresponding calculus rule, applied with ordinary arithmetic — so the only error anywhere is standard floating-point rounding of the values themselves.

(a) $(e,\,3e)$ : $f'(1)=3e\approx8.1548$ . (b) $(5,\,-0.8)$ : $f'(-4)=-\tfrac45$ . (c) The algebra rules are the calculus rules — nothing is approximated, so nothing needs a step size.

Remember

Reverse mode (this unit's hero) and dual numbers (its mirror) bracket the field: backward for one output and many inputs, forward for one input and many outputs. Both differentiate the program, both are exact — and neither ever writes a derivative formula.

Next up

Unit 8 · Taylor & MacLaurin Series →

Linearization kept one derivative and got a line. Unit 8 keeps them all — and proves, from a single flat spot on a hiking trail, exactly how far each cut of the series can be trusted. Then it takes the second-order term into two variables, where it becomes the Hessian: the judge that tells a bowl from a dome from a saddle at the bottom of every loss valley.

← All units