Unit 17 · Machines with Memory — The Math Behind the Machine

Why order matters

"Dog bites man" and "Man bites dog" use the same three words. How can a machine that reads them tell which story it is?

Imagine this

Two newspaper headlines. "Dog bites man." Nobody cares. "Man bites dog." Front page!

Both use the same three words. Only the order is different — and the order is the whole story. A machine that reads must notice the order.

In Unit 16 every word became a list of numbers. So how do we turn a whole sentence into numbers? Here are three ideas. The first two fail, and how they fail tells us what we need.

Idea 1 · Throw the words in a bag. Add up the word vectors, or just count the words. People call this a bag of words. It is quick. But "dog bites man" and "man bites dog" have exactly the same bag: one "dog", one "bites", one "man". The sums are the same. The order is lost.

Idea 2 · Look at the last few words. Keep a small window, say the last 3 words. This is what the neural language model of Unit 16 did: it saw only a fixed window of the words just before. A window keeps local order, but it forgets everything older. Take the announcement "The train to Delhi from platform four is late". When "late" arrives, a 3-word window holds only "four is late". What is late? The word "train" has already dropped out.

Idea 3 · Read one word at a time and keep a running note. Think of a shopkeeper with a notebook. After each customer, he updates one line. But a plain running total does not care about order either: 5 + 3 is the same as 3 + 5. So we add one twist. Before writing in the new word, we first change the old note a little — turn it, shrink it — with a matrix. Then an old word has been turned many times and a new word only once. The order leaves its mark.

A tiny example. Give each word a toy vector (hand-made, not trained): $\text{dog}=(1,0)$ , $\text{man}=(0,1)$ , $\text{bites}=(1,1)$ . The note starts at $(0,0)$ . At each word we do two things: turn the note a quarter turn with $W=\begin{pmatrix}0&-1\\ 1&0\end{pmatrix}$ , then add the word. A quarter turn sends $(a,b)$ to $(-b,a)$ .

dog bites man: after "dog" the note is $(1,0)$ . Turn it: $(0,1)$ ; add "bites": $(1,2)$ . Turn it: $(-2,1)$ ; add "man": $(-2,2)$ .
man bites dog: after "man", $(0,1)$ . Turn: $(-1,0)$ ; add "bites": $(0,1)$ . Turn: $(-1,0)$ ; add "dog": $(0,0)$ .

Two different notes: $(-2,2)$ and $(0,0)$ . Without the turn, both would end at the plain sum $(2,2)$ . The turn is what makes the note remember order.

Try: In running note, set the turn to 0° and watch both sentences land on the same point, (2, 2) — that is the bag of words again. Turn it back up and they split apart. Now try 180°: they land together again, at (0, 0). Press ↻ another order for B to try the other orders of the same three words: at 90° none of them lands on A's note. Then open window and slide the window size down to 3: "train" falls out before "late" arrives.

turn each step90°

keep1

Why it works

Unfold the rule over three words. The note ends as $W^2\,\text{word}_1+W\,\text{word}_2+\text{word}_3$ . Every position wears its own stamp: the last word goes in as it is, the one before is turned once, the first is turned twice. Swap two words and they trade stamps, so the note changes.

A plain sum gives every word the same stamp — no turn at all. That is exactly why it cannot see order.

The realization

$\begin{gathered}\text{new note}=\\ W\,(\text{old note})+\text{new word}\end{gathered}$

A bag of words forgets order. A window forgets the past. A running note keeps both, if it changes the old note before adding the new word. Every word leaves its mark, and older marks have been changed more times.

The three readers side by side.

	sees word order?	remembers far back?	size of the summary
bag of words	no	yes — every word counts, but order is gone	fixed
window of k words	only inside the window	no — only the last k words	fixed
running note	yes	yes, if old marks do not fade (§7)	fixed

Rule of thumb. If the question is "which words appear?" (is this review about food or about service?), a bag is often enough. If the question is "who did what to whom?", you need a reader that keeps order — a running note.

Trap

A running total is still a bag. Adding numbers does not care about order, so "keep a sum of the words" loses order just as badly as counting them. The note only remembers order because the old note is changed (turned, shrunk) before each new word is added.

Pause & predict

With a plain bag of words (just count the words), which pair of sentences looks different?

Pause & predict

Use the same rule (quarter turn, then add) and the same toy vectors. What is the note after reading "bites dog" (just these two words)?

Pause & predict

Change the turn to a half turn, 180°, which sends $(a,b)$ to $(-a,-b)$ . Can this reader still tell "dog bites man" from "man bites dog"?

The road ahead. The unit has five acts.

Why a reader needs memory (§1–§3): order, the recurrent cell, and the same cell copied along time.
A cell that talks (§4–§5): the many jobs one cell can do, and a small language model that learns to write, one character at a time.
Blame through time (§6–§8): training sends blame back through every word; it is multiplied by the same matrix again and again, so eigenvalues decide whether memory fades or explodes; then first aid.
Gates, and reading both ways (§9–§11): the LSTM's express lane, the GRU's blend dial, and readers that read in both directions or in floors.
Translating through one summary (§12–§14): an encoder–decoder that turns English into Hindi, how it picks and grades its words, and the bottleneck that Unit 18 removes.

In one sentence: A sentence is an ordered list, so a reader must keep a running note and change the old note before adding each new word — a bag or a plain total would give "dog bites man" and "man bites dog" the same note.

The recurrent cell: one small machine, used again and again

What exactly happens inside the machine each time it reads one more word?

Imagine this

You sit in a long lecture with one small card for notes. After each new sentence from the teacher, you rewrite the card: keep some of what was there, add the new point, and squeeze it so it still fits on the card.

The card never grows. It can never hold everything, so you must choose what to keep. You use the same habit every time. That habit is the recurrent cell. The card is its memory.

Now we make the running note a real network part. The note is a list of numbers called the hidden state, $\mathbf h_t$ — "the card after word $t$ ". At each word the cell does one small job:

$\mathbf h_t=\tanh\big(W\,\mathbf h_{t-1}+U\,\mathbf x_t+\mathbf b\big).$

Read it aloud, from the inside out. $W\mathbf h_{t-1}$ : take the old card and rewrite it — the matrix $W$ decides how each old number is carried over, mixed or shrunk. $U\mathbf x_t$ : write the new word in — the matrix $U$ decides where on the card each part of the word goes. $\mathbf b$ : a small shift, as in every layer of Unit 15. And $\tanh$ : squeeze every number to between −1 and 1, so the note never overflows the card. It is the bend of Unit 15.

The most important fact: $W$ , $U$ and $\mathbf b$ are the same at every word. The cell does not grow a new set of weights for word 7. One small machine reads a sentence of any length.

A tiny example with one number. Let the note be a single number. Take $w=0.5$ , $u=1$ , $b=0$ , start with $h_0=0$ , and feed three inputs $x=(1,0,0)$ — one real word, then two blanks.

$\begin{aligned}h_1&=\tanh(0.5\cdot0+1\cdot1)\\ &=\tanh(1)\approx0.7616\\ h_2&=\tanh(0.5\cdot0.7616+0)\\ &=\tanh(0.3808)\approx0.3634\\ h_3&=\tanh(0.5\cdot0.3634+0)\\ &=\tanh(0.1817)\approx0.1797\end{aligned}$

Watch the first word fade. Each step keeps half of the note ( $w=0.5$ ), and tanh squeezes it a little more. After three steps only 0.18 is left of the 0.76. Keep this in mind — it is the seed of the whole unit.

Try: In one number, press next three times and read 0.7616, 0.3634, 0.1797. Push keep old to 1 and the first word lasts much longer (0.7616, 0.6420, …). Then switch to a note of 6 numbers: with slide down, every word enters at the top slot and slides down one slot per word, fading as it goes — a diagonal streak.

keep old

w

0.5

new word

u

Why it works

The note goes back into the cell, so the note after word 5 was built from the note after word 4, which was built from the note after word 3, and so on. That makes $\mathbf h_5$ a summary of all five words, in order — without the cell ever looking at more than one word at a time. Training (§6) then tunes $W$ and $U$ so that the summary keeps what later guesses will need.

The realization

$\mathbf h_t=\tanh\big(W\,\mathbf h_{t-1}+U\,\mathbf x_t+\mathbf b\big)$

A recurrent cell is one layer of Unit 15 with a second input: its own note from one word ago. The same weights are used at every word. The note is the memory — and with $|w|<1$ , what an old word wrote there fades a little at every step.

Trap

The note is not a list of the words read so far. It is a fixed-size summary. A note of 4 numbers has 4 numbers after 3 words and still 4 numbers after 300 words. Something must be squeezed out — and the network has to learn what to keep.

Pause & predict

Same cell, but $w=1$ : $u=1$ , $b=0$ , $h_0=0$ , inputs $x=(1,0)$ . What is $h_2$ ?

Pause & predict

A recurrent cell reads a 100-word review. How many different $W$ matrices does it use?

If you want the algebra · 2 proofs, step by step

Prove it · tanh squeezes, and its slope is 1 − tanh²

Claim. $\tanh z=\dfrac{e^{z}-e^{-z}}{e^{z}+e^{-z}}$ always lies strictly between −1 and 1, and its slope is $\tanh'(z)=1-\tanh^2 z$ , which is at most 1 (reached only at $z=0$ ).

The top

e^{z}-e^{-z}

is always smaller in size than the bottom

e^{z}+e^{-z}

, because

e^{\pm z}>0

. So

|\tanh z|<1

. This is why the note never overflows the card: every entry of

\mathbf h_t

stays inside

(-1,1)

Quotient rule, with

A=e^z-e^{-z}

B=e^z+e^{-z}

A'=B

B'=A

\begin{aligned}\tanh'(z)&=\frac{B\cdot B-A\cdot A}{B^2}\\ &=1-\frac{A^2}{B^2}\\ &=1-\tanh^2z.\end{aligned}

∎ Handy: the slope comes straight from the stored note. At

h_1=0.7616

, the slope is

1-0.5800\approx0.42

Prove it · written out, the first word is buried under powers of w

Claim. For the scalar cell with $b=0$ , $h_0=0$ : $h_3=\tanh\!\big(w\tanh(w\tanh(ux_1)+ux_2)+ux_3\big)$ . If every number stays small (so $\tanh v\approx v$ ), then $h_3\approx w^2ux_1+wux_2+ux_3$ .

Substitute step by step:

h_1=\tanh(ux_1)

h_2=\tanh(wh_1+ux_2)

h_3=\tanh(wh_2+ux_3)

. Put each inside the next. The same

w

and

u

appear at every level: one machine, used three times.

For small arguments

\tanh v\approx v

(its slope at 0 is 1). Replace each tanh by its argument:

\begin{aligned}h_3&\approx w(w\,ux_1+ux_2)\\ &\quad+ux_3\\ &=w^2ux_1+wux_2\\ &\quad+ux_3.\end{aligned}

∎ Word

k

T

enters with weight

w^{T-k}

. With

w=0.5

the first of three words gets

0.25

: the fading of §2, in one line.

In one sentence: A recurrent cell rewrites one fixed-size card at every word, $\mathbf h_t=\tanh(W\mathbf h_{t-1}+U\mathbf x_t+\mathbf b)$ , with the same weights each time — and with a small $w$ the first word fades: 0.7616, 0.3634, 0.1797.

Unrolling: the loop is a chain of twins

A 5-word sentence and a 500-word story: does the network need a bigger brain for the longer one?

Imagine this

A dabbawala carries a lunch box from station to station. Film him, and print one photo at every station. Lay the photos in a row. It looks like a long chain of dabbawalas passing the box along.

But it is one man, doing the same job again and again. The chain is just the loop, laid out in time.

We often draw the recurrent cell as one box with a loop arrow: its note goes out and comes straight back in. That is compact, but hard to reason about. So we unroll it: draw one copy of the box for every word, and turn the loop into an arrow from each copy to the next.

For a three-word sentence the unrolled picture is

$\mathbf h_0\xrightarrow{\ \text{cell}\ }\mathbf h_1\xrightarrow{\ \text{cell}\ }\mathbf h_2\xrightarrow{\ \text{cell}\ }\mathbf h_3,$

with the words $\mathbf x_1,\mathbf x_2,\mathbf x_3$ coming in from below. Look at it and you see an old friend: a deep network from Unit 15, three layers deep. There is one difference. In Unit 15 every layer had its own matrix. Here every "layer" uses the same $W$ , $U$ and $\mathbf b$ . We say the weights are shared (or tied).

Count the numbers it learns. Say the note has $h$ numbers and each word has $d$ . Then $W$ is $h\times h$ , $U$ is $h\times d$ and $\mathbf b$ has $h$ entries, so

$\begin{aligned}\text{numbers to learn}&=h\cdot h+h\cdot d+h\\ &=h(h+d)+h.\end{aligned}$

With $h=4$ and $d=3$ : $16+12+4=32$ . For a 3-word sentence: 32. For a 3000-word book: still 32. The unrolled chain gets longer; the list of weights does not.

Try: Press ▶ fold and watch the chain fold back into one cell with a loop; press it again to unroll. Press light up W: every gold arrow flashes at the same moment, because they are one matrix. Drag the length from 2 to 8 words: the chain grows, the weight count stays 32.

drag the picture to orbit

words5

Why share one set of weights?

Because the rules of a language do not depend on where you are in the sentence. "A describing word comes before its noun" is as true at word 3 as at word 300. With shared weights, whatever the cell learns at one position works at every position, so every sentence it reads teaches the same small machine many times over. And it can read a sentence longer than any it saw in training: it just runs the same step more times.

The realization

$\begin{gathered}\text{unrolled RNN}=\\ \text{a deep network whose layers}\\ \text{all share }W,\,U,\,\mathbf b\end{gathered}$

A recurrent network is a loop. Unroll it and it is a chain as long as the sentence, one copy of the cell per word. So everything we learned about deep networks — forward pass, backward pass, blame that fades — carries over. The only new twist is that the copies are twins.

Trap

Unrolling does not make copies of the weights. The boxes in the picture are one cell drawn once per word. Change $W$ and every box changes at once. That is why, in training, the blame for $W$ is collected from every box and added up (§6).

Pause & predict

A cell has a note of $h=3$ numbers and reads words of $d=2$ numbers. It reads a 50-word sentence. How many weights and shifts does it learn?

Pause & predict

A recurrent network was trained only on sentences of 10 words. Now it is given a 25-word sentence. What happens?

If you want the algebra · 1 proof, step by step

Prove it · the weight count does not depend on the length

Claim. A recurrent cell with note size $h$ and word size $d$ learns $h(h+d)+h$ numbers, whatever the number of words $T$ . Without sharing, the $T$ -step chain would need $T$ times as many.

Shapes:

W\mathbf h_{t-1}

must give

h

numbers from

h

numbers, so

W

h\times h

U\mathbf x_t

gives

h

numbers from

d

, so

U

h\times d

\mathbf b

has

h

. Total

h^2+hd+h=h(h+d)+h

. The shape rule of Unit 15: rows for where the numbers go, columns for where they come from.

Unrolling makes

T

copies, but every copy points at the same

W,U,\mathbf b

. The list of distinct numbers is the same list. If each step had its own, we would have

T\big(h(h+d)+h\big)

. ∎ Example:

h=4

d=3

: 32 numbers shared, or

320

for 10 unshared steps. Sharing is also what lets the cell read a sentence longer than any it saw in training.

In one sentence: Unrolled, a recurrent cell is a deep network as long as the sentence whose layers are twins, so it learns only $h(h+d)+h$ numbers — 32 for $h=4$ , $d=3$ — whether it reads 3 words or 3000.

Many jobs, one cell

The cell only reads. How do we make it say something — a label, a tag for every word, a caption, the next word?

Imagine this

One pen can write very different things: a caption under a photo, a single word on a review ("great!"), a label on each item of a list, or a full letter in another language.

The recurrent cell is that pen. What changes is only where the inputs go in and where we read the answers out.

So far the cell only reads. To make it say something, we add a read-out layer. At any word $t$ we can turn the note into scores and then into probabilities with softmax (Unit 14):

$\hat{\mathbf y}_t=\operatorname{softmax}\big(V\,\mathbf h_t+\mathbf c\big).$

Read it aloud: "look at the card, give every possible answer a score, turn the scores into shares that add up to 1." Where we put inputs and read-outs gives the four classic shapes:

Many → one. Read a whole review, then read out once at the end: happy or unhappy. (Sentiment.)
One → many. Give it one input — a photo's vector — and let it write a caption word by word.
Many → many, same length. Read out at every word: tag each word as a noun, a verb or other.
Many → many, different length. Read a whole English sentence, then write a Hindi one. This needs two cells, and it is §12.

And one favourite: the language model. At every word, guess the next word. This is the neural language model of Unit 16, but now the note carries all earlier words, not just a fixed window.

A tiny example. After reading "the train to", the read-out gives the scores Delhi 2, Mumbai 1, late 0.5, the −1. Softmax turns them into probabilities: $e^2\approx7.389$ , $e^1\approx2.718$ , $e^{0.5}\approx1.649$ , $e^{-1}\approx0.368$ , total $\approx12.124$ . So Delhi gets $7.389/12.124\approx0.609$ , Mumbai $0.224$ , late $0.136$ , the $0.030$ . If the true next word is "Delhi", this step's loss is $-\ln0.609\approx0.495$ — the surprise of Unit 14. The loss for the whole sentence is the sum of these, one per word.

Try: Click through the five shapes and count the blue and cyan arrows. In next word, press ▶ read: after each word the bars show the cell's guess for the word that comes next. After "the train to" the bars are §4's softmax: 0.609, 0.224, 0.136, 0.030 (the other steps' bars are made-up numbers, for the picture).

Why it works

The note $\mathbf h_t$ is a summary of everything read up to word $t$ . So any question about that part of the sentence can be answered from it by one small read-out layer.

Which job the cell learns is decided by where we attach the loss. Training only rewards the note for what the read-outs are asked. Ask once, at the end (sentiment), and the note learns to carry the gist all the way to the end. Ask at every word (tagging, next word), and it learns to keep what each word needs right then.

The realization

$\begin{gathered}\hat{\mathbf y}_t=\operatorname{softmax}(V\mathbf h_t+\mathbf c)\\ L=\sum_t-\ln\hat y_t[\text{true word}]\end{gathered}$

One cell, many jobs. Feed inputs at one step or at every step; read out at the end or at every step. When we read out at every step, each step adds its own surprise to the loss, and the total is what training makes small.

Trap

"Many → many" comes in two kinds, and they need different machines. Tagging gives one answer per input word, at the same position — one cell is enough. Translating writes a sentence of a different length, in a different order — it needs a reader and a separate writer (§12).

Pause & predict

You want to read a customer's 40-word complaint and decide one thing: is it about delivery, price or quality? Which shape, and how many read-outs?

Pause & predict

The read-out after "the train to" gives Delhi 0.609, Mumbai 0.224, late 0.136, the 0.030. The true next word is "Mumbai". What is this step's loss?

If you want the algebra · 1 proof, step by step

Prove it · each read-out sends back "prediction minus truth"

Claim. With $\hat{\mathbf y}_t=\operatorname{softmax}(\mathbf o_t)$ , $\mathbf o_t=V\mathbf h_t+\mathbf c$ and $L=\sum_t-\ln\hat y_t[\text{true}_t]$ , the blame at each read-out is $\partial L/\partial\mathbf o_t=\hat{\mathbf y}_t-\mathbf y_t$ (one-hot truth), and $\partial L/\partial V=\sum_t(\hat{\mathbf y}_t-\mathbf y_t)\,\mathbf h_t^{\top}$ .

Only the

t

-th term of the sum depends on

\mathbf o_t

. For that term, softmax plus cross-entropy gives prediction minus truth (Unit 15):

\hat{\mathbf y}_t-\mathbf y_t

. Example from §4: truth "Delhi",

\hat{\mathbf y}=(0.609,0.224,0.136,0.030)

, so the blame is

(-0.391,0.224,0.136,0.030)

V

is used at every step, so (Rule B plus sharing) its gradient adds one outer product per step:

\sum_t(\hat{\mathbf y}_t-\mathbf y_t)\mathbf h_t^{\top}

. ∎ And each read-out also sends

V^{\top}(\hat{\mathbf y}_t-\mathbf y_t)

into its note

\mathbf h_t

, where it joins the blame arriving from later steps.

In one sentence: Put inputs in and read outputs out at one step or at every step, and the same cell can caption, judge, tag, translate or guess the next word — with the total loss the sum of each step's surprise.

An RNN language model: learning to talk, then talking

Your phone suggests the next word while you type. Could a machine write a whole sentence that way — and why does it so often get stuck repeating itself?

Imagine this

Open the keyboard on your phone, type "the train to", and keep tapping the middle suggestion. The phone writes a whole sentence by itself. Very often it gets stuck, repeating the same phrase again and again.

You have just watched a language model talk. It guesses one word, then listens to what it just said, and guesses the next. A machine that talks by listening to itself.

In §4 the read-out at every word guessed the next word. That is a language model. We use it in two ways: first to learn, then to talk.

1 · Learning to talk (training). Take a real sentence. At every position the cell reads the true words so far and guesses the next one. The true next word is the target. The step's loss is its surprise, $-\ln p(\text{true next word})$ , and the sentence's loss is the sum of the surprises. One detail matters a lot: at every step we feed in the true previous word — even when the model's own guess was wrong. This is called teacher forcing. It works like training wheels: it keeps the learner on the road while it learns.

2 · Talking (generating). Now there is no true sentence. Start from a few words — the seed. The read-out gives its probabilities; pick one word; feed that word back in as the next input; guess again. Repeat until the model writes a full stop.

3 · Which word to pick? Greedy: always take the top word. Safe, but dull — it tends to fall into a loop. Or sample: roll a die whose sides are weighted by the probabilities. A dial called the temperature $T$ makes the die bolder or tamer. Divide every score by $T$ before the softmax:

$p_i=\frac{e^{s_i/T}}{\sum_j e^{s_j/T}}.$

Read it aloud: "divide each score by the temperature, then softmax as usual." A small $T$ stretches the gaps between the scores, so the favourite takes even more. A big $T$ shrinks the gaps, so the long shots get a chance.

A tiny example. Take §4's scores after "the train to": Delhi 2, Mumbai 1, late 0.5, the −1.

$T=1$ : the scores stay as they are: 0.609, 0.224, 0.136, 0.030.
$T=0.5$ : the scores double to 4, 2, 1, −2: 0.842, 0.114, 0.042, 0.002. Bold and repetitive — Delhi nearly every time.
$T=2$ : the scores halve to 1, 0.5, 0.25, −0.5: 0.434, 0.263, 0.205, 0.097. Adventurous — and "the train to the" now comes up about one time in ten.

how to pick	what it does	how it writes	use it for
greedy	always the top word	safe, flat, often loops	short, exact answers
sample, T < 1	favourite gets even more	bold, a little repetitive	factual text with some variety
sample, T = 1	the model's own odds	as varied as its training text	faithful imitation
sample, T > 1	long shots get a chance	adventurous, makes mistakes	brainstorming, surprise

Try: First read the bars: after "the train to " the model's top guess is "p", 0.171 — two of its cities, Pune and Patna, start with p. Press ▶ talk (greedy): it writes its favourite line, "patna will arrive on platform three.", and then repeats it. Now ▶ talk (sample) with dice #1 and turn the temperature up: at $T=0.2$ it writes the greedy line again; at $T=0.5$ a clean announcement ("patna will leave from platform one. …"); at $T=1$ "patna is cancelled today. the train to surat is cancelled today."; at $T=2$ it babbles ("pendy mare tom frimn …"). Type the seed "the trian to " (a typo) and talk greedily: the slip snowballs into "buny misun". Last, open training: the true sentence is fed in, and its 26 surprises add up to 6.507.

the text so far — seed written by the model

the note after the last character (48 numbers)

guess for the next character

seed

temperature

T

How this little model was made

Characters, not words: the 26 letters, the space and the full stop — 28 symbols, each fed in as a one-hot input $\mathbf x_t$ . The cell is exactly §2's, $\mathbf h_t=\tanh(W\mathbf h_{t-1}+U\mathbf x_t+\mathbf b)$ , with 48 numbers in the note, and the read-out is §4's, $\operatorname{softmax}(V\mathbf h_t+\mathbf c)$ . That is $48\cdot48+48\cdot28+48+28\cdot48+28=5\,068$ numbers.

Training used teacher forcing on strings of about eight sentences, backprop through time (§6) over 96 characters, the gradient clipped at length 5 (§8) and $W$ started at 0.9 times an orthogonal matrix (§8), then 6 000 steps of Adam (Unit 11). A short training script, train-u17-talk.py, kept with this site's source code, rebuilds these exact weights. A few of its 300 sentences: the train to pune is late. · passengers for kochi please go to platform six. · my mother makes chai at six.

Why it works

Everything written so far — the seed and the model's own words — has been folded into the note, one character at a time. So every guess is made with the whole text so far in mind, and training has tuned those guesses to match real text. Writing is nothing more than guessing the next character, again and again, and believing your own guesses.

The realization

$\begin{gathered}\text{train:}\quad L=-\sum_t\ln p(\text{true next})\\ \text{talk:}\quad\text{pick from }\operatorname{softmax}(s/T)\\ \text{and feed the pick back in}\end{gathered}$

A language model learns with the true text fed in (teacher forcing) and talks by feeding back its own picks. Greedy picking is dull and loops; sampling with a temperature trades safety for variety.

Trap · exposure bias

While training, the cell only ever saw true text. While talking, it sees its own guesses — including its mistakes — and it never practised recovering from one. So one early slip can snowball. To see it, pretend the model had slipped and written "trian" instead of "train": give it the seed "the trian to", and it carries on with "buny misun" — words that are in none of its 300 sentences. This mismatch between training and talking is called exposure bias.

Pause & predict

Scores Delhi 2, Mumbai 1, late 0.5, the −1. At temperature $T=0.5$ , what probability does Delhi get?

Pause & predict

You turn the temperature down, closer and closer to 0. What does sampling turn into?

Pause & predict

A model is being trained on "the train to pune is late." At the 5th character it guesses "u" instead of "t". Under teacher forcing, what is fed in at the next step — and what would be fed in if the model were talking?

If you want the algebra · 2 proofs, step by step

Prove it · temperature keeps the order and only changes the gaps

Claim. With $p_i=e^{s_i/T}/\sum_je^{s_j/T}$ , the ratio of two shares is $p_i/p_j=e^{(s_i-s_j)/T}$ . So the order of the words never changes; as $T\to0$ the top word takes everything (greedy), and as $T\to\infty$ every share tends to $1/n$ (a fair die).

Both shares have the same denominator, so it cancels:

\dfrac{p_i}{p_j}=\dfrac{e^{s_i/T}}{e^{s_j/T}}=e^{(s_i-s_j)/T}

. If

s_i>s_j

this ratio is above 1 for every

T>0

: the favourite stays the favourite. Delhi over Mumbai (gap 1):

e^{1}\approx2.718

T=1

e^{2}\approx7.389

T=0.5

e^{0.5}\approx1.649

T=2

T\to0

, every gap to the top word is divided by a tiny number, so

p_j/p_{\text{top}}=e^{-(s_{\text{top}}-s_j)/T}\to0

: the top word's share goes to 1. As

T\to\infty

every exponent goes to 0, every ratio to 1, and each of the

n

words gets

1/n

. ∎ Check with §5's numbers:

0.842/0.114\approx7.39\approx e^{2}

, and

0.434/0.263\approx1.65\approx e^{0.5}

Prove it · adding up the surprises is scoring the whole sentence

Claim. For a text $w_1,\dots,w_T$ , the teacher-forcing loss $L=\sum_{t=1}^{T}-\ln p(w_t\mid w_1,\dots,w_{t-1})$ equals $-\ln P(w_1,\dots,w_T)$ . So making $L$ small is exactly maximum likelihood (Unit 14).

The chain rule of probability writes a sentence as a string of guesses:

P(w_1,\dots,w_T)=P(w_1)\,P(w_2\mid w_1)\cdots P(w_T\mid w_1,\dots,w_{T-1})

. Each factor is one read-out of the language model, made with the true text so far fed in — teacher forcing supplies exactly these conditions.

Take

-\ln

of both sides; the logarithm turns the product into a sum:

-\ln P=\sum_t-\ln P(w_t\mid\dots)=L

. ∎ In §5's widget the sentence "the train to pune is late." costs a total surprise of 6.507 over its 26 characters, so the tiny model gives the whole sentence probability

e^{-6.507}\approx0.0015

— about 1 in 670.

In one sentence: A recurrent language model learns by guessing each next word with the true text fed in (teacher forcing) and talks by feeding back its own picks — greedy loops, and a temperature $T$ in $\operatorname{softmax}(s/T)$ turns the same scores 2, 1, 0.5, −1 into 0.842 of the time "Delhi" at $T=0.5$ or only 0.434 at $T=2$ .

Backprop through time: blame walks back along the words

The model guessed the last word wrong. How much of that is the fault of the way it handled word 1, word 2, word 3…?

Imagine this

A relay team loses the race. The coach starts at the finish line and walks back along the track, asking each runner: "How much of the delay was yours?"

Every runner here is the same person — our one cell, used at every word. So when the coach adds up the blame for that one person, he adds it from every leg of the race.

An unrolled cell is a deep network (§3), so we train it exactly as in Unit 15: a forward pass that stores every note, then a backward pass with the same two rules. We just walk back along time instead of along layers. People call this backpropagation through time, or BPTT.

Rule A, at every tanh. Multiply the incoming blame by the slope of tanh, which is $1-h_t^2$ (you stored $h_t$ on the way forward).
Rule B, at every $w\,h_{t-1}+u\,x_t+b$ . Blame for the old note: multiply by $w$ . Blame for the weight: blame times the note that came in, $h_{t-1}$ .
The new twist: add up. The same $w$ is used at every step. So its gradient is the sum of what each step reports.

A tiny example. Take §2's cell ( $w=0.5$ , inputs $1,0,0$ , notes $0.7616,\ 0.3634,\ 0.1797$ ). How much does a small nudge to $h_1$ move $h_3$ ? Walk back one step at a time. Each step is "slope of tanh times $w$ ":

$\begin{aligned}\frac{\partial h_3}{\partial h_2}&=w\,(1-h_3^2)\\ &=0.5\,(1-0.1797^2)\approx0.4838\\ \frac{\partial h_2}{\partial h_1}&=w\,(1-h_2^2)\\ &=0.5\,(1-0.3634^2)\approx0.4340\\ \frac{\partial h_3}{\partial h_1}&=0.4838\times0.4340\approx0.2100\end{aligned}$

A nudge to the first note reaches the third note at only about a fifth of its size. The blame going back shrinks the same way. That is the first word's voice fading — now seen from the backward side.

Try: Press ▶ play. Watch four green notes appear, then a red disc that starts big at word 4 and shrinks at every step back. At the last step, see the four reports for $w$ added up to −0.4655. Then press cut the blame after 2 steps: the notes still flow all the way, but the blame stops at word 3, and only two reports are added: −0.3211.

step 1 / 10

Every word casts a vote. Look at the last card of the step machine. The copy of the cell at word 4 says "make $w$ bigger by this much", the copy at word 3 says its own amount, and so on. Because all the copies are one cell (§3), the one weight hears every vote and adds them up. A word near the end, where the blame is still loud, votes loudly; a word far back votes in a whisper.

With a real note, each step is a matrix. When the note is a list of numbers, one step back multiplies the blame by a matrix, not a number. Going forward, a small nudge to $\mathbf h_{t-1}$ changes $\mathbf h_t$ by $\operatorname{diag}(1-\mathbf h_t^2)\,W$ times that nudge (the Jacobian of Unit 6): first $W$ mixes the nudge, then each entry is scaled by its own tanh slope. So the blame reaching word $k$ from word $T$ is a product of $T-k$ such matrices — and every one of them contains the same $W$ . Keep that sentence in mind: it is the whole of §7.

Truncated BPTT: cut the walk short. A novel has 100 000 words. Storing every note and walking the blame back through all of them for every update is far too slow. So in practice we cut the text into chunks of $k$ words — say $k=20$ — and walk the blame back only inside a chunk. A 1 000-word text with $k=20$ becomes $1000/20=$ 50 chunks. The note still flows forward from one chunk into the next, so the model can still use old words. But no blame crosses a cut, so the model can never learn a link between two words more than $k$ apart.

	blame walks back	cost per update	links it can learn
full BPTT	to the first word	grows with the whole text	any distance (if the blame survives, §7)
truncated, k words	back to the last cut	about k steps, fixed	at most k words apart

Rule of thumb. Choose $k$ a little longer than the longest link you need the model to learn — a sentence or two — and no longer than your memory allows.

Why it works

An unrolled RNN is just a very deep network in disguise, so the chain rule of Unit 15 needs no new ideas: blame multiplies along the chain, and a shared weight adds up its blame from every place it was used. The only new facts are about time: the chain is as long as the text, and every link in it contains the same matrix.

The realization

$\begin{gathered}\frac{\partial L}{\partial w}=\sum_{t=1}^{T}\frac{\partial L}{\partial z_t}\,h_{t-1}\\ \frac{\partial \mathbf h_T}{\partial \mathbf h_k}=\prod_{t=k+1}^{T}\operatorname{diag}(1-\mathbf h_t^2)\,W\end{gathered}$

Backprop through time is Unit 15's backward pass on the unrolled chain. The shared weight collects blame from every step, added up. The blame reaching an early word is a product of one factor per step, each containing the same $W$ — so if each factor shrinks things, the product shrinks fast.

Trap

Truncating the blame does not truncate the memory. The note is carried forward across every cut, so word 45 can still feel word 5. What is lost is only the training signal: the model gets no blame telling it how word 5 should have helped word 45.

Pause & predict

Suppose every step's factor $w(1-h_t^2)$ is exactly 0.5. How much of a nudge to $h_1$ reaches $h_5$ ?

Pause & predict

In the step machine, the copies of the cell at words 2, 3 and 4 report $-0.1444$ , $-0.1588$ and $-0.1623$ for $w$ ; word 1 reports 0. The four copies are one cell. What is $\partial L/\partial w$ ?

Pause & predict

A 1 000-word text is trained with truncated BPTT, chunks of $k=20$ words (1–20, 21–40, 41–60, …). Can training learn that word 5 should change the guess at word 45?

If you want the algebra · 4 proofs, step by step

Prove it · the blame to an early note is a product of one factor per step

Claim. For $h_t=\tanh(z_t)$ , $z_t=wh_{t-1}+ux_t+b$ : $\dfrac{\partial h_T}{\partial h_k}=\displaystyle\prod_{t=k+1}^{T}w\,(1-h_t^2)$ .

One step:

\dfrac{\partial h_t}{\partial h_{t-1}}=\tanh'(z_t)\cdot\dfrac{\partial z_t}{\partial h_{t-1}}=(1-h_t^2)\cdot w

. Rule A (the tanh slope) times Rule B (the weight on the wire).

h_T

depends on

h_k

only through the chain

h_k\to h_{k+1}\to\cdots\to h_T

. The chain rule multiplies the steps:

\frac{\partial h_T}{\partial h_k}=\frac{\partial h_T}{\partial h_{T-1}}\cdots\frac{\partial h_{k+1}}{\partial h_k}.

∎ §6's numbers:

0.4838\times0.4340\approx0.2100

. For a vector note the factors are matrices,

\operatorname{diag}(1-\mathbf h_t^2)\,W

, multiplied in order.

Prove it · a shared weight's gradient is the sum of every step's report

Claim. $\dfrac{\partial L}{\partial w}=\displaystyle\sum_{t=1}^{T}\frac{\partial L}{\partial z_t}\,h_{t-1}$ .

Pretend each step had its own copy

w_t

, all set equal to

w

. Then

L

is a function

L(w_1,\dots,w_T)

evaluated at

w_1=\dots=w_T=w

. A trick: give each use of the weight its own name.

Moving

w

moves every copy at once, so by the chain rule (Unit 6)

\dfrac{dL}{dw}=\sum_t\dfrac{\partial L}{\partial w_t}

. Rule B gives each copy's share:

\dfrac{\partial L}{\partial w_t}=\dfrac{\partial L}{\partial z_t}\,h_{t-1}

. ∎ Step machine:

-0.1444-0.1588-0.1623+0=-0.4655

. The copy at step 1 reports 0 because

h_0=0

Prove it · with a note of many numbers, each step back multiplies by the same matrix

Claim. For $\mathbf h_t=\tanh(W\mathbf h_{t-1}+U\mathbf x_t+\mathbf b)$ : $\dfrac{\partial\mathbf h_t}{\partial\mathbf h_{t-1}}=\operatorname{diag}(1-\mathbf h_t^2)\,W$ , and the blame moves back as $\dfrac{\partial L}{\partial\mathbf h_{t-1}}=W^{\top}\operatorname{diag}(1-\mathbf h_t^2)\,\dfrac{\partial L}{\partial\mathbf h_t}$ .

Entry

i

of the new note is

\tanh(z_i)

with

z_i=\sum_jW_{ij}h_{t-1,j}+\dots

. So

\partial h_{t,i}/\partial h_{t-1,j}=(1-h_{t,i}^2)\,W_{ij}

: Rule A (the slope of entry

i

) times Rule B (the weight on the wire from

j

i

). As a matrix, that is row

i

W

scaled by

1-h_{t,i}^2

\operatorname{diag}(1-\mathbf h_t^2)\,W

. This is the Jacobian of Unit 6, one step of the chain.

By the chain rule in matrix form (Unit 6), the blame for

h_{t-1,j}

collects

\sum_i\frac{\partial L}{\partial h_{t,i}}(1-h_{t,i}^2)W_{ij}

: that is the transposed Jacobian times the blame,

W^{\top}\operatorname{diag}(1-\mathbf h_t^2)\,\partial L/\partial\mathbf h_t

. ∎ Walking back

T-k

steps, the blame meets

W^{\top}

exactly

T-k

times, with a tanh slope (never more than 1) between each — the powers of §7.

Prove it · truncated BPTT: how many chunks, and how far the blame reaches

Claim. Cutting a text of $N$ words into chunks of $k$ words gives $\lceil N/k\rceil$ chunks, and the blame from word $t$ reaches back at most to the first word of its chunk, $k-1$ words earlier. For $N=1000$ , $k=20$ : 50 chunks, and word 45's blame stops at word 41.

Chunk

m

holds words

(m-1)k+1

mk

. The last chunk ends at or after word

N

when

mk\ge N

, so there are

\lceil N/k\rceil

chunks:

\lceil1000/20\rceil=50

. Word 45 is in chunk

\lceil45/20\rceil=3

, which holds words 41 to 60.

The backward walk stops at the start of the chunk, so from word

t

it covers only words

t,t-1,\dots

down to the chunk's first word: at most

k-1

steps back. ∎ The forward pass is not cut: the note leaving chunk 2 is the note entering chunk 3, so word 45 still feels word 5 — it just can never be taught to use it.

In one sentence: Backprop through time runs Rule A and Rule B back along the unrolled chain, adds up the shared weight's votes from every step, and multiplies one factor per step on the way back — 0.4838 × 0.4340 ≈ 0.21 here — while truncated BPTT cuts that walk into chunks of $k$ words (50 chunks for 1 000 words and $k=20$ ).

Why memory fades or explodes

Why can a plain recurrent network remember the last few words but not something from twenty words ago?

Imagine this

Ten children stand in a line and play the whisper game. The teacher whispers a word to the first child, and each child passes on what they heard. Suppose every child speaks at exactly half the loudness they heard. That is ten halvings, so the tenth child says the word about a thousand times more quietly than the teacher did. Nobody hears it.

Now suppose every child speaks one and a half times louder. By the tenth child it is a shout, 58 times the start. Only a line where everyone speaks exactly as loud as they heard keeps the word alive. A recurrent network's memory and its blame both travel down such a line.

Strip the cell down to its loop: no tanh, no new words, just $h_t=w\,h_{t-1}$ . Every step is one child in the line, passing on $w$ times what it heard. After $T$ steps,

$h_T=w^T\,h_0,\qquad\frac{\partial h_T}{\partial h_0}=w^T.$

The same power shows up forward (how much of the first word is left) and backward (how much blame reaches it). Put in numbers for $T=10$ :

$w=0.5$ : $0.5^{10}\approx0.000977$ . Less than a thousandth is left. The memory has faded.
$w=1.5$ : $1.5^{10}\approx57.67$ . The blame is 58 times too big. It has exploded.
$w=1$ : $1^{10}=1$ . Held steady — the knife edge.

Powers are ruthless. A number a little below 1 goes to 0; a little above 1 runs away. After 20 steps, $w=0.9$ leaves $0.9^{20}\approx0.1216$ and $w=1.1$ gives $1.1^{20}\approx6.7275$ . After 50 steps even $0.9$ leaves $0.005$ , and $1.1$ gives about 117.

Why this matters. Read "The keys to the old wooden cabinet in the hall are …". The verb must be "are", not "is", because of "keys" — nine words back. To learn that, the blame from "are" must travel back nine steps to "keys". If each step passes on half, only $0.5^{9}\approx0.00195$ of it arrives. The network never finds out that "keys" was the word that mattered.

Try: Set $w=0.5$ and read the gold bar at $t=10$ : 0.000977. Press explode 1.5: 57.67. Now slide $w$ slowly across 1 and watch the bars tip from falling to rising. Turn on with tanh: the dashed outlines are the plain powers, and the squeeze only ever makes the bars shorter. At $w=1.5$ it even turns the explosion into fading — the note gets stuck near 0.86, where tanh is flat.

w

0.5

t

Now with a real note: a matrix. When the note is a list of numbers, the loop is $\mathbf h_t=W\mathbf h_{t-1}$ , so $\mathbf h_T=W^T\mathbf h_0$ . Powers of a matrix. How do we see where those go? With the eigenvectors of Unit 4: the special directions that $W$ only stretches, never turns. Along an eigenvector with eigenvalue $\lambda$ , each step just multiplies by $\lambda$ — back to the one-number whisper line.

A tiny example: $W=\begin{pmatrix}0.9&0.4\\ 0.1&0.6\end{pmatrix}$ . Its eigenvalues are $\lambda_1=1$ and $\lambda_2=0.5$ (check: they add to the trace $1.5$ and multiply to the determinant $0.54-0.04=0.5$ ). The directions are $(4,1)$ for $\lambda_1=1$ and $(1,-1)$ for $\lambda_2=0.5$ .

Start the note at $\mathbf v=(1,0)$ . Split it along the two directions: $(1,0)=0.2\,(4,1)+0.2\,(1,-1)$ . Now every step keeps the first part whole and halves the second:

$\begin{aligned}W^t\mathbf v&=0.2\,(4,1)+0.2\cdot0.5^t\,(1,-1)\\ &\longrightarrow\ (0.8,\,0.2).\end{aligned}$

After one step, $(0.9,0.1)$ ; after two, $(0.85,0.15)$ ; after ten, almost exactly $(0.8,0.2)$ . The part along the $\lambda=1$ direction survives. The part along the $\lambda=0.5$ direction halves every step and is gone. So a matrix is a bundle of whisper lines, one per eigen-direction, each with its own loudness factor $\lambda$ .

That gives the rule: look at the biggest eigenvalue size, $|\lambda|_{\max}$ (people call it the spectral radius). Below 1, everything fades. Above 1, something explodes. Backward, the blame is multiplied at each step by the transpose $W^{\top}$ (rows and columns swapped) — and $W^{\top}$ has the same eigenvalues as $W$ , so the same verdict holds for the blame. And the tanh slope $1-h^2$ is never more than 1, so it can only add fading.

Try: On keep (the worked matrix) press ▶ multiply: the blue part ( $\lambda=1$ ) stays, the orange part halves at every step, and the dots settle at $(0.8,0.2)$ , where the bars go flat at 0.8246. Try fade: both parts shrink and the bars fall. Explode: the bars climb past 1 and the dots run off the map. Rotate: there are no eigen-lines — every direction turns — and the dots walk round a circle with every bar at exactly 1.

W =

steps shown30

biggest \|λ\|	forward: the note	backward: the blame	what helps
below 1	old words fade away	vanishes before it reaches early words	gates: an express lane (§9, §10)
exactly 1	kept along that direction	reaches back at full size	the knife edge gates try to hold
above 1	grows until tanh flattens it	explodes while the note is small, and training jumps wildly (once tanh has squeezed the note flat, the blame fades instead)	clip the gradient, start near 1 (§8)

Why it works

Split the note along the eigen-directions and the matrix stops mixing: each part is multiplied by its own $\lambda$ at every step, exactly like one child in the whisper line. After many steps, the part with the biggest $|\lambda|$ outgrows or outlasts all the others. That is why one number — the spectral radius — decides the fate of both the memory and the blame.

The realization

$\begin{gathered}\mathbf h_T=W^T\mathbf h_0\\ |\lambda|_{\max}<1\Rightarrow\text{fades}\\ |\lambda|_{\max}>1\Rightarrow\text{explodes}\end{gathered}$

The gradient through time carries powers of one matrix. So its eigenvalues decide: along each eigen-direction, memory is multiplied by $\lambda$ at every step. Below 1 it fades away, above 1 it blows up, and only exactly 1 holds on. This is the central fact of recurrent networks.

Trap

Small entries do not mean fading. Every entry of $W=\begin{pmatrix}0.9&0.4\\ 0.1&0.6\end{pmatrix}$ is below 1, yet one eigenvalue is exactly 1 and part of its memory never fades. And a matrix with big entries can still fade. Look at the eigenvalues, not the entries.

Pause & predict

A one-number loop has $w=0.5$ . After how many steps is the memory first below one thousandth ( $10^{-3}$ )?

Pause & predict

$W=\begin{pmatrix}0.5&0\\ 0&1.2\end{pmatrix}$ and the note starts at $(1,1)$ . What happens over many steps?

Try it, then answer

In the eigen-memory widget, load rotate. Its matrix turns the note by about 37° each step without stretching it. What are the sizes $|\lambda|$ of its eigenvalues, and what does the note's length do?

Pause & predict

Every entry of $W=\begin{pmatrix}0.9&0.4\\ 0.1&0.6\end{pmatrix}$ is smaller than 1. Start the note at $(1,0)$ and multiply by $W$ again and again. Where does the note end up?

If you want the algebra · 3 proofs, step by step

Prove it · powers of W, one eigen-direction at a time

Claim. If $W\mathbf e_1=\lambda_1\mathbf e_1$ , $W\mathbf e_2=\lambda_2\mathbf e_2$ and $\mathbf v=a\mathbf e_1+b\mathbf e_2$ , then $W^t\mathbf v=a\lambda_1^t\mathbf e_1+b\lambda_2^t\mathbf e_2$ . For $W=\begin{pmatrix}0.9&0.4\\ 0.1&0.6\end{pmatrix}$ , $\mathbf v=(1,0)$ : $W^t\mathbf v=0.2(4,1)+0.2\cdot0.5^t(1,-1)$ .

A matrix spreads over sums:

W\mathbf v=aW\mathbf e_1+bW\mathbf e_2=a\lambda_1\mathbf e_1+b\lambda_2\mathbf e_2

. Apply

W

again and each part gets one more factor of its own

\lambda

. Along an eigenvector the matrix is just a number (Unit 4).

For the worked

W

\det(W-\lambda I)=\lambda^2-1.5\lambda+0.5=(\lambda-1)(\lambda-0.5)

. Check the directions:

W(4,1)=(3.6+0.4,\ 0.4+0.6)=(4,1)

and

W(1,-1)=(0.5,-0.5)

. Trace 1.5 is the sum of the eigenvalues; determinant 0.5 is their product.

Split

(1,0)=a(4,1)+b(1,-1)

4a+b=1

and

a-b=0

, so

a=b=0.2

. Hence

W^t(1,0)=0.2(4,1)+0.2\cdot0.5^t(1,-1)\to(0.8,0.2)

. ∎

t=1

(0.8+0.1,\ 0.2-0.1)=(0.9,0.1)

— the same as

W(1,0)

directly.

Prove it · the tanh slope can only add fading

Claim. For the scalar cell, $\Big|\dfrac{\partial h_T}{\partial h_k}\Big|\le|w|^{T-k}$ . So if $|w|<1$ the blame fades at least as fast as $|w|^{T-k}$ . For a vector note the same holds with the largest stretch of $W$ in place of $|w|$ .

Each factor is

w(1-h_t^2)

, and

0<1-h_t^2\le1

because

|h_t|<1

. So each factor's size is at most

|w|

. The squeeze of tanh never amplifies; it only shrinks, and shrinks most when the note is near ±1 ("saturated").

A product of

T-k

numbers, each of size at most

|w|

, has size at most

|w|^{T-k}

. ∎ The bound only works in one direction: with

|w|>1

the tanh can still tame an explosion, but while the notes stay small, the blame grows like

|w|^{T-k}

Prove it · complex eigenvalues turn the note, and their size sets the length

Claim. $R=\begin{pmatrix}a&-b\\ b&a\end{pmatrix}$ with $b\ne0$ has eigenvalues $a\pm bi$ , of size $r=\sqrt{a^2+b^2}$ , and $\|R\mathbf v\|=r\|\mathbf v\|$ for every $\mathbf v$ . For the rotate preset, $a=0.8$ , $b=0.6$ , $r=1$ .

\det(R-\lambda I)=(a-\lambda)^2+b^2=0

gives

\lambda=a\pm bi

, each of size

\sqrt{a^2+b^2}

. No real solution: there is no direction that

R

leaves on its own line — every direction gets turned.

\|R(x,y)\|^2=(ax-by)^2+(bx+ay)^2=(a^2+b^2)(x^2+y^2)

. So each step multiplies the length by

r

and turns by the angle

\theta

with

\cos\theta=a/r

. ∎

0.8^2+0.6^2=1

\theta\approx36.87°

: after

t

steps the note has turned

36.87°\,t

and kept its length — the spiral staircase.

In one sentence: Memory and blame pass down a whisper line that multiplies by the same matrix at every step, so after $T$ steps they carry $W^T$ — and the biggest eigenvalue size decides whether they fade (0.5¹⁰ ≈ 0.000977) or explode (1.5¹⁰ ≈ 57.67).

First aid for explosions: clip the gradient, start at the right volume

If the blame can explode, one bad step can undo hours of training. What stops that from happening?

Imagine this

School buses in India carry a speed governor. The driver can press the pedal as hard as he likes; the bus will not go faster than the limit. It still goes where he steers — only not at a crazy speed.

Gradient clipping is a speed governor for training.

When blame explodes, the gradient can become enormous for a single step. Picture the loss as a landscape: mostly gentle slopes, but here and there a cliff — a sudden wall where the loss jumps. That wall is what exploding powers of $W$ look like. Touch the edge of the cliff and the slope is huge. A plain step $\theta\leftarrow\theta-\eta\,\mathbf g$ (Unit 9) then throws the weights far away, and all the progress so far is lost.

The fix is simple. Pick a limit $c$ . If the gradient is longer than $c$ , shrink it to length $c$ and keep its direction:

$\text{if }\|\mathbf g\|>c:\qquad\mathbf g\leftarrow c\,\frac{\mathbf g}{\|\mathbf g\|}.$

Read it aloud: "divide the arrow by its own length, so it has length 1, then stretch it to length $c$ ." A tiny example. $\mathbf g=(30,40)$ , so $\|\mathbf g\|=\sqrt{900+1600}=50$ . With $c=5$ : $\mathbf g\leftarrow5\cdot(30,40)/50=(3,4)$ . Same direction, length 5. A gradient like $(1,2,2)$ , of length 3, is shorter than 5 and passes through untouched.

Clipping only helps with explosions. It cannot help with fading: if the blame reaching word 1 is $10^{-9}$ , there is no signal left to rescue. For fading we need a new design — that is §9.

Try: Press ▶ roll with clipping off: the ball creeps to the edge, feels a gradient of length 10.86 at the wall and is flung off the map at step 9. Turn clipping on and roll again: it walks down the wall in small steps and reaches the valley. Then lower the limit $c$ and see the steps get shorter.

drag the picture to orbit

limit

c

Start at the right volume. The whisper line also tells us how to set up $W$ before training begins. If $W$ is orthogonal — a pure turn or mirror, one of the "motions that don't lie" of Unit 3 — it never changes the length of any note. All its eigenvalues then have size exactly 1: the whisper starts at normal volume, neither fading nor exploding, and training can move it from there. So a common start is a random orthogonal $W$ , or even $W=I$ , the identity ("copy the old note"). The talking model of §5 was started at $0.9$ times an orthogonal matrix — every $|\lambda|=0.9$ , just under the knife edge.

Why it works

An explosion is a long gradient, and a long gradient is dangerous only because of its length: the direction still points downhill. Clipping keeps the useful part (the direction) and throws away the dangerous part (the length). A good start helps from the other side: with every $|\lambda|$ near 1, there is little to explode or to fade in the first place.

The realization

$\mathbf g\leftarrow\min\Big(1,\ \frac{c}{\|\mathbf g\|}\Big)\,\mathbf g$

Clipping keeps the direction of the gradient and caps its length. One exploding step can no longer throw training off a cliff. It is cheap first aid for explosions — and no cure for fading.

Trap

Do not clip each entry on its own. Capping every entry of $(30,40)$ at 5 gives $(5,5)$ , which points at 45° instead of the gradient's 53.13° — you are now walking a different way. Clip the whole arrow by one factor: $(3,4)$ still points at 53.13°.

Pause & predict

The gradient is $\mathbf g=(0,-12,5)$ and the limit is $c=2.6$ . What is the clipped gradient?

Pause & predict

In a plain recurrent network, the blame reaching word 1 of a long text is about $10^{-9}$ . You switch on clipping with $c=5$ . What happens to that blame?

If you want the algebra · 2 proofs, step by step

Prove it · clipping keeps the direction and caps the step

Claim. The clipped gradient $\mathbf g'=\min(1,c/\|\mathbf g\|)\,\mathbf g$ points the same way as $\mathbf g$ , has length $\min(\|\mathbf g\|,c)$ , and the step $\eta\mathbf g'$ is never longer than $\eta c$ . It is still a downhill direction.

\mathbf g'

\mathbf g

times a positive number

s=\min(1,c/\|\mathbf g\|)

, so it points the same way, and

\|\mathbf g'\|=s\|\mathbf g\|=\min(\|\mathbf g\|,c)

. Example:

s=5/50=0.1

turns

(30,40)

into

(3,4)

The slope of the loss along the step

-\eta\mathbf g'

-\eta\,\mathbf g\cdot\mathbf g'=-\eta s\|\mathbf g\|^2<0

. ∎ Downhill at the start, as a plain step is (Unit 9) — only shorter. Shrinking each entry to at most

c

separately would turn the direction; that is why we scale the whole vector.

Prove it · an orthogonal W never changes the note's length, so every |λ| is 1

Claim. If $Q^{\top}Q=I$ , then $\|Q\mathbf v\|=\|\mathbf v\|$ for every $\mathbf v$ , and every eigenvalue of $Q$ has size $|\lambda|=1$ . So $0.9\,Q$ has every $|\lambda|=0.9$ .

\|Q\mathbf v\|^2=(Q\mathbf v)^{\top}(Q\mathbf v)=\mathbf v^{\top}Q^{\top}Q\,\mathbf v=\mathbf v^{\top}\mathbf v=\|\mathbf v\|^2

. The motions that don't lie of Unit 3: turns and mirrors.

Q\mathbf v=\lambda\mathbf v

with

\mathbf v\ne\mathbf 0

(allowing complex numbers), lengths give

|\lambda|\,\|\mathbf v\|=\|\mathbf v\|

, so

|\lambda|=1

. Scaling by 0.9 scales every eigenvalue by 0.9. ∎ The rotate matrix of §7 is orthogonal: its eigenvalues

0.8\pm0.6i

have size 1.

In one sentence: When the gradient is longer than a limit $c$ , shrink it to length $c$ without turning it — $(30,40)$ becomes $(3,4)$ — and start $W$ with every $|\lambda|$ near 1; this tames explosions but cannot rescue memory that has faded.

The LSTM: a memory with an express lane

If every step multiplies the memory by a number below 1, how can a network ever carry a word across fifty steps?

Imagine this

A shopkeeper's ledger — the bahi-khata — passes from day to day. Each evening he makes three small choices. With an eraser: what to rub out. With a pen: what to write in. And at the window: what to read out to the customer.

The ledger itself is never torn up and rewritten. It rides from day to day almost untouched. That is the LSTM's secret.

Why does the plain cell forget? At every step its whole note is multiplied by $W$ and squeezed by tanh — the whisper line of §7. There is no way to carry a number along unchanged. So in 1997 Hochreiter and Schmidhuber added a second note that runs along the top: the cell state $\mathbf c_t$ . People call the whole design LSTM, "long short-term memory". (The eraser below, the forget gate, came three years later, from Gers, Schmidhuber and Cummins.) Think of the cell state as an express lane: at each step only two gentle things happen to it.

First, three gates are computed. A gate is a list of numbers between 0 and 1, made with a sigmoid (Unit 15). 0 means "shut", 1 means "wide open". Each gate looks at the new word and the last note:

the forget gate $\mathbf f_t$ — the eraser: how much of the old cell state to keep;
the input gate $\mathbf i_t$ — the pen: how much of the new candidate to write in;
the output gate $\mathbf o_t$ — the window: how much of the cell state to show as the note.

The candidate $\mathbf g_t=\tanh(\dots)$ is what the pen would write. Then:

$\begin{aligned}\mathbf c_t&=\mathbf f_t\odot\mathbf c_{t-1}+\mathbf i_t\odot\mathbf g_t\\ \mathbf h_t&=\mathbf o_t\odot\tanh(\mathbf c_t)\end{aligned}$

Read the first line aloud: "keep the part of the old ledger the eraser allows, and add the part of the draft the pen allows." The second: "show the ledger through the window, squeezed." The symbol $\odot$ just means "multiply entry by entry". Each gate has its own weights, like a small layer: $\mathbf f_t=\sigma(W_f\mathbf h_{t-1}+U_f\mathbf x_t+\mathbf b_f)$ , and the same shape for $\mathbf i_t$ , $\mathbf o_t$ and $\mathbf g_t$ (with tanh for $\mathbf g_t$ ).

A tiny example. Forget gate $f=0.9$ , old cell state $c_{t-1}=1$ , input gate $i=0.2$ , candidate $g=0.5$ :

$c_t=0.9\cdot1+0.2\cdot0.5=0.9+0.1=1.0.$

We erased a tenth, then wrote in a little. With the window fully open ( $o=1$ ), the note is $h_t=\tanh(1.0)\approx0.7616$ .

The full step, from the weights. In a real cell nobody sets the gates by hand: each gate adds up its weighted inputs into a score, then squeezes it. Take a one-number LSTM with $c_{t-1}=1$ , $h_{t-1}=0$ , input $x=1$ , and weights that give the four scores $z_f=2x=2$ , $z_i=0$ , $z_o=x=1$ , $z_g=x=1$ :

$\begin{aligned}f&=\sigma(2)\approx0.8808\\ i&=\sigma(0)=0.5\\ o&=\sigma(1)\approx0.7311\\ g&=\tanh(1)\approx0.7616\\ c_t&=0.8808\cdot1+0.5\cdot0.7616\\ &=0.8808+0.3808=1.2616\\ h_t&=0.7311\cdot\tanh(1.2616)\\ &=0.7311\cdot0.8515\approx0.6225\end{aligned}$

Notice $c_t=1.2616$ is bigger than 1: nothing squeezes the ledger itself. Only the note $h_t$ is squeezed, on its way out through the window.

Try: Start from the worked example and press ▶ one step: 0.9 stays, 0.1 flows in, the tank reads 1.0. Close the drain completely ( $f=1$ ) and shut the inlet ( $i=0$ ): the tank holds its level exactly — memory with no fading at all. Then shut the window ( $o=0$ ): the tank is still full, but the note shows 0. Last, open from scores: the scores 2, 0, 1, 1 are squeezed into $f=0.8808$ , $i=0.5$ , $o=0.7311$ , $g=0.7616$ , and the tank rises to 1.2616 with the note at 0.6225.

old

c_{t-1}

forget

f

0.9

input

i

0.2

candidate

g

0.5

output

o

score

z_f

score

z_i

score

z_g

score

z_o

Why the express lane saves the blame. Look at how the cell state moves from one step to the next: $\mathbf c_t=\mathbf f_t\odot\mathbf c_{t-1}+\dots$ Along this lane there is no $W$ to multiply by, and no tanh to squeeze. So walking back along the lane, the blame is multiplied by just $f$ at each step (§6's product, with a much kinder factor):

$\frac{\partial c_T}{\partial c_k}=f_{k+1}\,f_{k+2}\cdots f_T.$

With $f=0.9$ for ten steps: $0.9^{10}\approx0.349$ . Compare the plain cell with $w=0.5$ : $0.5^{10}\approx0.000977$ . The express lane carries about 350 times more blame back to the start.

Be honest, though: a gate of 0.9 still fades over long distances. After 50 steps, $0.9^{50}\approx0.00515$ . What saves the LSTM is that the network chooses $f$ at every word. When it needs to remember, it learns to hold $f$ very close to 1: $0.99^{50}\approx0.605$ . And when it wants to forget — at a full stop, say — it can drop $f$ to 0 and wipe the slate.

Start with the lane open. At the start of training all weights are small, so every gate score is near 0 and every gate near $\sigma(0)=0.5$ . A forget gate of 0.5 is the whisper line again: $0.5^{10}\approx0.001$ . So people start the forget gate's shift $b_f$ at 1 or 2. Then $f$ begins near $\sigma(1)\approx0.73$ or $\sigma(2)\approx0.88$ , and $0.8808^{10}\approx0.281$ of the blame survives ten steps instead of a thousandth.

This idea — a lane where the signal is added to, never rewritten — is worth remembering. It returns in Unit 18 as the residual path of every transformer block.

Try: Press ▶ read the sentence and watch the three memories travel along the lanes; gold rings are gates opening. Then open blame through time and slide the length from 5 to 50: the plain cell's blame collapses, the gated cells' stays high. Last, press honest LSTM (f = 0.9) and look at the accuracy at 50 words: 85.5%.

drag the picture to orbit

sentence length20

filler noise0.5

plain cell

w

0.9

LSTM forget

f

0.99

How the three cells are wired (the hand-set weights)

Each word is one number $s$ : chai is $+1$ , coffee is $-1$ , and each filler is a random number between $-A$ and $A$ (the filler noise). A marker $m$ is 1 on the first word and 0 after it.

$\begin{aligned}\text{plain: }&h_t=\tanh(w\,h_{t-1}+s_t)\\ \text{LSTM: }&i_t=\sigma(10m_t-5)\\ &c_t=f\,c_{t-1}+i_t\tanh(s_t)\\ \text{GRU: }&z_t=\sigma(10m_t-5)\\ &h_t=(1-z_t)h_{t-1}\\ &\qquad+z_t\tanh(s_t)\end{aligned}$

So the input gate and the update gate are wide open on the first word ( $\sigma(5)\approx0.993$ ) and nearly shut afterwards ( $\sigma(-5)\approx0.0067$ ). The answer is the sign of the final memory. Because these gates look only at the word, the blame along each lane is exactly $\prod w(1-h^2)$ , $\prod f$ and $\prod(1-z)$ .

Why it works

The plain cell must rewrite its whole note at every word, and every rewrite multiplies by $W$ . The LSTM does not rewrite its ledger: it only erases a little ( $\times f$ ) and adds a little ( $+\,i\,g$ ). Adding does not shrink the blame on its way back, and the erasing is a number the network chooses — so when it matters, the network can keep $f$ near 1 and hear a word from fifty steps away.

The realization

$\begin{gathered}\mathbf c_t=\mathbf f_t\odot\mathbf c_{t-1}+\mathbf i_t\odot\mathbf g_t\\ \frac{\partial c_T}{\partial c_k}=\prod_{t=k+1}^{T}f_t\end{gathered}$

The LSTM keeps a second memory on an express lane. Nothing multiplies it by $W$ or squeezes it — only the forget gate scales it and the input gate adds to it. So blame travels back along the lane multiplied only by $f$ , which the network can hold near 1 for as long as it needs to remember.

Trap

The "forget gate" is really a keep gate. $f=1$ means "keep everything", and $f=0$ means "forget everything". Read it as "how much of the old ledger survives" and you will never get the sign of the story backwards.

Pause & predict

One LSTM step: $f=0.5$ , $c_{t-1}=2$ , $i=1$ , $g=-0.4$ . What is the new cell state $c_t$ ?

Pause & predict

Along the express lane the forget gate stays at $f=0.95$ . How much blame reaches a word 20 steps back?

Try it, then answer

In the memory lab, keep the filler noise at 0.5, press honest LSTM (f = 0.9) and look at the accuracy curves at 50 words. What do you see?

Pause & predict

At the start of training every gate score is near 0. If the forget gate's shift starts at $b_f=0$ , about how much blame survives 10 steps along the lane? And with $b_f=2$ ?

If you want the algebra · 2 proofs, step by step

Prove it · along the express lane the blame is a product of forget gates

Claim. If the gates do not depend on $c_{t-1}$ (as in the memory lab), then $\dfrac{\partial c_T}{\partial c_k}=\displaystyle\prod_{t=k+1}^{T}f_t$ . In a full LSTM this product is the direct path along the lane; other paths go through $\mathbf h$ and $W$ and fade like a plain cell's.

From

c_t=f_tc_{t-1}+i_tg_t

: if

f_t

i_t

g_t

do not change when

c_{t-1}

changes, then

\partial c_t/\partial c_{t-1}=f_t

. No

W

, no tanh slope: the lane is a plain multiply by the gate.

Chain the steps from

k

T

as in §6: the product of the

f_t

. ∎

f=0.9

0.9^{10}\approx0.349

0.9^{50}\approx0.00515

f=0.99

0.99^{50}\approx0.605

. The lab's blame chart draws exactly these products.

Prove it · the cell state is a leaky sum of everything written in

Claim. With $c_0=0$ : $c_T=\displaystyle\sum_{k=1}^{T}\Big(\prod_{t=k+1}^{T}f_t\Big)\,i_kg_k$ . What word $k$ wrote is still there, scaled by the forget gates after it.

Unroll:

c_1=i_1g_1

c_2=f_2i_1g_1+i_2g_2

c_3=f_3f_2i_1g_1+f_3i_2g_2+i_3g_3

. Each new step multiplies everything already in the tank by

f

, then adds a new term.

The pattern holds at every step (if it holds for

c_{T-1}

, multiply by

f_T

and add

i_Tg_T

). ∎ In the lab, the input gate is 0.993 on the first word and 0.0067 afterwards, so the first word's term is about

0.993\tanh(1)\,f^{T-1}

and each filler adds almost nothing.

In one sentence: The LSTM carries a cell state on an express lane, $c_t=f\,c_{t-1}+i\,g$ — 0.8808 · 1 + 0.5 · 0.7616 = 1.2616 in the full step — so blame flows back multiplied only by the forget gate, $0.9^{10}\approx0.349$ instead of $0.5^{10}\approx0.000977$ , and a forget shift of 1–2 starts that lane open.

The GRU: a lighter cousin with one blend dial

Does an express lane really need a separate tank and three gates — or can one dial do the job?

Imagine this

A chaiwala keeps one big pot on the stove all day. Every hour he pours in some fresh tea. A little fresh, and the pot tastes much as before. A lot fresh, and the old taste is gone.

How much fresh tea to pour — that is one dial. The GRU is built around this dial.

In 2014 Cho and colleagues asked: can we keep the express lane with fewer parts? Their GRU ("gated recurrent unit") drops the separate cell state. It keeps one note and two gates:

the update gate $\mathbf z_t$ — the dial: how much new to pour in;
the reset gate $\mathbf r_t$ — how much of the old note to consult while drafting the new one.

$\begin{aligned}\tilde{\mathbf h}_t&=\tanh\big(W(\mathbf r_t\odot\mathbf h_{t-1})+U\mathbf x_t+\mathbf b\big)\\ \mathbf h_t&=(1-\mathbf z_t)\odot\mathbf h_{t-1}+\mathbf z_t\odot\tilde{\mathbf h}_t\end{aligned}$

Read the first line aloud: "draft a fresh note from the new word and the part of the old note the reset gate lets you look at." The second line is a blend: part old note, part fresh draft, with the dial $\mathbf z_t$ between 0 and 1. When $z$ is near 0, the old note passes straight through — the express lane again, now multiplied by $1-z$ at each step. The reset gate is how a GRU starts a new thought: with $r$ near 0 the draft ignores the old note, as if the sentence had just begun.

A tiny example. Old note $h_{t-1}=0.8$ , candidate $\tilde h_t=-0.4$ , dial $z=0.25$ :

$\begin{aligned}h_t&=0.75\cdot0.8+0.25\cdot(-0.4)\\ &=0.6-0.1=0.5.\end{aligned}$

Three quarters old, one quarter new. And since it is a blend, the new note always lies between the old note and the candidate.

The full step, from the weights. A one-number GRU with $h_{t-1}=0.5$ and input $x=1$ . The update score is 0 and the reset score is 2, so $z=\sigma(0)=0.5$ and $r=\sigma(2)\approx0.8808$ . The draft uses weight 1 on the word and 1 on the consulted old note:

$\begin{aligned}\tilde h_t&=\tanh(1\cdot x+1\cdot r\,h_{t-1})\\ &=\tanh(1+0.8808\cdot0.5)\\ &=\tanh(1.4404)\approx0.8938\\ h_t&=0.5\cdot0.5+0.5\cdot0.8938\approx0.6969\end{aligned}$

Try: Turn the dial $z$ from 0 to 1 and watch the new note slide from the old arrow to the candidate along a straight line: at $z=0.25$ it is (0.5, 0), at $z=0.5$ it is (0.2, 0.2). Lower the reset gate $r$ : the candidate stops looking at the old note. Press ▶ ten steps with $z=0.1$ : the note drifts only slowly — a long memory.

update

z

0.25

reset

r

What do the gates cost? Each gate, and each candidate, is a small layer of its own: a matrix for the old note ( $h\times h$ ), a matrix for the word ( $h\times d$ ) and a shift ( $h$ ). That is $h(h+d)+h$ numbers — exactly a plain cell's worth. A plain cell has one such block, a GRU has three ( $\mathbf z$ , $\mathbf r$ , the candidate), an LSTM four ( $\mathbf f$ , $\mathbf i$ , $\mathbf o$ , $\mathbf g$ ).

For a note of $h=128$ numbers and words of $d=64$ numbers: one block is $128\cdot192+128=24\,704$ . So a plain cell has 24 704 numbers to learn, a GRU $3\times24\,704=74\,112$ , and an LSTM $4\times24\,704=98\,816$ .

Plain, GRU or LSTM?

	plain cell	GRU	LSTM
gates	none	2: update $z$ , reset $r$	3: forget $f$ , input $i$ , output $o$
memories	one note $\mathbf h$	one note $\mathbf h$	note $\mathbf h$ + ledger $\mathbf c$
numbers (h = 128, d = 64)	24 704	74 112	98 816
blame along the lane	$\prod w(1-h^2)$ : fades	$\prod(1-z)$ : can stay near 1	$\prod f$ : can stay near 1
choose it when	short patterns, speed	you want gates at ¾ of the cost	long, complex sequences

Rule of thumb. On most tasks the GRU and the LSTM do about equally well; try the GRU first when data or memory is short, the LSTM when the sequences are long and rich. The plain cell is best kept for short memories — and for understanding the other two.

Why it works

A blend never has to shrink what it keeps: with $z$ near 0 the old note is copied almost exactly, so both the memory and the blame ride along at $1-z\approx1$ per step. The GRU simply merges the LSTM's eraser and pen into one dial — whatever is not kept is replaced — and that turns out to be enough.

The realization

$\mathbf h_t=(1-\mathbf z_t)\odot\mathbf h_{t-1}+\mathbf z_t\odot\tilde{\mathbf h}_t$

A GRU is one blend dial: keep the old note, or pour in the new. With $z$ near 0 the old note rides straight through — an express lane without a separate tank. It learns three blocks of weights where an LSTM learns four and a plain cell one.

Trap

Check which way the dial turns. On this page $z$ near 0 means "keep the old note" and $z$ near 1 means "replace it". Some books and code libraries swap the roles of $z$ and $1-z$ . The idea is the same; read which one multiplies the old note before you compute.

Pause & predict

A GRU step with $z=0.5$ , old note $h_{t-1}=0.2$ and candidate $\tilde h_t=0.6$ . What is $h_t$ ?

Pause & predict

An LSTM has a note of $h=10$ numbers and words of $d=5$ numbers. How many numbers does it learn?

Pause & predict

The reset gate is shut: $r=0$ . What does the GRU's draft $\tilde h_t$ depend on?

Pause & predict

A GRU's update gate stays at exactly $z=0$ for 50 words. What happens to its note?

If you want the algebra · 2 proofs, step by step

Prove it · plain 1 block, GRU 3, LSTM 4

Claim. With note size $h$ and word size $d$ , a plain cell learns $P=h(h+d)+h$ numbers, a GRU $3P$ and an LSTM $4P$ . For $h=128$ , $d=64$ : 24 704, 74 112, 98 816.

Every gate and every candidate has the form

\operatorname{act}(W_\ast\mathbf h_{t-1}+U_\ast\mathbf x_t+\mathbf b_\ast)

with its own

W_\ast

(

h\times h

U_\ast

(

h\times d

\mathbf b_\ast

(

h

P

numbers each. The reset gate multiplies the old note before

W

, but it does not change

W

's shape.

Count the blocks: plain has one; GRU has

\mathbf z,\mathbf r,\tilde{\mathbf h}

; LSTM has

\mathbf f,\mathbf i,\mathbf o,\mathbf g

. And

P=128\cdot192+128=24\,576+128=24\,704

. ∎ Any read-out layer

V

comes on top and is the same for all three.

Prove it · a GRU's note is always a blend, so it stays between −1 and 1

Claim. If $0\le z\le1$ , then $h_t=(1-z)h_{t-1}+z\tilde h_t$ lies between $h_{t-1}$ and $\tilde h_t$ . So if $|h_{t-1}|\le1$ (and $|\tilde h_t|<1$ , a tanh), then $|h_t|\le1$ .

Rewrite:

h_t=h_{t-1}+z\,(\tilde h_t-h_{t-1})

. It starts at the old note and moves a fraction

z

of the way toward the candidate.

z=0.25

: from 0.8, a quarter of the way to −0.4, which is

0.8-0.3=0.5

A point between two numbers in

[-1,1]

is itself in

[-1,1]

. Entry by entry, the same holds for vectors. ∎ So a GRU needs no squeeze after the blend. With

z

near 0 it moves hardly at all:

\partial h_t/\partial h_{t-1}

has the direct part

1-z

, close to 1.

In one sentence: A GRU blends old and new with one dial, $h_t=(1-z)h_{t-1}+z\tilde h_t$ — 0.5 · 0.5 + 0.5 · 0.8938 ≈ 0.6969 in the full step — uses a reset gate to decide how much old note to consult, and costs three blocks of $h(h+d)+h$ numbers against the LSTM's four.

Reading both ways, and stacking floors

"He said Teddy…" — is Teddy a person? You cannot tell until you read the next word. How can a reader use the words that come later?

Imagine this

Two friends read the same sentence. One starts at the left end, the other at the right end. Then, for every word, they put their two notes side by side.

Now every word knows what came before it and what comes after it.

Compare "He said Teddy bears are on sale" with "He said Teddy Roosevelt was a president". In the first, Teddy is part of "teddy bears" — a toy. In the second, Teddy is a man's name. The words before Teddy are identical. Only the words after it decide. A one-way reader standing at "Teddy" has read "He said Teddy" in both cases, so its note there must be the same. It cannot tell them apart.

A bidirectional RNN runs two cells over the sentence. The forward cell reads left → right, as before. The backward cell reads right → left. Each word's final note is the two notes side by side: $[\overrightarrow{\mathbf h}_t\,;\,\overleftarrow{\mathbf h}_t]$ . The forward half knows the past; the backward half knows the future.

A tiny example. Two one-number cells, both with $w=0.5$ , $u=1$ , $b=0$ , read $x=(0,0,1)$ : nothing, nothing, then one real word at the end.

Forward (words 1, 2, 3): $\overrightarrow h=(0,\ 0,\ \tanh1)\approx(0,\ 0,\ 0.7616)$ . Words 1 and 2 have not seen anything yet.
Backward (words 3, 2, 1): $\overleftarrow h_3=\tanh1\approx0.7616$ , $\overleftarrow h_2=\tanh(0.5\cdot0.7616)\approx0.3634$ , $\overleftarrow h_1=\tanh(0.5\cdot0.3634)\approx0.1797$ — §2's fading numbers, read from the other end.

So word 1's note is $(0,\ 0.1797)$ . Its forward half is blind; its backward half carries word 3's news. Word 1 learns about word 3 only through the backward reader.

Stacking floors. We can also build a recurrent network in floors, like a building. Floor 1 reads the words. Floor 2 reads floor 1's notes as its inputs, one per word, and keeps a note of its own. Time runs along each corridor; depth goes up the stairs. Lower floors catch short, local patterns — letters into words, words into phrases. Higher floors build on those and can follow longer-range patterns. Two to four floors are common.

What they cost ( $h=128$ , $d=64$ ). One-way, one floor: 24 704 (§10). Two-way: two cells, $2\times24\,704=49\,408$ . Two floors: floor 1 is 24 704, and floor 2 reads 128-number notes instead of 64-number words, so it is $128\cdot(128+128)+128=32\,896$ — together 57 600.

	each word's note knows	numbers (h = 128, d = 64)	can it write text?	typical use
one-way	the words before it	24 704	yes	language models, live speech
two-way	the whole sentence	49 408	no	tagging, reading a whole text, the encoder of §12
two floors	the words before it, in two layers of detail	57 600	yes	harder language tasks

Try: In toy numbers, switch between one-way and two-way: word 1's note goes from (0) to (0, 0.1797). Click any word to light up the words that can reach its note. Load the two Teddy sentences: one-way, the note at "Teddy" is the same in both; two-way, it differs. Then press two floors to see a second floor reading the first.

keep old

w

0.5

Why it works

Each reader is an ordinary recurrent cell, so each half of the note is an honest summary of one side of the sentence. Putting the halves side by side costs nothing extra to compute and gives every word a view of the whole sentence. Floors work for the same reason deep networks do (Unit 15): each floor builds a slightly more abstract summary on top of the one below.

The realization

$\begin{gathered}\text{note}_t=[\overrightarrow{\mathbf h}_t\,;\,\overleftarrow{\mathbf h}_t]\\ \text{floor }2\text{ reads floor }1\text{'s notes}\end{gathered}$

A two-way reader gives every word both its past and its future. Floors give the reader depth. Both are built from the same cell, run in a different direction or on a different input.

Trap

A two-way reader cannot write text. Its backward half needs the words to the right — and while you are writing, they do not exist yet. Two-way readers are for jobs where the whole sentence is already there: tagging words, judging a review, or reading the source sentence of a translation (§12).

Pause & predict

In the toy example, $x=(0,0,1)$ . Which reader gives word 1 a non-zero note, and how big is it?

Pause & predict

Can you use a two-way RNN as a language model that writes a sentence word by word, as in §5?

Pause & predict

Two floors of plain cells, $h=128$ , $d=64$ . Floor 1 has 24 704 numbers. How many does floor 2 have?

If you want the algebra · 1 proof, step by step

Prove it · what two-way readers and floors cost

Claim. With note size $h$ and word size $d$ , one plain cell learns $P=h(h+d)+h$ . A two-way reader learns $2P$ , and a second floor that reads the first floor's notes learns $h(h+h)+h$ . For $h=128$ , $d=64$ : 24 704, 49 408, and $24\,704+32\,896=57\,600$ for two floors.

The forward and the backward reader are two separate cells of the same shape, each with its own

W

U

\mathbf b

P+P=2P=49\,408

. Their notes sit side by side, so a two-way note has

2h=256

numbers.

Floor 2's input at each word is floor 1's note, which has

h

numbers instead of

d

. By §3's count its

U

h\times h

h\cdot h+h\cdot h+h=128\cdot256+128=32\,896

. ∎ When

h>d

, every floor above the first costs more than the first. A floor on top of a two-way floor reads

2h

numbers per word, so each of its readers learns

h(h+2h)+h

In one sentence: A two-way reader puts a forward note and a backward note side by side — word 1 of $x=(0,0,1)$ gets $(0,\ 0.1797)$ — so every word sees the whole sentence (but it cannot write text), and floors stack readers on readers: 24 704, 49 408 and 57 600 numbers for one-way, two-way and two floors.

Encoder–decoder: translate through one summary

English and Hindi sentences have different lengths and different word orders. How can one machine read one and write the other?

Imagine this

An interpreter at a meeting listens to a whole sentence in English. She holds its meaning in her head. Then she speaks it in Hindi, word by word.

Now give her a rule: she may keep only one small card of notes between listening and speaking. A short sentence fits easily. A very long speech does not.

To translate, the input and the output have different lengths and different word orders. "The train to Delhi is late" is six English words; in Hindi it is "Delhi ki train late hai", five words, in another order. So we use two recurrent cells:

The encoder reads the English sentence word by word. Its final note, $\mathbf s=\mathbf h_T$ , is the summary of the whole sentence (people also call it the context vector). The encoder is often a two-way reader (§11), because the whole English sentence is already there; then the summary puts the two readers' final notes side by side.
The decoder is a language model (§5) that starts from that summary and writes the Hindi sentence one word at a time. After each word it reads what it just wrote as its next input, and it stops when it writes a special word, $\langle\text{end}\rangle$ .

Training uses §5's trick. While learning, we feed the decoder the true previous Hindi word, not its own guess — teacher forcing — so one early slip does not ruin the rest of the sentence. At test time there is no teacher, and the decoder must use its own words.

The bottleneck. The summary has a fixed size — say 8 numbers in our toy, 512 in a real system. Picture a 5-word sentence and a 50-word paragraph, each word bringing 8 numbers of its own. The short one pours 40 numbers into the 8-number card: a squeeze of 5 to 1. The long one pours 400 numbers into the same 8: a squeeze of 50 to 1. And the first word must survive the whole encoder: if the encoder keeps 90% of an old word at each step (a toy rate), word 1 of a 50-word sentence speaks in the summary at $0.9^{49}\approx0.0057$ of its first strength. This is what researchers found with the first neural translation systems in 2014 (Cho and colleagues): good on short sentences, worse and worse on long ones.

Try: Press ▶ translate and follow the words through the orb. Then press 50 words: the orb stays the same size, 400 numbers crowd into its 8 slots, and word 1's gold spark fades to 0.0057. Press 5 words to compare: 40 numbers, word 1 at 0.6561. Switch on teacher forcing to see which words the decoder is fed while training.

drag the picture to orbit

sentence length6

Why it works

The encoder is trained together with the decoder, through the summary. So the blame from every wrong Hindi word flows back into the encoder and teaches it what to put on the card: whatever the decoder needs to write a good translation. The two halves learn a private code between them — and for short sentences, one card is enough.

The realization

$\begin{gathered}\text{English}\xrightarrow{\ \text{encoder}\ }\mathbf s\ (\text{fixed size})\\ \mathbf s\xrightarrow{\ \text{decoder}\ }\text{Hindi}\end{gathered}$

An encoder–decoder translates through one summary vector. It works, and it was a breakthrough. But everything the decoder knows about the source sentence must pass through that one fixed-size vector — a bottleneck that hurts more with every extra word. What if the decoder could look back at every English word, whenever it needs to? That question is Unit 18.

Trap

The summary is not a shorter sentence, and it is not a list of the English words. It is one note — the encoder's last card, a fixed list of numbers. Everything the decoder will ever know about the English sentence is on that card, however long the sentence was.

Pause & predict

An encoder has a note of 256 numbers. It reads a 40-word sentence whose words are vectors of 100 numbers. How many numbers go in, and how many come out as the summary?

Pause & predict

An encoder with a 512-number note reads a 20-word sentence, then a 40-word one. With the toy rate of 0.9 per step, what happens to the summary's size, and to word 1's voice in it?

If you want the algebra · 1 proof, step by step

Prove it · in a leaky summary, word 1 of n words speaks at aⁿ⁻¹

Claim. A toy linear encoder $\mathbf s_t=a\,\mathbf s_{t-1}+\mathbf e_t$ (keep a fraction $a$ of the old summary, add the new word) gives $\mathbf s_n=\sum_{k=1}^{n}a^{n-k}\mathbf e_k$ . With $a=0.9$ , word 1 is weighted $0.9^{n-1}$ : $0.9^{5}\approx0.59$ for 6 words, $0.9^{49}\approx0.0057$ for 50, and it first drops below 0.01 at $n=45$ .

Unroll as in §9's leaky sum: each new step multiplies everything earlier by

a

and adds one new word. The same shape as

c_T=\sum(\prod f)\,ig

, with every gate fixed.

0.9^{n-1}<0.01

means

n-1>\ln0.01/\ln0.9\approx43.7

, so

n-1\ge44

n\ge45

. Check:

0.9^{43}\approx0.0108

0.9^{44}\approx0.0097

. ∎ A trained encoder is cleverer than a fixed leak — gates help — but it still has only a fixed number of slots for a sentence of any length.

In one sentence: An encoder squeezes the whole source sentence into one fixed-size summary and a decoder writes the translation from it, trained with teacher forcing — and that single card is the bottleneck: 40 numbers into 8 for five words, 400 into the same 8 for fifty, with word 1 at 0.9⁴⁹ ≈ 0.0057.

Choosing the words, and grading them: greedy, beam search, BLEU

The decoder gives a probability to every possible next word. Which sentence should it actually write — and how do we score its translation without asking a human?

Imagine this

You are driving across a city. At every junction you take the road that looks fastest right now. Sometimes that road leads straight into a traffic jam, and a road that looked a little slower at first would have got you home sooner.

Keeping two or three routes in mind, instead of one, often finds the better trip.

At each step the decoder gives a probability for every possible next word. The probability of a whole sentence is the product of its word probabilities (the chain rule of probability, Unit 16). We want the sentence with the biggest product. But there are far too many sentences to try them all.

Greedy decoding takes the single most likely word at each step and never looks back. Beam search keeps the $k$ best partial sentences at every step ( $k$ is the beam width), grows each by one word, and again keeps only the best $k$ . With $k=1$ it is greedy.

A tiny example. The decoder is writing the English for "hamari train late hai". First word: "the" 0.5, "our" 0.4, "a" 0.1. Second word, after "the": "train" 0.4, "bus" 0.3, "rain" 0.3. After "our": "train" 0.9, "bus" 0.1. After "a": "train" 0.6, "bus" 0.4.

Greedy: take "the" (0.5), then its best, "train" (0.4). The sentence "the train" gets $0.5\times0.4=0.20$ .
Beam, $k=2$ : keep "the" (0.5) and "our" (0.4). Grow both: the train 0.20, the bus 0.15, the rain 0.15, our train $0.4\times0.9=0.36$ , our bus 0.04. The best is "our train", 0.36.

Greedy lost because "our" looked slightly worse at the first step, even though it led to a far surer second word. In practice we add log-probabilities instead of multiplying (products of many small numbers become too tiny for a computer): $\ln0.36\approx-1.022$ beats $\ln0.20\approx-1.609$ .

Try: Press ▶ search with width 1 (greedy): it commits to "the" and ends at 0.20. Switch to width 2 and search again: "our" survives the first cut and wins with 0.36. Width 3 keeps everything and finds the same 0.36.

Three ways to choose, side by side — greedy and beam from this section, sampling from §5.

	greedy	beam search, width k	sampling, temperature T
keeps	one best sentence-so-far	the k best sentences-so-far	one random path
work	1 path	about k paths	1 path
gets	a likely sentence, fast	a more likely sentence	a different sentence each time
weakness	locks in early choices	can be bland and generic	makes mistakes at high T
use it for	quick, short answers	translation, captions	stories, chat, ideas

Grading a translation without a human: BLEU. Checking thousands of translations by hand is slow. BLEU (Papineni and colleagues, 2002) gives a quick score by counting how many short pieces of the candidate also appear in a human's reference translation. A piece of $n$ words in a row is an $n$ -gram. Reference: "the train is running late". Candidate: "the train is late".

Words that match. Of the candidate's 4 words, how many appear in the reference? the ✓, train ✓, is ✓, late ✓: unigram precision $p_1=4/4=1$ .
Pairs that match. Of its 3 word pairs, how many appear in the reference? "the train" ✓, "train is" ✓, "is late" ✗ (the reference says "is running"): bigram precision $p_2=2/3$ .
Too short is punished. A candidate could get perfect precision by saying almost nothing. So when the candidate ( $c=4$ words) is shorter than the reference ( $r=5$ ), multiply by the brevity penalty $e^{1-r/c}=e^{1-5/4}\approx0.7788$ .
Combine. Take the geometric mean of the precisions: $\text{BLEU-2}=0.7788\cdot\sqrt{1\cdot\tfrac23}\approx0.7788\cdot0.8165\approx0.636$ .

One more rule: each word of the reference can be matched only as many times as it appears there. The candidate "the the the the" matches "the" only once, so $p_1=1/4$ , not $4/4$ . Real systems use pieces up to four words long (BLEU-4) and add up the counts over thousands of sentences before combining, but the recipe is the same.

Try: Start with "the train is late": 1, 2/3, 0.7788 → 0.636. Load word salad ("late is the train"): every word matches, but only one pair does, and the score drops to 0.450. Too short ("the train") has perfect precision and is still punished to 0.2231. the the the the shows why matches are clipped: one "the" counts, the others do not.

reference:

candidate

Why it works

A good translation shares many words, and many short runs of words, with a careful human one. Word matches check that the right content is there; pair matches check that it comes in a sensible order; the brevity penalty stops a machine from winning by saying less. None of this understands meaning — but averaged over thousands of sentences, it ranks translation systems much as people do.

The realization

$\begin{gathered}\text{score}=\sum_t\ln P(\text{word}_t\mid\dots)\\ \text{BLEU-2}=\text{BP}\cdot\sqrt{p_1\,p_2}\\ \text{BP}=e^{\min(0,\,1-r/c)}\end{gathered}$

Greedy picks the best word now; beam search keeps the $k$ best sentences-so-far and can recover when a slightly weaker first word leads somewhere much better. BLEU grades the result by the pieces it shares with a reference, with a penalty for being too short.

Trap

BLEU only counts overlapping pieces. "the train is delayed" means the same as "the train is running late", yet it scores only about 0.55, because "delayed" appears nowhere in the reference. A correct translation in different words can score low. BLEU is a quick average over many sentences, not a judge of one.

Try it, then answer

On this tree, does beam width 3 find a better sentence than beam width 2?

Pause & predict

Reference "the train is running late" (5 words). Candidate "the train" (2 words). What is its BLEU-2?

Pause & predict

Reference "the train is running late". The candidate "the the the the" has four words, and every one of them is "the", which does appear in the reference. What is its unigram precision?

If you want the algebra · 2 proofs, step by step

Prove it · adding logs picks the same winner, without the underflow

Claim. For sentences $A$ and $B$ , $\prod_tp_t^A>\prod_tp_t^B$ exactly when $\sum_t\ln p_t^A>\sum_t\ln p_t^B$ . A 200-word sentence whose words each have probability 0.01 has product $10^{-400}$ , which a computer's usual numbers round to 0, but log-score $200\ln0.01\approx-921.0$ .

\ln

turns a product into a sum,

\ln(xy)=\ln x+\ln y

, and it is increasing, so it never changes which of two positive numbers is bigger. Tree:

\ln0.36\approx-1.022>\ln0.20\approx-1.609

, the same verdict as

0.36>0.20

Ordinary 64-bit floats cannot hold positive numbers below about

10^{-308}

(

10^{-324}

with the tiniest special values), so

10^{-400}

becomes 0 and every long sentence would tie. The sum of logs stays an ordinary number. ∎ This is the same log-likelihood as Unit 14: training maximises it, and decoding searches for it.

Prove it · the pieces of BLEU, and why one zero sinks it

Claim. With clipped precisions $p_n=\dfrac{\sum_g\min(\text{count}_{\text{cand}}(g),\,\text{count}_{\text{ref}}(g))}{\sum_g\text{count}_{\text{cand}}(g)}$ over the candidate's $n$ -grams $g$ , and $\text{BP}=\min(1,e^{1-r/c})$ : $\text{BLEU-2}=\text{BP}\sqrt{p_1p_2}$ is 0 whenever $p_1$ or $p_2$ is 0, and for "the train is late" against "the train is running late" it is $0.7788\sqrt{2/3}\approx0.636$ .

Unigrams: the, train, is, late each appear once in the reference:

p_1=4/4

. Bigrams: "the train" and "train is" appear, "is late" does not:

p_2=2/3

. Lengths

c=4<r=5

, so

\text{BP}=e^{1-5/4}=e^{-0.25}\approx0.7788

. The

\min

is the clipping: "the the the the" gets

\min(4,1)/4=1/4

\sqrt{p_1p_2}=e^{\frac12(\ln p_1+\ln p_2)}

is the geometric mean: if one precision is 0 its logarithm is

-\infty

and the mean is 0. Here

\sqrt{1\cdot2/3}\approx0.8165

, and

0.7788\cdot0.8165\approx0.636

. ∎ A geometric mean rewards doing well on every piece size: word salad ("late is the train",

p_1=1

p_2=1/3

) falls to

0.7788\sqrt{1/3}\approx0.450

In one sentence: Greedy takes the best word each time and can land on 0.20, a beam of width 2 keeps two candidates and finds 0.36, and BLEU grades the result by matching pieces with a reference — "the train is late" scores 0.7788 · √(1 · ⅔) ≈ 0.636.

What to carry forward

The whole unit, one card at a time.

Order needs memory

A bag of words loses order; a window loses the past. A running note that changes the old note before adding the new word keeps both.

The recurrent cell

One card of fixed size, rewritten at every word:

\mathbf h_t=\tanh(W\mathbf h_{t-1}+U\mathbf x_t+\mathbf b)

, the same weights every time.

Unrolled twins

A deep network as long as the sentence whose layers are one cell:

h(h+d)+h

numbers for any length.

A cell that talks

Train with the true text fed in (teacher forcing); talk by feeding back your own picks.

\operatorname{softmax}(s/T)

: small

T

bold, big

T

adventurous. Beware exposure bias.

Backprop through time

Rule A and Rule B on the unrolled chain; the shared weight adds up every step's vote. Truncated BPTT cuts the blame into chunks of

k

words.

The whisper line

Blame is multiplied by the same matrix at every step, so the biggest

|\lambda|

decides:

0.5^{10}\approx0.001

1.5^{10}\approx58

First aid

\|\mathbf g\|>c

, rescale to length

c

: a cure for explosions, not for fading. Start

W

with every

|\lambda|

near 1.

LSTM

\mathbf c_t=\mathbf f\odot\mathbf c_{t-1}+\mathbf i\odot\mathbf g

: an express lane where blame is multiplied only by the forget (keep) gate. Start its shift at 1–2.

GRU

\mathbf h_t=(1-\mathbf z)\odot\mathbf h_{t-1}+\mathbf z\odot\tilde{\mathbf h}_t

: one blend dial and a reset gate. Blocks: plain 1, GRU 3, LSTM 4.

Both ways, and floors

Two-way notes

[\overrightarrow{\mathbf h};\overleftarrow{\mathbf h}]

see the whole sentence but cannot write; floors stack readers for depth.

Encoder–decoder

Squeeze the source into one summary, write the target from it. The summary is a fixed-size bottleneck.

Choose and grade

Greedy, beam search or sampling to choose; BLEU = brevity penalty × geometric mean of matching pieces to grade.

Where this goes next.

Unit 18 · Attention and Transformers. Two problems are left over from this unit. The decoder sees the source only through one summary card, and every word must wait for the word before it. Attention fixes both at once: the decoder — and then every word — can look directly at every other word, with weights from dot products (Unit 3) and softmax (Unit 14). The transformer's residual path is another express lane, and its causal mask is a one-way reader in disguise.
Unit 19 · The Maths Inside an LLM. A language model at huge scale: next-word probabilities, and how chat models pick their words — the temperature of §5, and its cousins top-k and top-p.

The realization

$\begin{gathered}\frac{\partial\mathbf h_T}{\partial\mathbf h_k}\ \text{carries}\ W^{T-k}\\ \Longrightarrow\ \text{eigenvalues decide}\end{gathered}$

A machine with memory multiplies by the same matrix at every word, so its memory and its blame live or die by that matrix's eigenvalues. Gates build an express lane where the multiplier is a number the network controls, close to 1 when it wants to remember.

In one sentence: A recurrent cell reads one word at a time with shared weights, learns to talk by guessing the next word, trains by backprop through time where powers of one matrix make memory fade or explode, is rescued by gates that open an express lane, reads both ways or in floors, and translates through a single summary — the bottleneck that attention removes next.

Practice arena — sixteen problems, solved in full

Sixteen problems, easy to hard: a forward pass by hand, one step with matrices, counting weights for every design in the unit, a read-out and its loss, temperature forwards and backwards, blame through three steps, truncated BPTT, the step at which memory is gone, eigenvalues that decide, clipping, a full LSTM step, recorded runs to diagnose, a full GRU step, notes read both ways, greedy against beam, and BLEU. Every number was checked by machine.

Three habits do most of the work. Write the notes down: keep a table of every $h_t$ (and $c_t$ ) on the way forward, because the way back needs them. Count steps, not words: from $h_k$ to $h_T$ there are $T-k$ factors. And look for the power: whenever something is multiplied by the same number again and again, it is $w^T$ , and $\ln$ turns "how many steps?" into a division.

Problem 1easyforward pass

A one-number recurrent cell has $w=0.8$ , $u=1$ , $b=0$ and $h_0=0$ . It reads $x=(1,\,-1,\,0.5)$ . Find $h_1$ , $h_2$ and $h_3$ (four decimals).

What this tests. $h_t=\tanh(wh_{t-1}+ux_t+b)$ , one step at a time. Plan. Keep old, add new, squeeze — three times.

Show the full solution

Step 1 — word 1.

h_1=\tanh(0.8\cdot0+1)=\tanh(1)\approx0.7616

Step 2 — word 2.

0.8\cdot0.7616=0.6093

; add

-1

-0.3907

h_2=\tanh(-0.3907)\approx-0.3720

Step 3 — word 3.

0.8\cdot(-0.3720)=-0.2976

; add

0.5

0.2024

h_3=\tanh(0.2024)\approx0.1997

answers at a glance: $h_1\approx0.7616$ , $h_2\approx-0.3720$ , $h_3\approx0.1997$ .

Remember

The note is a running mix: the old note, scaled by $w$ , plus the new word, then squeezed. A negative word can flip the sign of the whole note.

Problem 2easya step with matrices

A cell has a note of 2 numbers and words of 1 number: $\begin{aligned}W&=\begin{pmatrix}0.5&-0.5\\ 0&1\end{pmatrix},\quad U=\begin{pmatrix}1\\ 2\end{pmatrix},\\ \mathbf b&=\begin{pmatrix}0\\ -1\end{pmatrix},\quad \mathbf h_0=\begin{pmatrix}1\\ 0\end{pmatrix}.\end{aligned}$ (a) Find $\mathbf h_1$ for the word $x_1=0.5$ . (b) Find $\mathbf h_2$ for the word $x_2=0$ .

What this tests. The vector cell $\mathbf h_t=\tanh(W\mathbf h_{t-1}+Ux_t+\mathbf b)$ , with tanh applied entry by entry. Plan. Three pieces, add, squeeze each entry.

Show the full solution

Step 1 — (a), the three pieces.

W\mathbf h_0=(0.5,0)

Ux_1=(0.5,1)

\mathbf b=(0,-1)

. Sum:

(1,0)

Step 2 — (a), squeeze.

\mathbf h_1=(\tanh1,\tanh0)\approx(0.7616,0)

Step 3 — (b).

W\mathbf h_1=(0.5\cdot0.7616-0,\ 0)=(0.3808,0)

;

Ux_2=(0,0)

; add

\mathbf b

(0.3808,-1)

Step 4 — (b), squeeze.

\mathbf h_2\approx(\tanh0.3808,\ \tanh(-1))\approx(0.3634,\ -0.7616)

answers at a glance: (a) $\mathbf h_1\approx(0.7616,\ 0)$ . (b) $\mathbf h_2\approx(0.3634,\ -0.7616)$ .

Remember

The shift $\mathbf b$ acts at every step, even when the word is 0. Here it pushes the second entry to $\tanh(-1)$ on its own.

Problem 3mediumcounting weights

A word tagger uses notes of $h=64$ numbers and words of $d=32$ numbers. (a) How many numbers does one plain cell learn? (b) A two-way plain reader? (c) Two floors of one-way plain cells? (d) A one-way LSTM, and a one-way GRU? (e) A two-way LSTM with a read-out over 12 tags on top of each word's two-way note (read-out $V$ and $\mathbf c$ included)?

What this tests. The block count $h(h+d)+h$ and what changes when you add directions, floors and gates. Plan. Write the general block once; then ask, for each design, how many blocks and what each block reads.

Show the full solution

Step 1 — (a), one block.

P=h(h+d)+h=64\cdot96+64=6144+64=6208

Step 2 — (b). Two separate readers of the same shape:

2P=12\,416

Step 3 — (c). Floor 2 reads 64-number notes, not 32-number words:

64\cdot(64+64)+64=8192+64=8256

. Total

6208+8256=14\,464

Step 4 — (d). An LSTM has four blocks:

4P=24\,832

. A GRU has three:

3P=18\,624

Step 5 — (e). Two LSTM readers:

2\cdot24\,832=49\,664

. Each word's note has

2h=128

numbers, so

V

12\times128

and

\mathbf c

has 12:

1536+12=1548

. Total

49\,664+1548=51\,212

answers at a glance: (a) 6208. (b) 12 416. (c) 14 464. (d) LSTM 24 832, GRU 18 624. (e) 51 212.

Remember

Every design is built from the same block. Count the blocks, and check what each block reads: words ( $d$ numbers), notes ( $h$ ) or two-way notes ( $2h$ ).

Problem 4easyread-out and loss

After reading "the chai is", a language model's read-out gives the scores hot 3, sweet 1, ready 0, cold −1. (a) Turn them into probabilities (four decimals). (b) The true next word is "sweet". What is this step's loss? What would it have been if the true word were "hot"? (c) What blame does the read-out send back (prediction minus truth)?

What this tests. Softmax, the surprise $-\ln p$ of the true word, and Unit 15's "prediction − truth". Plan. Exponentiate, add, divide; then read off the true word's share.

Show the full solution

Step 1 — exponentiate.

e^3\approx20.0855

e^1\approx2.7183

e^0=1

e^{-1}\approx0.3679

; total

\approx24.1717

Step 2 — (a), divide. hot

\approx0.8310

, sweet

\approx0.1125

, ready

\approx0.0414

, cold

\approx0.0152

Step 3 — (b). Loss

=-\ln0.1125\approx2.185

. Had the true word been "hot":

-\ln0.8310\approx0.185

Step 4 — (c). Subtract the one-hot truth (1 at "sweet"):

(0.8310,\ 0.1125-1,\ 0.0414,\ 0.0152)=(0.8310,\ -0.8875,\ 0.0414,\ 0.0152)

answers at a glance: (a) 0.8310, 0.1125, 0.0414, 0.0152. (b) ≈ 2.185 (and ≈ 0.185 for "hot"). (c) (0.8310, −0.8875, 0.0414, 0.0152).

Remember

A confident wrong guess is expensive: "sweet" had 0.11 and costs 2.19. The blame pushes "hot" down by 0.83 and pulls "sweet" up by 0.89.

Problem 5mediumtemperature

A model has two candidate next words with scores 2 and 0. (a) Find the first word's probability at $T=1$ , $T=0.5$ and $T=2$ . (b) Another model gives two words the probabilities 0.8 and 0.2 at $T=1$ . At what temperature would it give them 0.6 and 0.4? (c) What does it give them as $T\to0$ ?

What this tests. $p_i\propto e^{s_i/T}$ , and the ratio rule $p_i/p_j=e^{(s_i-s_j)/T}$ used backwards. Plan. For two words, softmax is a sigmoid of the gap; for (b) find the gap first, then the $T$ that gives the new ratio.

Show the full solution

Step 1 — (a). With two words,

p_1=\dfrac{e^{2/T}}{e^{2/T}+e^{0}}=\sigma(2/T)

T=1

\sigma(2)\approx0.8808

T=0.5

\sigma(4)\approx0.9820

T=2

\sigma(1)\approx0.7311

Step 2 — (b), the gap. At

T=1

the ratio is

0.8/0.2=4=e^{\Delta}

, so the score gap is

\Delta=\ln4\approx1.3863

Step 3 — (b), the new temperature. We need

e^{\Delta/T}=0.6/0.4=1.5

, so

\Delta/T=\ln1.5\approx0.4055

and

T=1.3863/0.4055\approx3.419

Step 4 — (c). As

T\to0

the gap

\Delta/T

grows without limit: the shares go to

1

and

0

— greedy.

answers at a glance: (a) 0.8808, 0.9820, 0.7311. (b) $T=\ln4/\ln1.5\approx3.419$ . (c) 1 and 0.

Remember

Temperature only rescales the score gaps. Two shares always stand in the ratio $e^{\text{gap}/T}$ , so you can run the rule forwards or backwards.

Problem 6mediumblame through time

A one-number cell has $w=0.8$ , $u=1$ , $b=0$ , $h_0=0$ and reads $x=(1,0,0)$ . (a) Find $h_1,h_2,h_3$ . (b) Find $\partial h_3/\partial h_2$ , $\partial h_2/\partial h_1$ and $\partial h_3/\partial h_1$ . (c) Find $\partial h_3/\partial x_1$ .

What this tests. The BPTT product $\prod w(1-h_t^2)$ . Plan. Forward first and store every note; then one factor per step.

Show the full solution

Step 1 — (a).

h_1=\tanh1\approx0.7616

h_2=\tanh(0.6093)\approx0.5436

h_3=\tanh(0.4349)\approx0.4094

Step 2 — (b), one factor per step.

\begin{aligned}\frac{\partial h_3}{\partial h_2}&=0.8\,(1-0.4094^2)\\ &\approx0.6659\\ \frac{\partial h_2}{\partial h_1}&=0.8\,(1-0.5436^2)\\ &\approx0.5636\end{aligned}

Step 3 — (b), multiply.

\partial h_3/\partial h_1\approx0.6659\times0.5636\approx0.3753

Step 4 — (c).

h_1=\tanh(ux_1)

, so

\partial h_1/\partial x_1=u\,(1-h_1^2)\approx0.4200

. Then

\partial h_3/\partial x_1\approx0.3753\times0.4200\approx0.1576

answers at a glance: (a) 0.7616, 0.5436, 0.4094. (b) 0.6659, 0.5636, 0.3753. (c) $\approx0.1576$ .

Remember

With $w=0.8$ instead of 0.5, each factor is bigger, and the first word still reaches $h_3$ at more than a third of its size. The factors decide everything.

Problem 7mediumtruncated BPTT

A 2 500-character text is trained with truncated BPTT in chunks of $k=50$ characters (1–50, 51–100, …). (a) Give the general number of chunks for a text of $N$ characters, then apply it. (b) The blame from character 130 walks back to which character? (c) Can training learn a link between characters 30 and 70? Between 60 and 90? (d) Full BPTT would walk the blame from the last character back to the first. How many steps is that, against the longest walk in the truncated run?

What this tests. What truncation cuts (the blame) and what it keeps (the forward note). Plan. Find each character's chunk with $\lceil t/k\rceil$ ; blame never leaves its chunk.

Show the full solution

Step 1 — (a).

\lceil N/k\rceil

chunks:

\lceil2500/50\rceil=50

Step 2 — (b). Character 130 is in chunk

\lceil130/50\rceil=3

, which holds 101–150. Its blame walks back to character 101 (29 steps).

Step 3 — (c). 30 is in chunk 1 and 70 in chunk 2: no blame crosses the cut after 50, so that link cannot be learned. 60 and 90 are both in chunk 2 (51–100): yes.

Step 4 — (d). Full BPTT: from character 2 500 back to character 1 is 2 499 steps. Truncated: at most

k-1=49

steps, inside one chunk.

answers at a glance: (a) $\lceil N/k\rceil$ , here 50. (b) character 101. (c) 30 and 70: no; 60 and 90: yes. (d) 2 499 steps against at most 49.

Remember

Truncation makes training cheap and limits what can be learned to links shorter than $k$ — but the note still carries every earlier character forward.

Problem 8mediumfade or explode

In the loop $h_t=w\,h_{t-1}$ , the blame after $T$ steps is $w^T$ . (a) Write a formula for the first $T$ at which $w^T$ drops below $\varepsilon$ (for $0<w<1$ ). (b) Apply it for $w=0.8$ , $\varepsilon=10^{-3}$ . (c) For $w=1.2$ , find the first $T$ at which $w^T$ exceeds 1000.

What this tests. Turning "how many steps?" into a division with logarithms. Plan. Take $\ln$ of both sides; mind that $\ln w<0$ flips the inequality.

Show the full solution

Step 1 — (a).

w^T<\varepsilon\iff T\ln w<\ln\varepsilon\iff T>\dfrac{\ln\varepsilon}{\ln w}

(dividing by the negative

\ln w

flips the sign). So

T=\big\lfloor\ln\varepsilon/\ln w\big\rfloor+1

Step 2 — (b).

\ln10^{-3}/\ln0.8\approx-6.9078/-0.2231\approx30.96

, so

T=31

. Check:

0.8^{30}\approx0.00124

0.8^{31}\approx0.00099

Step 3 — (c).

1.2^T>1000\iff T>\ln1000/\ln1.2\approx6.9078/0.1823\approx37.89

, so

T=38

. Check:

1.2^{37}\approx850.6

1.2^{38}\approx1020.7

answers at a glance: (a) $T=\lfloor\ln\varepsilon/\ln w\rfloor+1$ . (b) 31. (c) 38.

Remember

Even gentle numbers are ruthless over a few dozen steps: 0.8 loses a factor of 1000 in 31 steps, and 1.2 gains one in 38.

Problem 9mediumeigenvalues decide

Two recurrent matrices: $A=\begin{pmatrix}0.6&0.2\\ 0.2&0.6\end{pmatrix}$ and $B=\begin{pmatrix}1.1&0.3\\ 0.3&1.1\end{pmatrix}$ . (a) Find the eigenvalues and eigenvectors of each and say whether memory fades or explodes. (b) Find $A^{10}\mathbf v$ and $B^{10}\mathbf v$ for $\mathbf v=(1,0)$ .

What this tests. Splitting $\mathbf v$ along eigen-directions (Unit 4) and powering each part. Plan. Both matrices have the form $\begin{pmatrix}p&q\\ q&p\end{pmatrix}$ : eigenvectors $(1,1)$ , $(1,-1)$ , eigenvalues $p\pm q$ .

Show the full solution

Step 1 — (a).

A(1,1)=(0.8,0.8)

A(1,-1)=(0.4,-0.4)

: eigenvalues 0.8 and 0.4, both below 1 — fades.

B(1,1)=(1.4,1.4)

B(1,-1)=(0.8,-0.8)

: eigenvalues 1.4 and 0.8;

|\lambda|_{\max}=1.4>1

— explodes.

Step 2 — split v.

(1,0)=\tfrac12(1,1)+\tfrac12(1,-1)

Step 3 — (b), A.

A^{10}\mathbf v=\tfrac12\,0.8^{10}(1,1)+\tfrac12\,0.4^{10}(1,-1)

. With

\tfrac12\,0.8^{10}\approx0.053687

and

\tfrac12\,0.4^{10}\approx0.0000524

\approx(0.05374,\ 0.05363)

Step 4 — (b), B.

B^{10}\mathbf v=\tfrac12\,1.4^{10}(1,1)+\tfrac12\,0.8^{10}(1,-1)\approx14.4627(1,1)+0.0537(1,-1)\approx(14.516,\ 14.409)

answers at a glance: (a) $A$ : 0.8 on $(1,1)$ , 0.4 on $(1,-1)$ — fades; $B$ : 1.4 on $(1,1)$ , 0.8 on $(1,-1)$ — explodes. (b) $A^{10}\mathbf v\approx(0.0537,\ 0.0536)$ , $B^{10}\mathbf v\approx(14.52,\ 14.41)$ .

Remember

After many steps the note points along the eigenvector with the biggest $|\lambda|$ , and its size goes like $|\lambda|_{\max}^t$ .

Problem 10easyclipping

The limit is $c=4$ and the step size $\eta=0.1$ . (a) Clip $\mathbf g=(6,-8,0)$ and give the weight change $-\eta\mathbf g'$ . (b) Clip $\mathbf g=(1,2,2)$ .

What this tests. $\mathbf g'=\min(1,c/\|\mathbf g\|)\,\mathbf g$ . Plan. Length first; scale only if it is too long.

Show the full solution

Step 1 — (a), length.

\|\mathbf g\|=\sqrt{36+64+0}=10>4

Step 2 — (a), scale. Multiply by

4/10=0.4

\mathbf g'=(2.4,-3.2,0)

, of length 4.

Step 3 — (a), step.

-0.1\,\mathbf g'=(-0.24,\ 0.32,\ 0)

Step 4 — (b).

\|\mathbf g\|=\sqrt{1+4+4}=3\le4

, so nothing changes:

\mathbf g'=(1,2,2)

answers at a glance: (a) $\mathbf g'=(2.4,-3.2,0)$ , change $(-0.24,0.32,0)$ . (b) $(1,2,2)$ , unchanged.

Remember

Clipping scales the whole vector by one number. The direction — the ratio 6 : −8 — is kept.

Problem 11mediumone LSTM step

A one-number LSTM has $c_{t-1}=2$ . Before their squeezes, the gates and the candidate come out as: forget 2, input 0, output 1, candidate 1. Find $f,i,o,g$ , then $c_t$ and $h_t$ .

What this tests. Gates are sigmoids, the candidate is a tanh; $c_t=fc_{t-1}+ig$ , $h_t=o\tanh c_t$ . Plan. Four squeezes, then the two formulas.

Show the full solution

Step 1 — the gates.

f=\sigma(2)\approx0.8808

i=\sigma(0)=0.5

o=\sigma(1)\approx0.7311

Step 2 — the candidate.

g=\tanh(1)\approx0.7616

Step 3 — the cell state.

c_t=0.8808\cdot2+0.5\cdot0.7616\approx1.7616+0.3808=2.1424

Step 4 — the note.

\tanh(2.1424)\approx0.9728

, so

h_t\approx0.7311\cdot0.9728\approx0.7112

answers at a glance: $f\approx0.8808$ , $i=0.5$ , $o\approx0.7311$ , $g\approx0.7616$ ; $c_t\approx2.1424$ , $h_t\approx0.7112$ .

Remember

The cell state may grow past 1 — it is never squeezed on the lane. Only the note $h_t$ is squeezed, when it is read out through the window.

Problem 12harddiagnose the runs

Three recurrent models were trained on the same task. For each, the size of the blame reaching the note $k$ steps back from the end was recorded:

run	k = 0	1	2	3	4	5
A	1	0.42	0.176	0.0741	0.0311	0.0131
B	1	2.1	4.41	9.26	19.4	40.8
C	1	0.97	0.941	0.913	0.885	0.859

(a) Find each run's factor per step. (b) Predict the blame 20 steps back for each. (c) One run is an LSTM, one a plain cell with small weights, one a plain cell with big weights. Which is which, and which fix does each plain cell need? (d) If run B's 40.8 were the whole gradient's length, by what factor would clipping at

c=5

scale it? (e) Run C's lane factor is

f

. What is the smallest

f

that would still deliver at least half the blame from 100 steps back?

What this tests. Reading a geometric sequence, and matching it to fade, explode or express lane. Plan. Divide neighbours; then raise the factor to the 20th power.

Show the full solution

Step 1 — (a). Neighbour ratios are steady: A ≈ 0.42 (

0.0131^{1/5}\approx0.42

), B ≈ 2.1 (

40.8^{1/5}\approx2.1

), C ≈ 0.97 (

0.859^{1/5}\approx0.97

Step 2 — (b). A:

0.42^{20}\approx2.9\times10^{-8}

. B:

2.1^{20}\approx2.8\times10^{6}

. C:

0.97^{20}\approx0.544

Step 3 — (c). C holds the blame steady at 0.97 per step: the LSTM's express lane with

f\approx0.97

. A fades (

|\lambda|<1

): the plain cell with small weights — it needs gates (an LSTM or GRU). B explodes (

|\lambda|>1

): the plain cell with big weights — it needs clipping first aid (and gates would help it too).

Step 4 — (d).

40.8>5

, so scale by

5/40.8\approx0.1225

Step 5 — (e). We need

f^{100}\ge0.5

, so

f\ge0.5^{1/100}=e^{\ln0.5/100}=e^{-0.006931}\approx0.99309

. Run C's 0.97 is far below that:

0.97^{100}\approx0.048

answers at a glance: (a) 0.42, 2.1, 0.97. (b) $\approx2.9\times10^{-8}$ , $\approx2.8\times10^{6}$ , $\approx0.544$ . (c) A = plain, small weights → gates; B = plain, big weights → clipping; C = LSTM. (d) $\approx0.1225$ . (e) $f\ge0.99309$ .

Remember

Blame through time is a geometric sequence. One ratio tells you everything: below 1 it fades, above 1 it explodes, near 1 you have an express lane.

Problem 13mediumone GRU step

A one-number GRU has old note $h_{t-1}=-0.5$ and input $x=2$ . Its update score is 1 and its reset score is −1. The draft uses weight 1 on the word and 2 on the consulted old note: $\tilde h_t=\tanh(1\cdot x+2\cdot r\,h_{t-1})$ . (a) Find $z$ , $r$ , $\tilde h_t$ and $h_t$ (four decimals). (b) Redo $\tilde h_t$ and $h_t$ with the reset gate wide open, $r=1$ . What did the reset gate change?

What this tests. The two GRU gates in the right places: $r$ inside the draft, $z$ in the blend. Plan. Squeeze the scores, draft, then blend $(1-z)h_{t-1}+z\tilde h_t$ .

Show the full solution

Step 1 — the gates.

z=\sigma(1)\approx0.7311

r=\sigma(-1)\approx0.2689

Step 2 — the draft.

r\,h_{t-1}\approx0.2689\cdot(-0.5)=-0.1345

; times 2:

-0.2689

; add the word:

2-0.2689=1.7311

\tilde h_t=\tanh(1.7311)\approx0.9392

Step 3 — the blend.

h_t=0.2689\cdot(-0.5)+0.7311\cdot0.9392\approx-0.1345+0.6866=0.5521

Step 4 — (b). With

r=1

\tilde h_t=\tanh(2+2\cdot(-0.5))=\tanh(1)\approx0.7616

and

h_t\approx-0.1345+0.5568=0.4223

. The mostly shut reset gate hid most of the negative old note from the draft, so the draft came out higher (0.9392 against 0.7616).

answers at a glance: (a) $z\approx0.7311$ , $r\approx0.2689$ , $\tilde h_t\approx0.9392$ , $h_t\approx0.5521$ . (b) $\tilde h_t\approx0.7616$ , $h_t\approx0.4223$ .

Remember

The reset gate decides how much of the old note the draft may consult; the update gate decides how much of the draft enters the note. The old note still passes through $(1-z)$ either way.

Problem 14mediumreading both ways

Two one-number cells, both with $w=0.5$ , $u=1$ , $b=0$ , read $x=(1,\ 0,\ 0.5)$ : the forward cell left to right, the backward cell right to left. (a) Find the three forward notes. (b) Find the three backward notes. (c) Write word 2's two-way note. (d) If $x_3$ changes, which half of word 1's note changes?

What this tests. Running the same cell in two directions, and which half carries which side of the sentence. Plan. Forward: start at word 1. Backward: start at word 3 and walk left.

Show the full solution

Step 1 — (a).

\overrightarrow h_1=\tanh1\approx0.7616

;

\overrightarrow h_2=\tanh(0.3808)\approx0.3634

;

\overrightarrow h_3=\tanh(0.1817+0.5)=\tanh(0.6817)\approx0.5926

Step 2 — (b).

\overleftarrow h_3=\tanh0.5\approx0.4621

;

\overleftarrow h_2=\tanh(0.2311)\approx0.2270

;

\overleftarrow h_1=\tanh(0.1135+1)=\tanh(1.1135)\approx0.8053

Step 3 — (c). Word 2's note is

(\overrightarrow h_2,\overleftarrow h_2)\approx(0.3634,\ 0.2270)

Step 4 — (d).

\overrightarrow h_1

depends on

x_1

only, so it cannot change.

\overleftarrow h_1

has read

x_3

x_2

and

x_1

: only the backward half changes.

answers at a glance: (a) 0.7616, 0.3634, 0.5926. (b) 0.4621, 0.2270, 0.8053 (words 3, 2, 1: $\overleftarrow h=(0.8053,\ 0.2270,\ 0.4621)$ in word order). (c) (0.3634, 0.2270). (d) The backward half.

Remember

Same cell, same weights, opposite direction. The forward half of a note knows the past; the backward half knows the future.

Problem 15mediumgreedy vs beam

A decoder's choices. First word: "the" 0.6, "my" 0.4. After "the": "tea" 0.3, "chai" 0.3, "cup" 0.4. After "my": "chai" 0.8, "tea" 0.2. (a) What does greedy decoding output, with what probability? (b) What does beam search with width 2 output? (c) Give both log-scores.

What this tests. Sentence probability as a product; keeping the $k$ best. Plan. List every two-word candidate the beam sees.

Show the full solution

Step 1 — (a). Greedy takes "the" (0.6), then its best, "cup" (0.4):

0.6\times0.4=0.24

Step 2 — (b), grow both beams. the tea 0.18, the chai 0.18, the cup 0.24, my chai

0.4\times0.8=0.32

, my tea 0.08.

Step 3 — (b), keep the best 2. "my chai" 0.32 and "the cup" 0.24. The winner is "my chai".

Step 4 — (c).

\ln0.24\approx-1.427

;

\ln0.32\approx-1.139

answers at a glance: (a) "the cup", 0.24. (b) "my chai", 0.32. (c) −1.427 and −1.139.

Remember

A first word that looks a little worse can lead to a much surer second word. Greedy never finds out; a beam of width 2 does.

Problem 16hardBLEU-2

Reference: "we drink chai every morning". (a) Candidate A: "each morning we drink chai". Find $p_1$ , $p_2$ , the brevity penalty and BLEU-2. (b) Candidate B: "we drink chai every evening". Find the same. (c) Candidate C: "we drink chai". Find the same. (d) Which candidate keeps the meaning, which scores highest, and what does that say about BLEU?

What this tests. Clipped $n$ -gram precision, the brevity penalty $\min(1,e^{1-r/c})$ and the geometric mean. Plan. List the candidate's words and pairs, tick those in the reference, count lengths.

Show the full solution

Step 1 — the reference's pieces. Words: we, drink, chai, every, morning (

r=5

). Pairs: "we drink", "drink chai", "chai every", "every morning".

Step 2 — (a). Words: each ✗ morning ✓ we ✓ drink ✓ chai ✓:

p_1=4/5

. Pairs: "each morning" ✗, "morning we" ✗, "we drink" ✓, "drink chai" ✓:

p_2=2/4

c=r=5

, so BP

=1

. BLEU-2

=\sqrt{0.8\cdot0.5}=\sqrt{0.4}\approx0.6325

Step 3 — (b). Words: we ✓ drink ✓ chai ✓ every ✓ evening ✗:

p_1=4/5

. Pairs: "we drink" ✓, "drink chai" ✓, "chai every" ✓, "every evening" ✗:

p_2=3/4

. BP

=1

. BLEU-2

=\sqrt{0.8\cdot0.75}=\sqrt{0.6}\approx0.7746

Step 4 — (c). Every word and both pairs match:

p_1=p_2=1

. But

c=3<r=5

: BP

=e^{1-5/3}=e^{-2/3}\approx0.5134

. BLEU-2

=0.5134\cdot\sqrt{1\cdot1}\approx0.5134

Step 5 — (d). Only A keeps the meaning ("each morning" is "every morning"). B gets a fact wrong (evening), and C drops when they drink it. Yet B scores highest, 0.7746, against A's 0.6325 and C's 0.5134: B shares the most pieces, and one wrong word costs it only one word and one pair. BLEU counts shared pieces, not meaning.

answers at a glance: (a) $p_1=4/5$ , $p_2=1/2$ , BP = 1, BLEU-2 ≈ 0.6325. (b) $p_1=4/5$ , $p_2=3/4$ , BP = 1, BLEU-2 ≈ 0.7746. (c) $p_1=p_2=1$ , BP ≈ 0.5134, BLEU-2 ≈ 0.5134. (d) A keeps the meaning, but B — which gets a fact wrong — scores highest: BLEU rewards overlap, so it is trusted only as an average over many sentences.

Remember

BLEU-2 = BP · √(p₁ p₂). Clip the counts, punish short candidates, and never read one sentence's BLEU as a verdict on its meaning.

Next up

Unit 18 · Attention and Transformers →

The decoder of this unit sees the English sentence only through one small summary card, and every word waits for the word before it. Attention throws both limits away: every word asks every other word a question, weighs the answers with softmax, and blends them — all at once. Queries, keys and values, why we divide by √d, many heads, positions as rotations, and a residual express lane that you will now recognise.

← Unit 16 · Words as Vectors · All units