Unit 19 · The Maths Inside an LLM — The Math Behind the Machine

The only thing an LLM does

Imagine this

You type on your phone, "I will reach home in", and the keyboard offers "10 minutes". It has not read your mind. It has seen millions of messages, and after "reach home in" a number of minutes usually comes.

Now imagine that keyboard grown enormous. It has read a large part of the internet, and it does not stop at one suggestion. It takes its own suggestion, adds it to the message, and suggests again. And again. That is a chatbot.

The question. When a chatbot writes a whole paragraph for you, what is it actually doing, step by step?

First, a word about words. The model does not read letters or whole words. It reads tokens: common words and pieces of rarer words, cut by the byte-pair method of Unit 16. GPT-2's list of tokens, its vocabulary, has 50 257 of them. In this unit a token is nearly always a whole word, to keep the pictures simple.

Here is everything a large language model (LLM) does. It is one loop of four moves.

Read. Take every token written so far. Your question, the earlier chat and the part of the reply already written are all one long list of tokens.
Score every token in the vocabulary. The transformer of Unit 18 reads the list and gives each of the 50 257 tokens a score, called a logit: "how well would you fit next?"
Turn the scores into probabilities with softmax (Unit 14). They are positive and add up to 1.
Pick one token, add it to the list, and go back to move 1. Stop when the model picks a special "end" token.

By hand. The text so far is "I drink". To keep it tiny, say the vocabulary has only five tokens: chai, coffee, water, cricket, the. The model gives them the scores $(3,\ 2,\ 1,\ -1,\ 0)$ . (We made these scores up by hand; a real model computes them.)

$\begin{aligned}e^{3}&\approx20.086,\quad e^{2}\approx7.389\\ e^{1}&\approx2.718,\quad e^{-1}\approx0.368\\ e^{0}&=1\\ \text{total}&\approx31.561\end{aligned}$

Divide each by the total:

$\begin{aligned}&(0.636,\ 0.234,\ 0.086,\\ &\ \ 0.012,\ 0.032).\end{aligned}$

Chai gets almost two thirds. Say we pick chai. The text is now "I drink chai", and the whole model runs again, from the start, on these three tokens, to choose the fourth. One run of the model gives one new token. A reply of 300 tokens is 300 runs.

The chain rule. How likely is the whole sentence? Multiply the guesses, each one given everything before it. This is the chain of guesses from Unit 16:

$\begin{aligned}&P(\text{chai, every}\mid\text{I drink})\\ &=P(\text{chai}\mid\text{I drink})\\ &\quad\times P(\text{every}\mid\text{I drink chai}).\end{aligned}$

The n-gram model of Unit 16 kept only the last one or two words of the "given" part. An LLM keeps all of them: attention (Unit 18) looks back at every earlier token, and the causal mask (Unit 18) makes sure each guess sees only the past.

Try: The picture opens after the first run: chai got 0.636 and coffee 0.234 — the numbers above. Press ▶ write and count the runs: four runs of the model write four tokens, "chai every morning .". Watch the arrows: at every run the whole text goes back into the model, including the words it just wrote itself.

Why does this work?

The game is simple, but playing it well is not. To guess what comes after "The capital of Japan is", the model must know geography. After "2 + 3 =" it needs arithmetic; after "def add(a, b):" it needs to know how Python code is written. Making the next-token guess good on trillions of tokens forces the weights to soak up grammar, facts and styles — because they all help the guess. And one step at a time is enough to write long, sensible text, because every step reads everything written before, including the words the model itself has just chosen.

Three next-word guessers, side by side. You have now met three machines that play the same game.

	n-gram (Unit 16)	recurrent net (Unit 17)	transformer LLM (Unit 18, this unit)
what it reads	the last one or two words	one running note of everything so far	every earlier token, directly
the start of a long text	forgotten after a few words	fades as the note is rewritten	kept exactly, up to the context length
how it learns	counting	one word after another	every position at once (the mask)
work for one new token	one look-up in a table	one step of the cell	attention over all earlier tokens (§6 makes it cheaper)

Rule of thumb. All three compute the same thing: a probability for the next token given the ones before. They differ only in how much of the past they can use. The transformer uses all of it, which is why it won.

Trap

The model does not plan the sentence and then type it out. There is no hidden finished answer waiting inside. Each token is chosen given only what is already written. When a reply starts well and then wanders, this is why: every new token becomes part of the "given" for all the tokens after it.

The realization

$\begin{gathered}P(w_1,\dots,w_n)\\ =\prod_{t=1}^{n}P\big(w_t\mid w_1,\dots,w_{t-1}\big)\end{gathered}$

A language model is one function: "given the text so far, a probability for every possible next token". Run it, pick a token, append it, run it again. Multiply the guesses along the way and you get the probability of the whole text.

Pause & predict

In the worked example you add 5 to every score: $(8,\ 7,\ 6,\ 4,\ 5)$ . What does the model do now?

Pause & predict

You ask a question 40 tokens long. The model writes a reply of 60 new tokens (the last one is the "end" token). How many times did the model run?

If you want the algebra · 1 proof, step by step

Prove it · the chain rule of probability

Claim. For any tokens $w_1,\dots,w_n$ , $P(w_1,\dots,w_n)=\prod_{t=1}^{n}P(w_t\mid w_1,\dots,w_{t-1})$ (for $t=1$ the "given" part is empty: $P(w_1)$ ).

Conditional probability is defined by

P(B\mid A)=P(A,B)/P(A)

. Multiply both sides by

P(A)

\begin{gathered}P(A,B)\\ =P(A)\,P(B\mid A).\end{gathered}

"A and then B" = "A" times "B, now that A has happened" (Unit 14).

Take

A=(w_1,\dots,w_{n-1})

— write it

w_{\lt n}

— and

B=w_n

\begin{aligned}&P(w_1,\dots,w_n)\\ &=P(w_1,\dots,w_{n-1})\\ &\quad\times P(w_n\mid w_{\lt n}).\end{aligned}

The last guess peels off, given everything before it.

Do the same to

P(w_1,\dots,w_{n-1})

, and again, until only

P(w_1)

is left. The factors that peel off are exactly

P(w_t\mid w_1,\dots,w_{t-1})

for

t=n,n-1,\dots,1

. ∎ No assumption was made about language. The n-gram's shortcut — "only the last two words matter" — is an extra assumption; the LLM does not make it.

The road ahead. The unit has four acts.

A machine that guesses the next word (§1–§3): the loop, how a whole sentence is scored, and where the model's millions of numbers live.
How it writes (§4–§7): picking the word — greedy, temperature, top-k and top-p — then the cache that makes writing fast and the clocks that make long texts possible.
Training at scale (§8): how loss falls as models and data grow, and how to split a fixed budget.
From guesser to assistant (§9–§13): copying good answers, fine-tuning with a thin low-rank patch, a reward learned from people's choices, the leash that keeps the model sensible, and a shortcut called DPO.

In one sentence: An LLM is your phone's autocomplete grown huge — read everything so far, give every token a probability, pick one, append it, repeat — and the chain rule multiplies those guesses into the probability of the whole text.

Scoring a whole sentence: likelihood, loss and perplexity

Imagine this

Think of the model as a gambler who starts with ₹1. Before each word it spreads its money over all the words that might come next. When the real word arrives, it keeps only the money it put on that word — and bets all of that again on the next word.

A good gambler ends the sentence with a decent amount left. But one word where it put almost nothing, and it is broke — however well it bet on every other word.

The question. How do we give a language model a score on a whole sentence — and why does every paper about LLMs talk in logarithms?

By hand. The sentence is "we drink hot chai". After "we", the model gave the true next token "drink" a probability of 0.5. After "we drink" it gave "hot" 0.25. After "we drink hot" it gave "chai" 0.8. Five small moves.

The gambler's money is the probability of the sentence (the chain rule of §1): $0.5\times0.25\times0.8=0.1$ . Ten paise left of the rupee.
Take logs, and the product becomes a sum: $\ln0.5+\ln0.25+\ln0.8\approx-0.6931-1.3863-0.2231=-2.3026$ , which is exactly $\ln0.1$ . This is the log-likelihood.
Average the surprise. The surprise of one token is $-\ln p$ (Unit 14). The average is $2.3026/3\approx0.7675$ nats per token. This is the cross-entropy loss (Unit 14), the one number an LLM is trained to make small.
Turn it back into a die. $e^{0.7675}\approx2.1544$ . This is the perplexity, the die of Unit 16: on average the model was as unsure as someone choosing among about 2.15 equally likely words. The same number is $(1/0.1)^{1/3}$ : one over the probability, shared out evenly over the three tokens.
In bits, divide by $\ln2$ : $0.7675/0.6931\approx1.1073$ bits per token.

Why logarithms? Three reasons. First, the product gets too small for a computer. A text of 1 000 tokens, each with probability 0.5, has probability $0.5^{1000}\approx9.33\times10^{-302}$ . At 2 000 tokens a computer's ordinary numbers run out and it says 0. The log is just $-693.1$ , no trouble at all. Second, a sum is easy to average, so texts of different lengths can be compared per token. Third, the slope of a sum is the sum of the slopes: every token's surprise can be pushed down on its own, and backprop (Unit 15) does it for all of them in one pass.

This is all of training. Take a huge pile of text, and make the average surprise small at every position. Unit 18's mask (§10) lets one pass grade every position of a text at once.

Try: At the start you see the worked example: money 0.1, loss 0.7675, perplexity 2.1544. Drag "hot" right down, or press one bad guess (hot = 0.01): its red bar shoots to 4.61, the money falls to 0.004 and the perplexity jumps to 6.30 — one bad guess ruins the average. Then press + token a few times: the money keeps falling as the sentence grows, but the loss per token stays a fair score. Press no idea: a model that spreads its money evenly over 6 words gets perplexity exactly 6 — a fair die with 6 faces.

Why does this work?

The surprise $-\ln p$ punishes confidence in the wrong place much more than it rewards confidence in the right place. At $p=0.9$ the surprise is only 0.105; at $p=0.01$ it is 4.6; at $p=0$ it is infinite. So a model that wants a small average loss must never be sure of the wrong thing. It learns to spread its money sensibly: most on the likely words, a little on everything that could happen. That is exactly what we want from a model of language.

Five ways to report the same thing. Papers and dashboards use all of these. They carry the same information.

number	how	for our sentence	better is	used for
probability	multiply the guesses	0.1	bigger (at most 1)	the idea; shrinks with every extra word
log-likelihood	add the logs	−2.3026	closer to 0	comparing answers of the same length
loss (nats per token)	average of $-\ln p$	0.7675	smaller (at least 0)	training — the number gradient descent pushes down
bits per token	loss ÷ ln 2	1.1073	smaller	compression: bits needed per token
perplexity	$e^{\text{loss}}$	2.1544	smaller (at least 1)	the feel: "a die with this many faces"

Rule of thumb. Train on the loss, feel it as perplexity. A model that knows nothing gives every token $1/V$ : its perplexity is exactly $V$ , 50 257 for GPT-2's vocabulary. Every good model lives far below that.

Trap

Perplexity is per token, so it depends on how the text was cut into tokens. Cut the same sentence into more, smaller pieces, and each piece is easier to guess: the perplexity per piece falls, though the model is no better. Compare perplexities only on the same text with the same tokenizer.

The realization

$\begin{gathered}\mathcal L=-\frac1n\sum_{t=1}^{n}\ln P\big(w_t\mid w_{\lt t}\big)\\ \text{perplexity}=e^{\mathcal L}\end{gathered}$

The loss is the average surprise of the true tokens: the log turns the gambler's product into a sum, and the average makes texts of any length comparable. Perplexity turns that average back into "how many equally likely choices".

Pause & predict

The model gets better at "hot": it now gives it 0.5 instead of 0.25. "drink" and "chai" stay at 0.5 and 0.8. What is the new perplexity?

Pause & predict

A model is graded on 1 000 tokens. It is excellent on 999 of them, but on one token it gave the true word a probability of exactly 0. What is its average loss?

Quick check

A new model with GPT-2's vocabulary of 50 257 tokens has a loss of $\ln 50\,257\approx10.82$ nats per token. What does that tell you?

If you want the algebra · 2 proofs, step by step

Prove it · perplexity is one over the average (geometric) probability

Claim. With loss $\mathcal L=-\frac1n\sum_t\ln p_t$ , the perplexity is $e^{\mathcal L}=\big(p_1p_2\cdots p_n\big)^{-1/n}$ .

A sum of logs is the log of the product:

\sum_{t=1}^n\ln p_t=\ln\big(p_1p_2\cdots p_n\big).

\ln a+\ln b=\ln(ab)

, used

n-1

times.

Call the product

P=p_1\cdots p_n

. Then

\mathcal L=-\tfrac1n\ln P

, so

\begin{aligned}e^{\mathcal L}&=e^{-\frac1n\ln P}\\ &=P^{-1/n}.\end{aligned}

∎ For "we drink hot chai":

0.1^{-1/3}=10^{1/3}\approx2.1544

. The

n

-th root is the geometric mean: the one probability that, used

n

times, gives the same product.

Prove it · a uniform guess has perplexity exactly V

Claim. A model that gives every one of $V$ tokens the probability $1/V$ has loss $\ln V$ and perplexity $V$ on any text.

Whatever the true token is, its probability is

1/V

, so every surprise is

-\ln(1/V)=\ln V

, and so is their average. The text does not matter: every token is equally unexpected.

e^{\ln V}=V

. ∎ GPT-2 at the very start of training: loss

\ln50\,257\approx10.82

, perplexity 50 257. A fair die with 50 257 faces.

In one sentence: A model is scored like a gambler who keeps only what it bet on each true word — multiply the probabilities for the sentence, take logs to turn the product into a sum, average for the loss, and exponentiate for the perplexity, the number of faces on its die.

Inside the stack, by the numbers

Imagine this

Picture a tall office tower. On the ground floor is a huge dictionary: one row for every token. Above it, floor after floor, every floor has the same two rooms. In the meeting room the words talk to each other — that is attention. In the desk room each word thinks alone — that is the feed-forward network.

Where do most of the people sit? Count the chairs.

The question. "GPT-2 small has 124 million parameters." Where do all those numbers live?

Call the width of the model $d$ : every token travels through the tower as a list of $d$ numbers. Count the weights in one floor — one transformer block of Unit 18.

The meeting room (attention) has four $d\times d$ matrices: $W_Q,W_K,W_V$ and the output mix $W_O$ . That is $4d^2$ numbers, however many heads there are.
The desk room (feed-forward) widens each token from $d$ to $4d$ numbers and brings it back: a $d\times4d$ matrix and a $4d\times d$ matrix, $8d^2$ numbers.
Together: $12d^2$ per block. The small extras — biases and the two layer norms — add $13d$ .
The ground floor is the token table: $V\times d$ numbers, one row per token. With weight tying (Unit 18) the same table also scores the next token at the top, so it is counted once.

GPT-2 small (Radford and colleagues, 2019) has $d=768$ , $L=12$ blocks, a vocabulary of $V=50\,257$ and a context of 1 024 tokens.

$\begin{aligned}\text{one block}&=12\times768^2+13\times768\\ &=7\,077\,888+9\,984\\ &=7\,087\,872\\ \text{12 blocks}&=85\,054\,464\\ \text{token table}&=50\,257\times768\\ &=38\,597\,376\\ \text{positions}&=1\,024\times768=786\,432\\ \text{final norm}&=2\times768=1\,536\\ \text{total}&=124\,439\,808\end{aligned}$

That is the "124 million". About 68 % sits in the blocks and 31 % in the token table.

Now make it big. A 7-billion-class model has $d=4\,096$ and $L=32$ . The blocks hold $12Ld^2=6\,442\,450\,944$ . With one shared table of 32 000 tokens ( $32\,000\times4\,096=131\,072\,000$ ) the count is $6\,573\,522\,944$ , about 6.6 billion. Now the blocks are 98 % of the model. (Real 7-billion models come out a little higher: their feed-forward rooms are slightly wider, and many keep a separate output table. But $12Ld^2$ is the right first guess.)

A variant: the mixture of experts. Think of a big hospital. It has eight specialists, but each patient sees only two of them; a receptionist decides which. A mixture-of-experts (MoE) block does the same: it keeps several feed-forward rooms, the experts, and a small router scores them for each token. Only the top two are run.

Say four experts get the router scores $(2,\ 1,\ 0.5,\ -1)$ . Keep the top two, experts 1 and 2, and softmax just their scores: $e^{2}/(e^{2}+e^{1})\approx0.7311$ and $0.2689$ . The token's output is $0.7311\times$ (expert 1's answer) $+\ 0.2689\times$ (expert 2's answer). Experts 3 and 4 are not computed at all.

With $d=4\,096$ , one expert holds $8d^2=134\,217\,728$ numbers. Eight experts hold $1\,073\,741\,824$ per layer, but each token wakes up only two: $268\,435\,456$ , a quarter.

Try: Press GPT-2 small: the read-out says 124 439 808, and the violet table is a big share of the tower. Press 7B-class: 6 573 522 944, and the table almost vanishes under 32 fat floors. Now press 8 experts, top 2: the tower grows to about 36.64 billion numbers, but a token uses only about 10.87 billion. Drag width d from 768 to 1 536: every floor now holds four times as many numbers and becomes twice as wide each way, while the violet table only doubles.

drag the picture to orbit

width d768

blocks L12

vocabulary V50 257

experts1

biases & layer norms learned position table

Why does this work?

Why do the blocks win as models grow? Every matrix inside a block connects all $d$ numbers of a token to all $d$ (or $4d$ ) numbers on the other side, so its size is " $d$ times $d$ ". Double the width and every block gets four times bigger, while the token table only doubles. Small models are mostly dictionary; big models are mostly floors. And why experts? Different tokens need different knowledge — Hindi, Python, chemistry. Experts let a model hold much more knowledge while each token pays only for the two rooms it visits.

Dense against mixture of experts, per layer, at $d=4\,096$ .

	dense feed-forward	8 experts, top 2
numbers stored	134 217 728	1 073 741 824 (8 ×)
numbers used per token	134 217 728	268 435 456 (2 ×)
memory needed	small	large: every expert must be loaded
extra parts	none	a router ( $d\times8$ numbers) and a rule to keep the experts evenly busy

Rule of thumb. Parameters $\approx12Ld^2+Vd$ . For a mixture of experts, quote two numbers: the total you must store, and the active count each token uses. Compute follows the active count; memory follows the total.

Trap

Parameters are not sentences stored in a drawer. They are the weights of the matrices above, and knowledge lives spread across all of them. And in a mixture of experts, "total parameters" is not "parameters used per token": the 8-expert tower of the widget stores 36.6 billion numbers but computes like a model of about 10.9 billion — and it still needs the memory of all 36.6 billion.

The realization

$\text{parameters}\approx12\,L\,d^{2}+V\,d$

The floors plus the dictionary. Each block holds $4d^2$ in attention and $8d^2$ in the feed-forward room. The width enters squared, so as models grow the floors swallow almost everything.

Pause & predict

You double the width $d$ of a model and keep everything else the same. Roughly how many times more numbers do the blocks hold?

Pause & predict

A mixture-of-experts layer has 16 experts, each as big as one dense feed-forward room, and its router picks the top 2 for every token. Compared with the dense room, what happens to the numbers stored and the work done per token?

If you want the algebra · 1 proof, step by step

Prove it · a block holds 12d² + 13d numbers, and GPT-2 small holds 124 439 808

Claim. A GPT-2 block of width $d$ has $12d^2+13d$ parameters. With $d=768$ , 12 blocks, $V=50\,257$ , a context of 1 024 and a tied output layer, the total is $124\,439\,808$ .

Attention.

W_Q,W_K,W_V,W_O

are

d\times d

, each with a bias of length

d

4d^2+4d.

Splitting into

h

heads cuts each matrix into

h

slices of width

d/h

; the total stays

d\times d

(Unit 18).

Feed-forward.

d\to4d

: a

d\times4d

matrix and a bias of

4d

4d\to d

: a

4d\times d

matrix and a bias of

d

\begin{aligned}&4d^2+4d+4d^2+d\\ &=8d^2+5d.\end{aligned}

The "4×" widening is GPT-2's choice; other models use other factors.

Two layer norms, each with a scale and a shift of length

d

4d

. Add the three parts:

\begin{aligned}&4d^2+4d+8d^2+5d+4d\\ &=12d^2+13d.\end{aligned}

d=768

7\,077\,888+9\,984=7\,087\,872

The rest. 12 blocks:

85\,054\,464

. Token table

V d=38\,597\,376

(tied, counted once). Position table

1\,024\,d=786\,432

. A final layer norm

2d=1\,536

. Sum:

\begin{aligned}&85\,054\,464+38\,597\,376\\ &+786\,432+1\,536\\ &=124\,439\,808.\end{aligned}

∎ Without tying, the output layer would add another

38\,597\,376

In one sentence: An LLM is a tower of identical floors, each holding 4d² numbers in the meeting room and 8d² at the desks, over a V × d dictionary — so parameters ≈ 12Ld² + Vd, and a mixture of experts stores many desk rooms but sends each token to only two.

Picking the word: greedy and temperature

Imagine this

Picture a dial on the model, like the thermostat on an air conditioner — but this one sets boldness. Turn it cold, and the model always says the most obvious word, like a shy student who only ever gives the textbook answer. Turn it warm, and it takes small chances. Turn it hot, and it gets more and more adventurous — like a friend telling stories at two in the morning — until, at the very hottest, it talks nonsense.

The question. The model hands us a probability for each of 50 000 tokens. Which one do we actually write?

Greedy: always take the top one. It is safe and it gives the same answer every time. It is also dull, and it falls into loops. Here is the tiny character model of Unit 17, writing greedily after "the train to ":

the train to patna will arrive on platform three. the train to patna will arrive on platform three. the train to patna…

— and so on forever. Once a sentence is the most likely thing to say, it stays the most likely thing to say next time too. Beam search (Unit 17) keeps the best few partial sentences instead of one. It is excellent when there is one right answer, as in translation. For open writing it finds text that is too probable — bland and repetitive. So chat models usually sample: they draw the next token at random, each token with its own probability, like a lottery where chai holds 64 tickets out of 100.

Temperature is the boldness dial. Before the softmax, divide every score by a number $T$ .

By hand. The scores are $(2,\ 1,\ 0,\ -1)$ for chai, coffee, water, the.

$T=1$ : the scores stay as they are. Softmax gives $(0.644,\ 0.237,\ 0.087,\ 0.032)$ .
$T=0.5$ (cold): dividing by 0.5 doubles the scores to $(4,\ 2,\ 0,\ -2)$ . Softmax gives $(0.865,\ 0.117,\ 0.016,\ 0.002)$ . The favourite grows stronger.
$T=2$ (hot): the scores halve to $(1,\ 0.5,\ 0,\ -0.5)$ . Softmax gives $(0.455,\ 0.276,\ 0.167,\ 0.102)$ . The underdogs catch up.

Read it aloud. The ratio between two tokens' probabilities is $e^{\text{gap}/T}$ , where "gap" is the difference of their scores. Chai against coffee has a gap of 1. At $T=1$ chai is $e^{1}\approx2.718$ times as likely; at $T=0.5$ it is $e^{2}\approx7.389$ times; at $T=2$ only $e^{0.5}\approx1.649$ times. Cold stretches the gaps; heat shrinks them. Turn $T$ all the way down and every gap becomes huge: all the probability lands on the top token — that is greedy. Turn it all the way up and every gap shrinks to nothing: every token gets $1/4$ .

You have met this dial before: it is the "how picky" knob of Unit 18, turned upside down, and the temperature of Unit 17's talking model.

Try: Press T = 0.5: chai climbs to 0.865. Press T = 2: chai drops to 0.455 and "the" rises to 0.102. Drag T all the way left: chai reaches 0.99995 — greedy. Drag it all the way right, to 10: the four bars nearly level out (0.289, 0.261, 0.236, 0.214), on their way to 1/4 each. The order of the four never changes.

temperature T1

Why does this work?

Softmax turns gaps between scores into ratios between probabilities: a gap of $g$ becomes a ratio of $e^{g}$ . Dividing every score by $T$ divides every gap by $T$ , so it bends all the ratios at once, in the same direction, without ever swapping two tokens. That is why one dial is enough to move smoothly from "always the obvious word" to "anything goes" — and why it never changes the model's opinion of which token is best, only how strongly it insists.

Ways to pick, side by side.

method	what it does	good for	risk
greedy	always the top token	short factual answers, code	dull; loops
beam search	keeps the best few partial sentences	translation, where one answer is right	bland, repetitive open text
sampling, $T=1$	draws with the model's own probabilities	variety	now and then a strange token (§5)
$T\lt 1$	sharpens: favourites grow	careful, focused writing	at the extreme, greedy again
$T\gt 1$	flattens: underdogs catch up	brainstorming, poems	nonsense as $T$ grows

Rule of thumb. For facts and code, stay cold ( $T$ near 0, or greedy). For chat, a little under or around 1. For ideas and stories, a little above 1 — and always with the tail cut off, as in the next section.

Trap

$T=0$ is not "the most accurate setting". It picks the most probable token at each step, which can still be wrong, and a string of individually most-likely tokens need not be the most likely text. And a high temperature does not make the model more creative in any deep sense: it adds randomness, not knowledge.

The realization

$\begin{gathered}p_i(T)=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}\\ \frac{p_i}{p_j}=e^{(z_i-z_j)/T}\end{gathered}$

Temperature divides the scores before the softmax. Every ratio between two tokens is raised to the power $1/T$ : cold makes the favourite a dictator, heat makes every token equal, and the order never changes.

Quick check

Scores $(2,\ 1,\ 0,\ -1)$ at $T=0.5$ . How many times more likely is chai (score 2) than coffee (score 1)?

Pause & predict

A bug makes the model double all of its scores before the softmax. Which temperature gives back exactly the probabilities of the model without the bug?

Pause & predict

You set $T=100$ for a model with a vocabulary of 50 000 tokens. What will it write?

If you want the algebra · 1 proof, step by step

Prove it · temperature bends every ratio, never the order, and has two limits

Claim. With $p_i(T)=e^{z_i/T}/\sum_je^{z_j/T}$ : (a) $p_i/p_j=e^{(z_i-z_j)/T}$ , so the order of the tokens is the same at every $T\gt0$ ; (b) as $T\to0$ all the probability goes to the top token (if it is unique); (c) as $T\to\infty$ every token gets $1/n$ .

(a) The totals cancel in a ratio:

\frac{p_i}{p_j}=\frac{e^{z_i/T}}{e^{z_j/T}}=e^{(z_i-z_j)/T}.

z_i\gt z_j

, the power is positive for every

T\gt0

, so the ratio is bigger than 1:

i

stays ahead of

j

. Scores

(2,1,0,-1)

: chai/coffee

=e^{1/T}

, which is 2.718 at

T=1

and 7.389 at

T=0.5

(b) Let

z_1

be the unique top score. Divide top and bottom by

e^{z_1/T}

p_1=\frac{1}{1+\sum_{j\ne1}e^{-(z_1-z_j)/T}}.

Every gap

z_1-z_j

is positive, so as

T\to0

each

e^{-(z_1-z_j)/T}\to0

and

p_1\to1

. At

T=0.1

1/(1+e^{-10}+e^{-20}+e^{-30})\approx0.99995

T\to\infty

, every

z_j/T\to0

, so every

e^{z_j/T}\to1

and every

p_j\to1/n

. ∎ At

T=10

(0.289,\ 0.261,\ 0.236,\ 0.214)

, close to

1/4

each.

In one sentence: Greedy always takes the top token and ends up in loops, so chat models sample — and temperature, a thermostat on boldness, divides the scores before the softmax: cold stretches every gap until the favourite always wins, heat shrinks every gap until any token can win.

Cutting the tail: top-k and top-p

Imagine this

A wedding buffet has 200 dishes. Most guests fill their plates from the ten favourites. Now suppose you choose your dish by throwing a dart that can land anywhere on the table, with a small chance for every dish. Each strange dish is rare. But there are so many of them that, every few plates, you end up eating a bowl of pickle for dinner.

The question. Sampling makes text lively. So why do good chatbots almost never drop a nonsense word into the middle of a sentence?

The danger is the tail. Say the 20 sensible tokens hold 0.9 of the probability, and the other 50 000-odd tokens share the last 0.1 — each one tiny. Then every step has a 1-in-10 chance of drawing junk. Over a 50-token reply, the chance of never drawing junk is $0.9^{50}\approx0.005$ . Almost every reply would contain a pickle.

The fix is to cut the tail off before drawing. There are two popular cuts.

Top-k: keep the $k$ most likely tokens, and share their probability back out so it adds up to 1 again (renormalise).
Top-p, also called nucleus sampling (Holtzman and colleagues, 2020): go down the list from the top, adding up probabilities, and stop as soon as the running total reaches $p$ . Keep those tokens, and renormalise.

By hand. "Every morning I drink hot …": chai 0.40, coffee 0.25, milk 0.15, water 0.10, soup 0.06, oil 0.04.

Top-k with $k=3$ . Keep chai, coffee, milk: together 0.80. Divide each by 0.80: $(0.5,\ 0.3125,\ 0.1875)$ . Oil is gone for good.
Top-p with $p=0.75$ . Running totals: 0.40, 0.65, 0.80, … The total first reaches 0.75 at the third token. Keep 3 tokens — the same three, here.

Where the two differ. Take a flat list: "My favourite colour is …" — blue 0.20, green 0.18, red 0.17, yellow 0.16, pink 0.15, black 0.14. Running totals 0.20, 0.38, 0.55, 0.71, 0.86: top-p 0.75 keeps 5 tokens, because many answers are fine. Now a peaked list: "The capital of India is New …" — Delhi 0.90, York 0.04, Zealand 0.03, Jersey 0.02, Moon 0.01. The first token alone passes 0.75: top-p keeps 1. Top-k 3 keeps 3 in both cases — too few for the colours, and for the capital it keeps "York" and "Zealand" in the draw. Top-p adapts to how sure the model is; top-k does not.

The order, in real samplers. Temperature first, then top-k, then top-p, then renormalise, then draw.

Try: On hot … set top-k to 3: chai, coffee, milk become 0.5, 0.3125, 0.1875. Set k back to off and top-p to 0.75: three tokens again. Switch to colour: five tokens survive; to capital: only Delhi. Press draw 1 000: the gold diamonds land on the pink bar tops. Then open write with it and press greedy: Unit 17's little model repeats one sentence forever. Press T 1 · top-p 0.9 and write again.

temperature T1

top-koff

top-poff

Why does this work?

Renormalising is the same as saying: "draw from the model as usual, but if the dart lands in the tail, throw it again". The kept tokens keep their relative odds exactly — chai stays 1.6 times as likely as coffee — so the model's judgement among the good tokens is untouched. Only the long, thin tail, where the model's own numbers are least trustworthy, is removed. And top-p adapts because it counts probability, not tokens: a sure model has its probability packed into a few tokens, an unsure one spreads it over many.

The two cuts, side by side (with the lists above, $k=3$ and $p=0.75$ ).

	top-k	top-p (nucleus)
keeps	always $k$ tokens	the fewest top tokens holding at least $p$
flat list (colours)	3 tokens — throws away 0.45 of good answers	5 tokens
peaked list (capital)	3 tokens — York and Zealand still in the draw	1 token
with temperature	same set at any $T$	heat spreads probability, so more tokens pass
setting you will see	$k$ = 40 or 50	$p$ = 0.9 or 0.95

Rule of thumb. Temperature → top-k → top-p → renormalise → draw. If you set only one, set top-p around 0.9 with $T$ near 1: it keeps the variety when many answers are fine and removes the pickle when one answer is right.

Trap

Top-p 0.9 does not mean "keep 90 % of the tokens". It means "keep the fewest tokens that together hold 90 % of the probability". For a sure model that can be a single token out of 50 000; for an unsure one, thousands.

The realization

$\begin{gathered}q_i=\frac{p_i}{\sum_{j\in S}p_j}\ \ \text{for }i\in S\\ q_i=0\ \ \text{otherwise}\end{gathered}$

Choose a set $S$ of good tokens — the top $k$ , or the fewest top tokens whose probabilities reach $p$ — and share the whole probability out among them in their old proportions. Then draw.

Quick check

The colour list (0.20, 0.18, 0.17, 0.16, 0.15, 0.14) with top-k 3. How much of the model's probability is thrown away?

Pause & predict

The capital list (0.90, 0.04, 0.03, 0.02, 0.01) with top-p 0.95. How many tokens are kept?

Pause & predict

You set top-k to 1 and temperature to 2. What does the sampler do?

If you want the algebra · 1 proof, step by step

Prove it · renormalising is the same as "draw again if you land in the tail"

Claim. Keep a set $S$ of tokens with total probability $P(S)=\sum_{j\in S}p_j$ . Drawing from the model and re-drawing whenever the token is outside $S$ gives token $i\in S$ with probability exactly $p_i/P(S)$ — the renormalised list.

Token

i

comes out on the first try with probability

p_i

. It comes out on the second try if the first draw missed

S

(probability

1-P(S)

) and the second gave

i

(1-P(S))\,p_i

. And so on. Each try is a fresh, independent draw.

Add all the tries with

r=1-P(S)

, the chance of missing

S

(a geometric series,

1+r+r^2+\dots=1/(1-r)

for

0\le r\lt1

\begin{aligned}&P(\text{get }i)\\ &=p_i\big(1+r+r^2+\dots\big)\\ &=\frac{p_i}{1-r}=\frac{p_i}{P(S)}.\end{aligned}

∎ This is the conditional probability

P(i\mid i\in S)

of Unit 14. With

S=

{chai, coffee, milk},

P(S)=0.8

0.40/0.8=0.5

0.25/0.8=0.3125

0.15/0.8=0.1875

. The ratios inside

S

are untouched.

In one sentence: Thousands of tiny probabilities add up to a real chance of nonsense, so before drawing we cut the tail — top-k keeps a fixed number, top-p keeps the fewest tokens that hold p of the probability — and renormalise, like a buffet where you only throw darts at the good dishes.

The KV cache: never recompute the past

Imagine this

A tailor is stitching a long kurta, one panel at a time. For every new panel she needs the measurements of every panel already stitched. A careless tailor would measure the whole kurta again before each new panel. A sensible tailor writes each measurement in her notebook once, and simply reads it back.

The question. To write token number 1 001, must the model redo all the work it already did for the first 1 000?

Recall how attention works (Unit 18). In every layer, every token makes a query, a key and a value. The new token's query is scored against the keys of all the earlier tokens, and it takes a blend of their values. So the new token needs every earlier key and value, in every layer.

Here is the key fact. Because of the causal mask (Unit 18), an earlier token never looks at anything that came after it. So its keys and values, in every layer, are exactly the same now as when it was written. Nothing new can change them. So keep them: the KV cache — the tailor's notebook of keys and values.

Count the work. Count the rows of keys and values the model computes in one layer.

Without a cache, run $t$ processes all $t$ tokens again: $1+2+\dots+n=\frac{n(n+1)}{2}$ rows to write $n$ tokens. For $n=1\,000$ : $500\,500$ rows.
With a cache, each run computes one new row and reads the rest from the notebook: $1\,000$ rows. Five hundred times less.

The price is memory. For every token the notebook holds a key and a value (that is the 2), in every layer, each $d$ numbers long, each number $b$ bytes:

$\begin{aligned}\text{bytes per token}&=2\times L\times d\times b\\ &=2\times32\times4\,096\times2\\ &=524\,288\\ &=0.5\ \text{MiB}\end{aligned}$

for a 7-billion-class model ( $L=32$ , $d=4\,096$ , 16-bit numbers of 2 bytes). A conversation of 4 096 tokens needs $4\,096\times0.5$ MiB $=2$ GiB — just for the notebook of one conversation. (A MiB is $2^{20}=1\,048\,576$ bytes; a GiB is $2^{30}$ .)

Three ways to share the notebook. The model has 32 attention heads, and in plain multi-head attention each head keeps its own keys and values: 2 GiB. In grouped-query attention (GQA), the 32 query heads are split into 8 groups, and each group shares one set of keys and values: 8 sets instead of 32, so $2\ \text{GiB}\times8/32=512$ MiB. In multi-query attention (MQA), all 32 heads share one set: 64 MiB. Sharing costs a little quality, because the heads can no longer each look for their own kind of key.

Two phases of writing. When you send a prompt, the model first reads it all at once, in parallel, filling the notebook — this is the prefill, and it is Unit 18's "every position at once". Then it decodes: one token per run, and each run reads the whole notebook. Decoding is slow not because of the arithmetic, but because of the reading: for every single token, all the model's weights and the whole notebook must be fetched from memory again. For a long chat, the notebook becomes a big part of that reading.

Try: Press ▶ write 10 tokens with the cache on: one new row per run, 10 rows in all. Switch to no cache and write again: every run redoes every earlier row — 55 rows, which is 10 × 11 / 2. Push text length to 1 000: 500 500 against 1 000. In the memory panel, press 7B-class: 2 GiB with 32 key-value heads, 512 MiB with 8, 64 MiB with 1.

text length10

memory panel

layers L32

width d4 096

tokens kept4 096

bytes per number2

Why does this work?

Because the past is frozen. A token's key and value depend only on the tokens up to it — the mask guarantees that — and those tokens never change once written. So a key computed at run 5 is still exactly right at run 1 000. The cache is not an approximation: the model with the cache computes exactly the same probabilities as the model without it, just without repeating itself.

Sharing keys and values, side by side (32 query heads, $L=32$ , $d=4\,096$ , 2 bytes, 4 096 tokens).

	multi-head (MHA)	grouped-query (GQA)	multi-query (MQA)
sets of keys and values	32, one per head	8, one per group of 4 heads	1, shared by all
notebook size	2 GiB	512 MiB	64 MiB
quality	the reference	almost the same	a little lower
speed of decoding	slowest: most to read	fast	fastest

Rule of thumb. Cache memory per token $=2\times L\times(\text{KV heads})\times(\text{head size})\times(\text{bytes})$ . Most recent open models use grouped-query attention: nearly all the quality of multi-head, at a fraction of the memory.

Trap

The cache saves computation, and it costs memory. It grows with every token of every conversation the server is holding at once. For long chats the notebook, not the model's weights, is often what fills the memory.

The realization

$\begin{gathered}\text{rows without a cache}=\frac{n(n+1)}{2}\\ \text{rows with a cache}=n\\ \text{bytes}=2\,L\,n_{kv}\,d_{\text{head}}\,b\,n\end{gathered}$

The mask freezes the past, so every earlier key and value can be written once and read back forever: the work falls from a triangle to a line, and the notebook grows by the same few hundred kilobytes with every token.

Quick check

Without a cache, how many rows of keys and values (in one layer) does the model compute to write 2 000 tokens?

Pause & predict

A model uses grouped-query attention with 8 key-value heads, and its notebook for one long chat is 512 MiB. The designers switch to 4 key-value heads. What happens to the notebook?

If you want the algebra · 2 proofs, step by step

Prove it · 1 + 2 + … + n = n(n + 1)/2

Claim. Writing $n$ tokens without a cache computes $1+2+\dots+n=\frac{n(n+1)}{2}$ rows of keys and values per layer.

Write the sum forwards and backwards, and add the two lines column by column:

\begin{aligned}S&=1+2+\dots+n\\ S&=n+\dots+2+1\\ 2S&=n\times(n+1).\end{aligned}

Every column adds up to

n+1

, and there are

n

columns.

2S=n(n+1)

and

S=n(n+1)/2

. ∎

n=1\,000

500\,500

. The cache does

n

: the ratio

(n+1)/2

grows with the length of the text.

Prove it · the cache changes nothing: earlier keys and values never change

Claim. In a model with a causal mask, the key and value of token $j$ in every layer depend only on tokens $1,\dots,j$ . So adding tokens after $j$ never changes them, and reading them from a cache gives exactly the same result as recomputing them.

Layer 0: token

j

's vector is its embedding (plus position) — it depends on token

j

only. The start of an induction on the layers.

Suppose in layer

\ell

every token's vector

\mathbf x_i^{(\ell)}

depends only on tokens

1,\dots,i

. Token

j

's key and value in this layer are

\mathbf x_j^{(\ell)}W_K

and

\mathbf x_j^{(\ell)}W_V

: they depend only on tokens

1,\dots,j

. Its attention output uses the keys and values of tokens

i\le j

only (the mask gives every

i\gt j

a share of 0), and each of those depends only on tokens up to

i\le j

. The feed-forward room and layer norm act on token

j

alone. So

\mathbf x_j^{(\ell+1)}

depends only on tokens

1,\dots,j

. ∎ Without the mask (a BERT-style reader, Unit 18) every token looks at the future too, a new token changes every earlier vector, and no cache is possible.

In one sentence: Because the mask freezes the past, the model writes every token's keys and values into a notebook once and reads them back forever — n rows of work instead of n(n+1)/2 — and pays in memory, 2·L·d·b bytes per token, which grouped-query attention shrinks by sharing the notebook between heads.

Longer contexts: rotary positions at full size

Imagine this

A clock with only a seconds hand cannot tell 10:15 from 10:16: the hand is in the same place every minute. Add a minutes hand and an hours hand, and it tells every moment of the day apart. The fast hand sorts out small differences; the slow hands sort out big ones.

A transformer tells positions apart with exactly such a clock — with many hands, turning at many speeds.

The question. How does a model with a context of thousands of tokens tell a word 3 positions back from one 3 000 positions back — and why do models trained on short texts get confused by long ones?

In Unit 18 we met rotary positions (RoPE) with one pair of numbers: the query of the word at position $m$ is turned by the angle $m\theta$ , the key of the word at position $n$ by $n\theta$ , and their dot product depends only on the gap $m-n$ . Now the full picture.

Split the $d$ numbers into $d/2$ pairs. Each pair is a little arrow on its own clock face.
Each pair turns at its own speed. Pair $i$ turns by $\theta_i=10\,000^{-2i/d}$ radians per position, for $i=0,1,\dots,d/2-1$ . With $d=8$ there are four pairs: $\theta=(1,\ 0.1,\ 0.01,\ 0.001)$ .
Each speed has its period, the number of positions for one full turn, $2\pi/\theta_i$ : about $6.28,\ 62.8,\ 628$ and $6\,283$ positions. A seconds hand, a minutes hand, an hours hand and a days hand.
Score as usual. The dot product of the turned query and the turned key is the sum over the pairs. In each pair only the angle between the two hands, $(m-n)\theta_i$ , matters.

By hand. Let every pair of the query and of the key be $(1,0)$ . The query stands at $m=5$ , the key at $n=3$ , a gap of 2. Pair $i$ contributes $\cos(2\theta_i)$ :

$\begin{aligned}&\cos2+\cos0.2+\cos0.02+\cos0.002\\ &\approx-0.416+0.980+1.000+1.000\\ &\approx2.564.\end{aligned}$

Now move both words 1 000 positions later: $m=1\,005$ , $n=1\,003$ . Every hand has turned a long way, but the angle between each pair of hands is still $2\theta_i$ , and the score is still $2.564$ . Where the two words stand does not matter — only how far apart they are.

Why models break on longer texts. Say a model was trained on texts of at most 2 048 tokens. The slowest pair ( $\theta=0.001$ ) has then only ever seen the two hands up to $2\,048\times0.001=2.048$ radians apart (about 117°). Ask it to connect words 8 000 positions apart, and that pair shows an angle of 8 radians — more than a full turn, something it never saw in training. The fast pairs are fine: they went round and round during training and have seen every angle.

The fix: position interpolation (Chen and colleagues, 2023). Squeeze the new, longer positions into the trained range. To read 8 192 tokens with a model trained on 2 048, multiply every position by $2\,048/8\,192=0.25$ . Position 6 000 is treated as position 1 500, and every angle is back inside what the model knows. The cost: neighbours now look closer than before, so the model gets a short extra training on long texts to get used to it.

Try: The clocks start at the worked example: m = 5, gap 2, score 2.564. Drag query position m to 1 005: every hand turns, but every gold arc and the score stay the same. Now drag gap to 6 000: the slowest clock's arc runs past its green wedge and turns red. Switch on interpolate ×0.25: the gap now acts like 1 500 and the arc is back inside the wedge.

drag the picture to orbit

query position m5

gap m − n2

interpolate ×0.25 (read 8 192 with a 2 048 model)

Why does this work?

Turning both arrows by the same extra angle never changes the angle between them — just as moving a meeting from 3 to 5 o'clock does not change the fact that it lasts an hour. So the score can only see the gap $m-n$ . And using many speeds gives both precision and reach: the fast hands separate "1 back" from "2 back", the slow hands separate "a sentence back" from "a chapter back", and no two gaps inside the trained range look alike on all the clocks at once.

Four ways to handle position and length, side by side.

method	how it marks position	past the trained length
learned table (GPT-2)	one learned row per position	no rows exist: it cannot go further
clock tags (sinusoidal)	sines and cosines added to the word	defined, but unfamiliar to the model
rotary (RoPE)	turns q and k; the score sees only the gap	slow hands meet new angles; quality drops
RoPE + interpolation	positions squeezed by trained ÷ new length	angles stay familiar; a short fine-tune restores quality

Rule of thumb. Pair $i$ turns at $10\,000^{-2i/d}$ radians per position. To stretch a model's context by a factor $s$ , squeeze positions by $1/s$ (or, in later variants, slow down only the slow hands) and fine-tune briefly on long texts.

Trap

Rotary positions turn the queries and keys only — never the values. Position is needed to decide where to look; what is handed over, the value, is just the word's content.

The realization

$\begin{gathered}\theta_i=10\,000^{-2i/d}\\ (R_{m\theta_i}\mathbf q)\cdot(R_{n\theta_i}\mathbf k)\\ =\mathbf q\cdot(R_{(n-m)\theta_i}\mathbf k)\end{gathered}$

Each pair of numbers is a clock hand turning at its own speed. The score of a query and a key depends only on the gap between them, fast hands give precision, slow hands give reach — and squeezing positions keeps every hand inside the angles the model has learned.

Pause & predict

A query at position 40 and a key at position 30 give some score. You move the query to position 540 and the key to 530. What happens to the score?

Quick check

A model trained on 4 096 tokens must read 16 384. With position interpolation, what is the scale, and where does position 10 000 land?

If you want the algebra · 1 proof, step by step

Prove it · turning both arrows leaves only the gap

Claim. With the turn $R(a)=\begin{pmatrix}\cos a&-\sin a\\ \sin a&\cos a\end{pmatrix}$ , for any pairs $\mathbf q,\mathbf k$ : $(R(m\theta)\mathbf q)\cdot(R(n\theta)\mathbf k)=\mathbf q\cdot\big(R((n-m)\theta)\mathbf k\big)$ . It depends on $m$ and $n$ only through $n-m$ .

A dot product is a row times a column:

(A\mathbf q)\cdot(B\mathbf k)=\mathbf q^{\mathsf T}A^{\mathsf T}B\,\mathbf k

. Transpose of a product:

(A\mathbf q)^{\mathsf T}=\mathbf q^{\mathsf T}A^{\mathsf T}

Turning back by

a

undoes turning by

a

, so

R(a)^{\mathsf T}=R(-a)

. And two turns add up:

R(-a)R(b)=R(b-a)

. So

\begin{gathered}R(m\theta)^{\mathsf T}R(n\theta)\\ =R\big((n-m)\theta\big).\end{gathered}

Multiply the matrices out and use

\cos(b-a)=\cos a\cos b+\sin a\sin b

and

\sin(b-a)=\sin b\cos a-\cos b\sin a

Put 1 and 2 together:

\mathbf q^{\mathsf T}R((n-m)\theta)\mathbf k

. The full score is the sum of this over the

d/2

pairs, each with its own

\theta_i

, so it too depends only on

n-m

. ∎ With every pair

(1,0)

: each pair gives

\cos((n-m)\theta_i)

. Gap 2,

d=8

\cos2+\cos0.2+\cos0.02+\cos0.002\approx2.564

, at any

m

In one sentence: Rotary positions give every pair of numbers its own clock hand, turning at 10 000^(−2i/d) radians per step, so a score sees only the gap between two words — fast hands for precision, slow hands for reach — and squeezing positions keeps the slow hands inside the angles the model learned.

Scaling laws: bigger, with more data, predictably better

Imagine this

A school has a fixed budget. It can hire more teachers or buy more books. All teachers and no books, and the teachers have nothing to teach from. A huge library with one teacher, and nobody explains anything. The best school splits the money between the two.

Training an LLM is the same choice. The budget is computing time. The teachers are the model's parameters; the books are the tokens of text it reads.

The question. You have a fixed amount of computing. Should you train a bigger model, or feed a smaller one more text?

What a training run costs. A model with $N$ parameters reading $D$ tokens needs about

$C\approx6ND\ \text{ operations.}$

Why 6? In the forward pass each parameter is used once per token for a multiply and an add: 2 operations. The backward pass has two jobs for every weight — pass the blame back to the inputs, and find the weight's own slope (Rules A and B of Unit 15) — each another multiply and add: 4 more. So the backward pass costs about twice the forward. For DeepMind's Chinchilla, $N=70$ billion and $D=1.4$ trillion: $C=6\times70\times10^{9}\times1.4\times10^{12}=5.88\times10^{23}$ .

How the loss falls. Train many models of different sizes on different amounts of text, and the final loss follows a power law. On log–log axes a power law is a straight line — logs, once again, turn a hard curve into a simple one. Hoffmann and colleagues (2022) fitted

$\begin{aligned}L(N,D)=1.69&+\frac{406.4}{N^{0.34}}\\ &+\frac{410.7}{D^{0.28}}.\end{aligned}$

Read it aloud. 1.69 is the floor no model can beat: the randomness of language itself. The second term is what a small model loses by being small; it shrinks as $N$ grows. The third is what a model loses by reading too little; it shrinks as $D$ grows. Ten times more parameters multiplies the second term by $10^{-0.34}\approx0.457$ : it roughly halves.

Two real models.

Gopher: 280 billion parameters, 300 billion tokens. $C=5.04\times10^{23}$ . The formula gives $L\approx1.9933$ .
Chinchilla: 70 billion parameters, 1.4 trillion tokens. $C=5.88\times10^{23}$ — about the same budget. The formula gives $L\approx1.9366$ .

The model four times smaller, fed almost five times more text, wins. Gopher had hired too many teachers and bought too few books.

The rule of thumb: about 20 tokens per parameter, $D\approx20N$ . Then $C=6N\times20N=120N^2$ , so

$\begin{gathered}N\approx(C/120)^{1/2}\\ D\approx20N.\end{gathered}$

For $C=10^{23}$ : $N\approx28.9$ billion parameters and $D\approx577$ billion tokens. (An honest note: the fitted formula itself puts its lowest point at Chinchilla's budget near 32 billion parameters on 3 trillion tokens. Its valley is so flat there that Chinchilla's own split is only 0.007 higher. The paper's two other methods gave about 20 tokens per parameter, and that became the rule everyone quotes.)

Training, in practice. The optimiser is Adam (Unit 11). The learning rate is warmed up from zero over the first few thousand steps, then slowly lowered along a cosine curve. Numbers are stored in 16 bits for speed, with a 32-bit master copy of the weights so small updates are not lost. Each step reads a batch of millions of tokens.

Try: Press Chinchilla's budget: the green Chinchilla point sits on the pink line, near the bottom of the valley (1.9366; the lowest point is 1.9300). Press Gopher's budget: the orange Gopher point sits far along the pink line towards big models and little text, well up the side of the valley (1.9933). Drag budget C up: the pink line slides forward, towards the corner of big models and lots of text, and the gold lowest point moves to bigger models and more tokens together.

drag the picture to orbit

budget C5.9e23

Why does this work?

Each ingredient runs into diminishing returns on its own. Make the model ten times bigger and its term only halves, while the data term stays where it was — the loss hits a wall set by too little reading. So spending all the budget on one ingredient wastes it. The best split sits where one more rupee buys the same drop in loss whether it is spent on parameters or on tokens. That balance point moves out steadily as the budget grows, which is why it can be predicted before the expensive run starts.

Two models, one budget.

	Gopher (2021)	Chinchilla (2022)
parameters N	280 billion	70 billion
tokens D	300 billion	1.4 trillion
tokens per parameter	about 1.07	20
compute 6ND	$5.04\times10^{23}$	$5.88\times10^{23}$
fitted loss	1.9933	1.9366 — better
cost to use	4 times the parameters on every reply	cheaper for every reply, forever

Rule of thumb. $C\approx6ND$ , and for the best loss per rupee of training, $D\approx20N$ . Models that will answer millions of questions are often trained on even more tokens than that: a smaller model costs less every time it is used.

Trap

A scaling law predicts the average loss on text like the training text — not any particular skill. Abilities such as arithmetic can seem to appear suddenly while the loss falls smoothly. And the law is a fitted curve: trust it within the range of sizes it was fitted on, and less far beyond.

The realization

$\begin{gathered}C\approx6ND\\ L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}\end{gathered}$

Training cost is six operations per parameter per token, and loss falls as a sum of power laws in size and data — straight lines on log–log axes. With a fixed budget, balance the two: roughly twenty tokens for every parameter.

Quick check

About how many operations does it take to train a 1-billion-parameter model on 20 billion tokens?

Pause & predict

You double the compute budget and keep the rule $D=20N$ . By how much should the model grow?

Pause & predict

In the fitted formula, you make the model 10 times bigger but keep the same number of tokens. What happens to the loss?

If you want the algebra · 2 proofs, step by step

Prove it · training costs about 6 operations per parameter per token

Claim. For a network whose work is mostly matrix products, one training step on one token costs about $6N$ operations, so reading $D$ tokens costs $C\approx6ND$ .

Forward. In

\mathbf y=\mathbf xW

, every weight

w_{ij}

is used once: one multiply

x_iw_{ij}

and one add into

y_j

. That is

2N

operations per token. Attention's own

n^2

scores add a little more; for today's models it is a small share.

Backward. Rule A of Unit 15 sends the blame back to the inputs,

\partial\mathcal L/\partial\mathbf x=(\partial\mathcal L/\partial\mathbf y)W^{\mathsf T}

: another multiply–add per weight,

2N

. Rule B finds each weight's slope,

\partial\mathcal L/\partial w_{ij}=x_i\,\partial\mathcal L/\partial y_j

, added up:

2N

more. So the backward pass costs about twice the forward. It is still one sweep that reuses every shared piece, as Unit 7 showed — just two jobs per weight instead of one.

Total per token:

2N+2N+2N=6N

. For

D

tokens:

C\approx6ND

. ∎ Chinchilla:

6\times7\times10^{10}\times1.4\times10^{12}=5.88\times10^{23}

Prove it · the best split grows both N and D as powers of the budget

Claim. Minimise $L=E+AN^{-\alpha}+BD^{-\beta}$ with $6ND=C$ . At the best split the two losses are in the ratio $\alpha AN^{-\alpha}=\beta BD^{-\beta}$ , and $N^{*}\propto C^{\beta/(\alpha+\beta)}$ , $D^{*}\propto C^{\alpha/(\alpha+\beta)}$ .

Put

D=C/(6N)

into

L

, and set the slope with respect to

N

to zero. Since

D^{-\beta}=(6N/C)^{\beta}

grows like

N^{\beta}

\begin{aligned}&\frac{dL}{dN}=-\alpha AN^{-\alpha-1}\\ &\qquad+\beta BD^{-\beta}N^{-1}=0\\ &\Rightarrow\ \alpha AN^{-\alpha}=\beta BD^{-\beta}.\end{aligned}

Spend one more unit on parameters or on tokens: at the best split both buy the same drop in loss.

Use

D=C/(6N)

again:

\alpha AN^{-\alpha}=\beta B(6N)^{\beta}C^{-\beta}

, so

N^{\alpha+\beta}\propto C^{\beta}

, which gives

N^{*}\propto C^{\beta/(\alpha+\beta)}

, and

D^{*}=C/(6N^{*})\propto C^{\alpha/(\alpha+\beta)}

. ∎ With

\alpha=0.34

\beta=0.28

N^{*}\propto C^{0.45}

and

D^{*}\propto C^{0.55}

— both grow, the tokens a little faster. The rule

D=20N

is the simpler choice of exponents 0.5 and 0.5.

In one sentence: Training costs about 6ND operations, and loss falls as a floor plus two power laws — one in parameters, one in tokens — so for a fixed budget, like a school choosing between teachers and books, the best split grows both together, about twenty tokens for every parameter.

Teaching it to follow instructions

Imagine this

A new apprentice joins a busy dhaba. He already knows how to chop, fry and knead — he learned that at home, over years. What he does not know is how this kitchen answers an order. So for a few weeks he watches the head cook, and copies how each plate should look when a customer asks for it.

The question. A model trained on internet text only ever learned to continue text. How does it learn to answer?

Ask a freshly trained model "What is the capital of India?" and it may well reply "What is the capital of Nepal? What is the capital of Bhutan?" — it has seen many lists of quiz questions, and continuing the list is a perfectly likely thing to do. It knows the answer; it does not know that you wanted it.

Supervised fine-tuning (SFT) is the apprentice's weeks of copying. People write thousands of pairs: an instruction, and a good answer. The model keeps training with exactly the same next-token loss as before — with one change. Each conversation is laid out as one token list, with special marker tokens for who is speaking (a chat template):

<user>capitalofIndia?<assistant>NewDelhi.

and only the answer tokens count in the loss. This is the loss mask: the question's tokens are read, but the model is never graded on guessing them.

By hand. The model gives the three answer tokens the probabilities 0.5 (New), 0.8 (Delhi) and 0.9 (the full stop).

$\begin{aligned}-\ln0.5&\approx0.6931\\ -\ln0.8&\approx0.2231\\ -\ln0.9&\approx0.1054\\ \text{SFT loss}&=\frac{-\ln(0.5\times0.8\times0.9)}{3}\\ &=\frac{-\ln0.36}{3}\approx\frac{1.0217}{3}\\ &\approx0.3406\end{aligned}$

The question's tokens had probabilities 0.02 (capital), 0.6 (of), 0.1 (India) and 0.7 (?). Without the mask the loss would be the average over all seven tokens, 1.1577 — most of it from guessing what the user would type. The model would spend its effort learning to write users' questions. (In real training the "end" token after the answer is counted too, so the model also learns when to stop.)

Try: With the mask on, the loss is 0.3406, the average of the three pink surprises. Switch the mask off: the loss jumps to 1.1577, pulled up by "capital", which the model could never guess. Click "Delhi" and drag its probability to 0.99: only the masked-in loss moves.

p of Delhi0.8

Why does this work?

The model already has the knowledge and the language; pretraining put them there. Fine-tuning changes a habit: after the marker <assistant>, the most likely continuation becomes a helpful answer instead of another question. Changing a habit needs far less data than learning a language — thousands of good examples, not trillions of tokens — which is why a small, carefully written set of answers can turn a text-continuer into an assistant.

Pretraining and fine-tuning, side by side.

	pretraining	supervised fine-tuning
data	trillions of tokens of text from anywhere	thousands to a million conversations written or checked by people
loss	next-token surprise on every token	the same surprise, on the answer tokens only
it learns	language, facts, ways of reasoning	the manners of an assistant: answer, follow instructions, stop
cost	enormous	small

Rule of thumb. Same loss, same model, a mask, and much better data. What goes into the answers is what comes out of the assistant: a few thousand excellent examples beat many mediocre ones.

Trap

Fine-tuning teaches the style of good answers by copying them. It cannot tell the model which of two decent answers people prefer — the more helpful one, the more honest one, the kinder one. For that we need people's choices, not only their examples. That is the next three sections.

The realization

$\mathcal L_{\text{SFT}}=-\frac{1}{|A|}\sum_{t\in A}\ln P\big(w_t\mid w_{\lt t}\big)$

$A$ is the set of answer positions. The whole conversation is read, but only the assistant's tokens are graded — the apprentice copies the master's plates, not the customers' orders.

Pause & predict

You fine-tune a model on chat data but forget the loss mask, so every token of every conversation is graded. What extra thing does the model learn?

Quick check

A short chat: the user's two tokens get probabilities 0.01 and 0.2; the answer's two tokens get 0.9 and 0.6. What is the SFT loss (with the mask)?

If you want the algebra · 1 proof, step by step

Prove it · a masked token sends no "predict me" blame

Claim. Write the SFT loss with a mask $m_t$ (1 for answer tokens, 0 otherwise): $\mathcal L=-\frac{1}{\sum_tm_t}\sum_tm_t\ln q_{t,w_t}$ , where $q_t$ is the softmax at position $t$ . Then the blame on the scores at position $t$ is $\frac{m_t}{\sum_sm_s}(\mathbf q_t-\mathbf y_t)$ : zero at every masked position.

Only the term for position

t

contains the scores

\mathbf z_t

at that position. Its slope is the one from Unit 15: for

-\ln

softmax, it is prediction minus truth,

\mathbf q_t-\mathbf y_t

\mathbf y_t

is 1 at the true token and 0 elsewhere.

That term is multiplied by

m_t/\sum_sm_s

, so

\frac{\partial\mathcal L}{\partial\mathbf z_t}=\frac{m_t}{\sum_sm_s}\,(\mathbf q_t-\mathbf y_t),

which is the zero vector when

m_t=0

. ∎ The question's tokens still matter: the answer's guesses attend to them, and blame flows back into their vectors through attention. What the mask removes is only the push to predict them.

In one sentence: Supervised fine-tuning is an apprentice copying the master's plates — the same next-token loss on written examples of good answers, with a mask so that only the assistant's tokens are graded.

LoRA: fine-tune a giant by changing a thin slice

Imagine this

Clip-on lenses for a phone camera: wide-angle, macro, fish-eye. The phone stays exactly the same. A small lens clipped on top changes what it does, you can swap lenses in seconds, and ten of them fit in a pocket.

The question. To teach a 7-billion-number model a new job — answer like a doctor, write in Hindi, follow a company's style — must we change all 7 billion numbers?

Full fine-tuning updates every weight: $W\to W+\Delta W$ . For one $4\,096\times4\,096$ matrix, $\Delta W$ is $16\,777\,216$ numbers — and Adam (Unit 11) keeps two more notebooks for every weight it trains, so the memory for training is several times the model.

LoRA, low-rank adaptation (Hu and colleagues, 2021), freezes $W$ and learns the change as a product of two thin matrices, with a small rank $r$ :

$\begin{gathered}\Delta W=BA\\ B\ \text{is}\ d\times r,\quad A\ \text{is}\ r\times d\end{gathered}$

With $d=4\,096$ and $r=8$ : $B$ and $A$ together hold $2\times4\,096\times8=65\,536$ numbers, against $16\,777\,216$ — just 0.39 %.

By hand. Take $d=3$ , $r=1$ : $B=(1,\ 0,\ 2)^{\mathsf T}$ and $A=(1,\ -1,\ 0.5)$ . Then

$\begin{aligned}\Delta W=BA&=\begin{pmatrix}1\\0\\2\end{pmatrix}\begin{pmatrix}1&-1&0.5\end{pmatrix}\\ &=\begin{pmatrix}1&-1&0.5\\0&0&0\\2&-2&1\end{pmatrix}.\end{aligned}$

Look at the rows: the first is $(1,-1,0.5)$ , the second is 0 times it, the third is 2 times it. Every row is a multiple of one row, so $\Delta W$ has rank 1 (Unit 5). Six numbers describe nine; at full size, 65 536 describe 16.8 million.

How training starts. $A$ starts small and random, and $B$ starts at zero. So at the first step $BA=0$ and the model is exactly the original. Training then grows the change from nothing.

Where the adapters go. A common choice puts one on the query matrix and one on the value matrix of every layer. In a 7-billion-class model (32 layers, $d=4\,096$ , $r=8$ ): $32\times2\times65\,536=4\,194\,304$ trainable numbers — about 0.06 % of the model. One base model can carry many adapters and switch between them; or an adapter can be merged, $W'=W+BA$ , so using it costs nothing extra.

Try: Start at rank 1: one smooth wave already holds 63.7 % of the change (measured by its squared size, σ₁² out of the sum of all σ²). Press ▶ build it layer by layer: each rank adds one more wave, and by rank 4 the right field looks almost like the left (98.1 %). Watch the read-out: the error left over is exactly the size of the singular values you dropped — Eckart–Young, at work. At rank 8, LoRA stores 384 numbers instead of 576 here; at d = 4 096 it would be 65 536 instead of 16 777 216.

drag the picture to orbit

rank r1

show what is still missing

Why does this work?

Because the change needed to adapt a trained model is nearly low-rank: a new job mostly needs a few new directions, not a whole new matrix. The SVD of Unit 5 writes any change as a sum of rank-1 layers, biggest first, and Eckart–Young says the best rank- $r$ copy keeps exactly the first $r$ layers, with an error equal to the size of the dropped singular values. When those fall fast, a tiny $r$ captures almost everything. Hu and colleagues found ranks as small as 1 to 8 were often enough.

Full fine-tuning against LoRA (one $4\,096\times4\,096$ matrix, $r=8$ ).

	full fine-tuning	LoRA
numbers trained	16 777 216	65 536 (0.39 %)
memory while training	weights + slopes + Adam's two notebooks for all of them	slopes and notebooks for the thin slice only
stored per new job	a whole new copy of the model	a small file of $B$ and $A$
speed when used	normal	normal, once merged into $W$
quality	the reference	close for most jobs

Rule of thumb. A LoRA adapter on a $d\times d$ matrix has $2dr$ numbers. Start with $r=8$ on the query and value matrices; raise $r$ only if the job is far from what the model already does.

Trap

LoRA does not make the model you run any smaller: the full frozen $W$ is still there, and after merging $W+BA$ is a full matrix again. What LoRA shrinks is what you train and what you store for each new job.

The realization

$\begin{gathered}W'=W+BA\\ \operatorname{rank}(BA)\le r\\ \text{numbers}=2dr\ \ll\ d^2\end{gathered}$

Freeze the giant, and learn its change as a thin product of two matrices. It is Eckart–Young as a training method: bet that the change lives in a few directions, and learn only those.

Quick check

A $1\,024\times1\,024$ weight matrix gets a LoRA adapter with $r=4$ . How many numbers does the adapter train?

Pause & predict

LoRA starts with $B=0$ and a random $A$ . Before the first training step, what does the fine-tuned model output?

Pause & predict

A target change has singular values $6,\ 3.6,\ 2.2,\ 1.3,\ \dots$ and a LoRA of rank 2 is trained as well as possible. Which singular values decide the error that is left?

If you want the algebra · 2 proofs, step by step

Prove it · a product BA with inner size r has rank at most r

Claim. If $B$ is $d\times r$ and $A$ is $r\times d$ , then $\operatorname{rank}(BA)\le r$ .

Column

j

BA

B

times column

j

A

(BA)_{:,j}=\sum_{k=1}^{r}a_{kj}\,\mathbf b_k

, a combination of the

r

columns

\mathbf b_1,\dots,\mathbf b_r

B

. Matrix times vector = a mix of the matrix's columns (Unit 2's span).

So every column of

BA

lies in the span of

r

vectors, a space of dimension at most

r

. The rank — the dimension of the column space — is at most

r

. ∎ In the tiny example,

r=1

: every column of

\Delta W

is a multiple of

B=(1,0,2)^{\mathsf T}

, and every row a multiple of

A

Prove it · the error of the best rank-r copy is the size of the dropped singular values

Claim. Let $\Delta W=\sum_{i}\sigma_i\mathbf u_i\mathbf v_i^{\mathsf T}$ be its SVD (orthonormal $\mathbf u_i$ , orthonormal $\mathbf v_i$ , $\sigma_1\ge\sigma_2\ge\dots$ ). Keeping the first $r$ layers leaves an error of size $\|\Delta W-\Delta W_r\|_F=\sqrt{\sigma_{r+1}^2+\sigma_{r+2}^2+\cdots}$ , and by Eckart–Young no rank- $r$ matrix does better.

The error is the sum of the dropped layers:

E=\sum_{i\gt r}\sigma_i\mathbf u_i\mathbf v_i^{\mathsf T}

. Its squared size is the sum of all its squared entries,

\|E\|_F^2=\operatorname{tr}(E^{\mathsf T}E)

. The Frobenius size of Unit 5: add up the squares of all entries.

Multiply out:

E^{\mathsf T}E=\sum_{i,j\gt r}\sigma_i\sigma_j\,\mathbf v_i(\mathbf u_i^{\mathsf T}\mathbf u_j)\mathbf v_j^{\mathsf T}

. Since

\mathbf u_i^{\mathsf T}\mathbf u_j

is 1 when

i=j

and 0 otherwise, only

\sum_{i\gt r}\sigma_i^2\mathbf v_i\mathbf v_i^{\mathsf T}

is left, and each

\operatorname{tr}(\mathbf v_i\mathbf v_i^{\mathsf T})=\|\mathbf v_i\|^2=1

\|E\|_F^2=\sum_{i\gt r}\sigma_i^2.

∎ That no other rank-

r

matrix beats this is Eckart–Young (Unit 5). In the widget:

r=2

leaves

\sqrt{2.2^2+1.3^2+\dots}\approx2.754

In one sentence: LoRA freezes the giant and learns its change as a thin product BA of rank r — a clip-on lens of 2dr numbers instead of d² — and it works because the change a new job needs lives in a few directions, exactly the ones Eckart–Young says a rank-r copy keeps.

A reward from human choices

Imagine this

A cricket selector cannot give every batter an exact score out of 100. But put two batters in the nets side by side, and she can tell you which one is better. Watch enough pairs, and a ranking appears — even a rating for every player, like the ratings chess players carry.

The question. How can a machine learn what people like, when nobody can write down a formula for "a good answer"?

For one prompt the model writes two answers, and a person picks the better one. This is easy for people — much easier than writing a perfect answer, and much more consistent than giving marks out of 10. Collect many such choices. Then give every answer a hidden score, its reward $r$ , and ask the scores to explain the choices.

The Bradley–Terry model. The chance that answer A is preferred to answer B depends only on the difference of their rewards:

$\begin{gathered}P(\text{A beats B})=\sigma(r_A-r_B)\\ \sigma(x)=\frac{1}{1+e^{-x}}\end{gathered}$

$\sigma$ is the logistic curve: softmax of Unit 14 with just two choices. Equal rewards give 0.5; a big lead gives nearly 1.

By hand. $r_A=1.5$ , $r_B=0.5$ .

The prediction: $\sigma(1.5-0.5)=\sigma(1)\approx0.7311$ .
A person prefers A. The loss is the surprise of that choice: $-\ln0.7311\approx0.3133$ .
The nudge. The slope of the loss pushes the winner's reward up and the loser's down, by the same amount: the step size times $1-\sigma(1)\approx0.2689$ . With a step size of 1: $r_A\to1.7689$ , $r_B\to0.2311$ , and now $\sigma(1.5379)\approx0.823$ .
If the scores were equal, the prediction would be 0.5 and the loss $\ln2\approx0.6931$ : a coin toss.

The reward model. In practice the rewards come from a network: a copy of the LLM whose last layer is replaced by a single number, the score of a whole answer. It is trained on hundreds of thousands of people's choices with exactly this loss.

Try: The worked example is loaded: A at 1.5, B at 0.5, P = 0.731. Press left is better: A rises to 1.769, B falls to 0.231, and P becomes 0.823. Press 20 votes from a crowd a few times and watch the four answers sort themselves. Then press add 10 to every reward: every dot moves, but no probability changes.

step size1

Why does this work?

The nudge is $1-\sigma$ : the chance the model gave to the choice that did not happen. When the rewards already agree strongly with the person, $\sigma$ is near 1 and the nudge is tiny. When they disagree — the model had the order backwards — the nudge is big. So the rewards learn mostly from their mistakes, just like the "prediction − truth" of Unit 15. Over many choices, answers people keep picking drift up, answers they keep rejecting drift down, and the gaps settle where the predicted chances match how often people actually agree.

Three ways to collect human judgement.

	write a good answer	give marks out of 10	pick the better of two
effort for people	high	low	low
consistency	varies with the writer	poor: one person's 7 is another's 5	good: comparing is easy
teaches	what a good answer looks like (§9)	a noisy score	which answer is better — a reward

Rule of thumb. Ask people to compare, not to score. The Bradley–Terry loss $-\ln\sigma(r_w-r_l)$ turns comparisons into rewards, with a nudge of $1-\sigma$ to the winner and the loser.

Trap

Only differences of rewards mean anything. Add 10 to every reward and every probability stays the same. So a reward of 3.2 on its own says nothing; "3.2 against 1.1" does.

The realization

$\begin{gathered}P(y_w\succ y_l)=\sigma(r_w-r_l)\\ \mathcal L=-\ln\sigma(r_w-r_l)\\ \frac{\partial\mathcal L}{\partial r_w}=-(1-\sigma)\end{gathered}$

Give every answer a hidden score, predict each choice with the logistic of the score gap, and nudge the winner up and the loser down by how surprised you were — a selector who rates players only by comparing two at a time.

Pause & predict

A reward model gives answer A 1.5 and answer B 0.5. A bug adds 10 to every reward it outputs. What is the new chance that A is preferred to B?

Quick check

The reward model had the order backwards: $r_w=0$ for the answer the person preferred, $r_l=2$ for the other. With step size 1, how much does each reward move?

If you want the algebra · 1 proof, step by step

Prove it · the Bradley–Terry nudge is 1 − σ, and adding a constant changes nothing

Claim. For $\mathcal L=-\ln\sigma(r_w-r_l)$ : $\partial\mathcal L/\partial r_w=-(1-\sigma)$ and $\partial\mathcal L/\partial r_l=+(1-\sigma)$ , with $\sigma=\sigma(r_w-r_l)$ . And the loss is unchanged if every reward gets the same constant added.

The logistic has a neat slope:

\sigma'(x)=\sigma(x)(1-\sigma(x))

. So

\begin{aligned}&\frac{d}{dx}\big(-\ln\sigma(x)\big)\\ &=-\frac{\sigma(x)(1-\sigma(x))}{\sigma(x)}\\ &=-(1-\sigma(x)).\end{aligned}

\sigma(x)=1/(1+e^{-x})

, so

\sigma'(x)=e^{-x}/(1+e^{-x})^2=\sigma(x)\,(1-\sigma(x))

With

x=r_w-r_l

\partial x/\partial r_w=1

and

\partial x/\partial r_l=-1

. By the chain rule the two slopes are

-(1-\sigma)

and

+(1-\sigma)

: a gradient step raises

r_w

and lowers

r_l

by the same amount. Worked example:

1-\sigma(1)\approx0.2689

Adding

c

to every reward gives

(r_w+c)-(r_l+c)=r_w-r_l

: the same

x

, the same probability, the same loss. ∎ So rewards are only defined up to a shift; papers often fix the average reward at 0.

In one sentence: Like a selector comparing two batters at a time, a reward model learns from people's choices — the chance one answer beats another is the logistic of their reward gap, and each choice nudges the winner up and the loser down by how surprised the model was.

Chasing the reward on a leash

Imagine this

On Makar Sankranti the sky fills with kites. The wind pulls each kite up and away; the string holds it near the hand that flies it. Cut the string, and the kite races off wherever the wind blows — and comes down in a tree. Keep the string too short, and the kite never rises at all. Flying well is the balance between the two.

The question. We now have a reward model that scores answers. Why not simply train the LLM to get the highest reward it possibly can?

Because the reward model is not perfect. It learned from a limited set of people's choices, so it has blind spots. Push an LLM hard against it, and the LLM finds strange answers the reward model happens to love — very long answers, flattering phrases, favourite words repeated — that no person would actually prefer. This is reward hacking: the kite with a cut string.

The leash. Keep the tuned model $\pi$ close to the sensible model it started from, the reference $\pi_{\text{ref}}$ (the model after §9). Measure "close" with the KL divergence of Unit 14, and subtract it from the reward:

$\begin{aligned}\text{objective}=\ &\mathbb E_{y\sim\pi}\big[r(y)\big]\\ &-\beta\,\mathrm{KL}\big(\pi\,\|\,\pi_{\text{ref}}\big).\end{aligned}$

The first term is the wind: more reward is better. The second is the string: every step away from the old habits costs $\beta$ per unit of KL. This is the objective of RLHF, reinforcement learning from human feedback.

The beautiful answer. For a fixed prompt, the best possible $\pi$ has a closed form:

$\begin{gathered}\pi^{*}(y)=\frac{\pi_{\text{ref}}(y)\,e^{r(y)/\beta}}{Z}\\ Z=\sum_{y}\pi_{\text{ref}}(y)\,e^{r(y)/\beta}\end{gathered}$

Read it aloud. Start from the old habits, $\pi_{\text{ref}}(y)$ . Multiply each answer by $e^{r/\beta}$ : high reward, big boost. Divide by the total so everything adds up to 1. It is softmax once again — the rewards act as scores, $\beta$ as the temperature of §4, and the old habits as a head start.

By hand. Three answers: the reference gives them $(0.5,\ 0.3,\ 0.2)$ , and their rewards are $(0,\ 1,\ 2)$ . Take $\beta=1$ .

The boosts:

$\begin{aligned}0.5e^{0}&=0.5\\ 0.3e^{1}&\approx0.8155\\ 0.2e^{2}&\approx1.4778\\ Z&\approx2.7933\\ \pi^{*}&\approx(0.1790,\,0.2919,\,0.5291)\\ \mathbb E[r]&\approx1.3501\\ \mathrm{KL}&\approx0.3228\\ \text{objective}&\approx1.3501-0.3228\\ &=1.0272=\beta\ln Z\end{aligned}$

Now change the string. $\beta=0.5$ : $(0.0367,\ 0.1626,\ 0.8008)$ — the kite flies far towards the best answer. $\beta=2$ : $(0.3250,\ 0.3215,\ 0.3534)$ . $\beta=10$ : $(0.4648,\ 0.3082,\ 0.2271)$ — hugging the old habits. As $\beta\to0$ all the probability goes to the highest-reward answer; as $\beta\to\infty$ , $\pi^{*}$ becomes $\pi_{\text{ref}}$ .

In practice a real model has far too many possible answers to list, so this cannot be computed directly. RLHF uses a reinforcement-learning method called PPO: the model writes answers, the reward model scores them, and the weights are nudged, step by step, towards more reward and less KL. The closed form is the target that PPO is walking towards.

Try: At β = 1 you see the worked example: (0.1790, 0.2919, 0.5291), reward 1.3501, KL 0.3228. Drag β down to 0.5: answer C takes 0.801. Drag it up to 10: the orange bars nearly match the blue ones. Now switch on the reward model has a bug: a flattering answer D gets reward 6 from the reward model, though people rate it −2. At β = 1 it grabs 0.593 of the probability and the true quality falls to −0.638; at β = 5 it gets only 0.028.

β (string)1

the reward model has a bug

Why does this work?

The KL term charges for every bit of surprise between the new habits and the old ones, so the cheapest way to earn reward is to move probability between answers the reference already considered reasonable. An answer the reference would almost never write can only be reached by paying a lot of KL — which is exactly where a flawed reward model's blind spots hide. The string does not stop the kite from rising; it stops it from flying into places no sensible model would go.

Three lengths of string (the example above).

	small β (long string)	medium β	large β (short string)
*π at β = 0.5 / 1 / 10**	(0.037, 0.163, 0.801)	(0.179, 0.292, 0.529)	(0.465, 0.308, 0.227)
reward	high	good	little gain
distance (KL)	far from the reference	moderate	almost none
risk	reward hacking	the balance	barely changes

Rule of thumb. The best tuned model is the reference, reweighted by $e^{r/\beta}$ . Choose $\beta$ so the model improves on the reward and on what people actually think — watch both, because the reward alone will keep rising even as the answers get worse.

Trap

A larger $\beta$ means a shorter string, not a weaker one: the KL costs more, so the model stays closer to its old habits. Small $\beta$ is the long, loose string that lets reward hacking happen.

The realization

$\begin{gathered}\max_{\pi}\ \mathbb E_{y\sim\pi}\big[r(y)\big]-\beta\,\mathrm{KL}\big(\pi\,\|\,\pi_{\text{ref}}\big)\\ \pi^{*}(y)=\frac{\pi_{\text{ref}}(y)\,e^{r(y)/\beta}}{Z}\\ \text{best value}=\beta\ln Z\end{gathered}$

Chase the reward, pay for every step away from the old habits, and the best policy is the old habits reweighted by exponentiated reward — a kite as high as the wind can lift it on a string of length set by $\beta$ .

Pause & predict

You make $\beta$ very small — almost no leash. Where does $\pi^{*}$ put its probability?

Quick check

Two answers, reference $(0.5,\ 0.5)$ , rewards $(0,\ \ln3)$ , $\beta=1$ . What is $\pi^{*}$ ?

Pause & predict

One possible answer has reference probability exactly 0 — the old model would never write it — but a huge reward. What does $\pi^{*}$ give it?

If you want the algebra · 1 proof, step by step

Prove it · the best leashed policy is π_ref · e^(r/β) / Z, and its value is β ln Z

Claim. Over all probability lists $\pi$ on the answers, $J(\pi)=\sum_y\pi(y)r(y)-\beta\,\mathrm{KL}(\pi\,\|\,\pi_{\text{ref}})$ is largest at $\pi^{*}(y)=\pi_{\text{ref}}(y)e^{r(y)/\beta}/Z$ , and there $J=\beta\ln Z$ .

Write both terms as one sum, using

\mathrm{KL}(\pi\|\pi_{\text{ref}})=\sum_y\pi(y)\ln\frac{\pi(y)}{\pi_{\text{ref}}(y)}

, and call the boosted reference

w(y)=\pi_{\text{ref}}(y)\,e^{r(y)/\beta}

\begin{aligned}J&=\sum_y\pi(y)\,r(y)\\ &-\beta\sum_y\pi(y)\ln\frac{\pi(y)}{\pi_{\text{ref}}(y)}\\ &=-\beta\sum_y\pi(y)\ln\frac{\pi(y)}{w(y)}.\end{aligned}

r=\beta\ln e^{r/\beta}

, so the reward folds into the log.

By the definition of

\pi^{*}

w(y)=Z\,\pi^{*}(y)

. Split the log:

\begin{aligned}J&=-\beta\sum_y\pi(y)\ln\frac{\pi(y)}{\pi^{*}(y)}\\ &\quad+\beta\ln Z\\ &=\beta\ln Z-\beta\mathrm{KL}(\pi\|\pi^{*}).\end{aligned}

\sum_y\pi(y)=1

turns

\sum_y\pi(y)\ln Z

into

\ln Z

A KL is never negative, and it is 0 only when the two lists are equal (Unit 14). So

J\le\beta\ln Z

, with equality exactly at

\pi=\pi^{*}

. ∎ The worked example:

Z\approx2.7933

\beta\ln Z\approx1.0272=1.3501-0.3228

In one sentence: Chasing an imperfect reward on its own ends in reward hacking, so RLHF subtracts β times the KL from the reference — and the best policy is then the old habits reweighted by e^(r/β), a kite that rises with the wind but stays on its string.

DPO: skip the reward model

Imagine this

A cook wants to know which of her dishes customers like. She could hire a food critic to score every plate. Or she could notice something simpler: the dishes she has started making more often than she used to are exactly the ones she now thinks are better. Her own change of habits already is a score. She is her own judge.

The question. RLHF needs a reward model, a reference model, the model being tuned, and a delicate reinforcement-learning loop. Is there a simpler way to learn from people's choices?

Turn the closed form around (Rafailov and colleagues, 2023). If $\pi^{*}(y)=\pi_{\text{ref}}(y)e^{r(y)/\beta}/Z$ , then taking logs and rearranging,

$\begin{aligned}r(y)=\ &\beta\ln\frac{\pi^{*}(y)}{\pi_{\text{ref}}(y)}\\ &+\beta\ln Z.\end{aligned}$

The reward is hiding inside the policy: it is $\beta$ times the log of how much more often the model says $y$ than the reference did. Now put this into Bradley–Terry (§11), which only needs the difference $r_w-r_l$ for two answers to the same prompt. The awkward $\beta\ln Z$ is the same for both, so it cancels:

$\begin{aligned}\mathcal L_{\text{DPO}}=-\ln\sigma\Big(\beta\Big[&\ln\frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)}\\ &-\ln\frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}\Big]\Big).\end{aligned}$

No reward model, no reinforcement learning — one supervised-style loss on pairs of answers, trained with ordinary gradient descent. This is direct preference optimisation, DPO.

By hand. $\beta=0.1$ . The reference gave the preferred answer $y_w$ and the rejected answer $y_l$ the probability 0.2 each. After some training, the policy gives $y_w$ 0.3 and $y_l$ 0.1.

$\begin{aligned}\ln\tfrac{0.3}{0.2}&\approx0.4055\\ \ln\tfrac{0.1}{0.2}&\approx-0.6931\\ \text{margin}&=0.1\times(0.4055+0.6931)\\ &\approx0.1099\\ \mathcal L&=-\ln\sigma(0.1099)\approx0.6397\end{aligned}$

At the start, when $\pi=\pi_{\text{ref}}$ , the margin is 0 and the loss is $\ln2\approx0.6931$ : a coin toss. Training has moved it down to 0.6397. Each step pushes $\ln\pi(y_w)$ up and $\ln\pi(y_l)$ down, with the weight $\beta\,\sigma(-\text{margin})\approx0.1\times0.473=0.0473$ : big while the model still gets the pair wrong, small once it gets it right.

Try: At the start both answers sit at the reference, 0.2: margin 0, loss 0.6931. Press worked example: 0.3 and 0.1 give margin 0.1099 and loss 0.6397. Press ▶ train 30 steps and watch the preferred answer rise, the rejected one fall, and the steps shrink as the loss falls. Then press halve both: both bars drop by the same 0.693, and the margin and loss do not change at all.

π(preferred)0.2

π(rejected)0.2

β0.1

Why does this work?

Because the closed form of §12 is a two-way street. Every reward function has one best leashed policy, and every policy is the best leashed policy for some reward function — the one hiding in its log-ratios (up to a constant for each prompt, which never matters). So instead of fitting a reward and then chasing it, DPO fits the policy directly, using the policy's own log-ratios as the reward in Bradley–Terry. The constant $\ln Z$ that we could never compute drops out of every comparison.

RLHF against DPO.

	RLHF (reward model + PPO)	DPO
models during training	policy, reference, reward model (and a value network)	policy and reference
training	write answers, score them, reinforcement-learning steps	one loss on stored pairs, plain gradient descent
stability	delicate: many settings to tune	steady, like fine-tuning
cost	high	about the cost of fine-tuning
new answers during training	yes — it can explore	no — only the stored pairs

Rule of thumb. With a fixed set of people's choices, DPO is the simple, strong first choice. RLHF earns its extra cost when the model should keep writing fresh answers and be scored on them as it learns.

Trap

DPO drops the reward model, not the reference. Every log-ratio needs $\pi_{\text{ref}}(y)$ , so the frozen starting model is still run on every pair — and the leash $\beta$ is still there, built into the loss.

The realization

$\begin{gathered}r(y)=\beta\ln\frac{\pi(y)}{\pi_{\text{ref}}(y)}+\beta\ln Z\\ \mathcal L_{\text{DPO}}=-\ln\sigma\big(r_w-r_l\big)\\ \text{(the }\beta\ln Z\text{ cancels)}\end{gathered}$

The best leashed policy's log-ratios are a reward. Put them into Bradley–Terry and the unknown constant cancels: learn from people's choices with one loss, the model acting as its own judge.

Quick check

At the very start of DPO the policy is a copy of the reference. What is the loss on any pair, whatever $\beta$ ?

Pause & predict

The policy halves the probability of both answers in a pair. What happens to the DPO loss on that pair?

If you want the algebra · 1 proof, step by step

Prove it · in DPO the unknown constant cancels

Claim. If the policy is the best leashed policy for some reward, $\pi(y)=\pi_{\text{ref}}(y)e^{r(y)/\beta}/Z$ , then Bradley–Terry's $r_w-r_l$ equals $\beta\ln\frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)}-\beta\ln\frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}$ , with no $Z$ left.

Take logs and solve for the reward:

\begin{aligned}\ln\pi(y)&=\ln\pi_{\text{ref}}(y)\\ &\quad+\frac{r(y)}{\beta}-\ln Z\\ r(y)&=\beta\ln\frac{\pi(y)}{\pi_{\text{ref}}(y)}\\ &\quad+\beta\ln Z.\end{aligned}

Z

depends on the prompt only, not on which answer

y

we look at.

Both answers of a pair share the prompt, so they share

\beta\ln Z

, and it cancels in the difference:

\begin{aligned}r_w-r_l&=\beta\ln\frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)}\\ &\quad-\beta\ln\frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}.\end{aligned}

Put this into

-\ln\sigma(r_w-r_l)

(§11) and you have the DPO loss. ∎ The slope (with §11's

1-\sigma(x)=\sigma(-x)

\partial\mathcal L/\partial\ln\pi(y_w)=-\beta\,\sigma(-m)

and

\partial\mathcal L/\partial\ln\pi(y_l)=+\beta\,\sigma(-m)

, where

m

is the margin. In the worked example

\beta\sigma(-0.1099)\approx0.0473

In one sentence: DPO turns the leash's closed form around — a policy's log-ratio against the reference is its hidden reward — so plugging it into Bradley–Terry cancels the unknown constant and trains on people's choices with one plain loss, the model acting as its own judge.

What to carry forward

The whole unit fits on thirteen cards. Each has one picture you should be able to draw from memory.

The loop

Read everything, give every token a probability, pick one, append, repeat. One run of the model, one new token.

A bet on every word

Multiply the probabilities of the true tokens; take logs; average: loss 0.7675. Perplexity

e^{\text{loss}}

= a die with 2.15 faces.

The tower

12Ld² in the floors (4d² attention, 8d² feed-forward) + Vd in the dictionary. GPT-2 small: 124 439 808.

The thermostat

Divide the scores by T: cold makes the favourite a dictator (greedy), hot makes every token equal. The order never changes.

The nucleus

Keep the fewest top tokens holding p of the probability, renormalise, draw. It adapts: 5 colours, 1 capital.

The KV notebook

The mask freezes the past: keep every key and value. n rows of work, not n(n+1)/2 — paid for in memory, 2·L·d·b bytes per token.

Clock hands

Pair

i

turns

10\,000^{-2i/d}

radians per position; the score sees only the gap. Squeeze positions to stretch the context.

The budget

C ≈ 6ND. Loss = floor + a power law in N + a power law in D. Balance them: about 20 tokens per parameter.

The apprentice

Same next-token loss on good answers, with a mask: only the assistant's tokens are graded.

The clip-on lens

Freeze W, learn ΔW = BA of rank r: 2dr numbers instead of d². Eckart–Young says the few top directions carry the change.

The selector

P(A beats B) = σ(r_A − r_B). Each choice nudges the winner up and the loser down by 1 − σ. Only gaps matter.

The kite

Maximise reward − β·KL. The best policy is π_ref · e^(r/β), normalised. Larger β, shorter string.

Its own judge

A policy's log-ratio is its hidden reward. Put it into Bradley–Terry, the constant cancels: DPO, one plain loss.

Where every tool came from. Almost nothing in this unit was new mathematics. It was the course's own tools, put to work at enormous scale.

tool	where you met it	what it does inside an LLM
dot product	Unit 3	every attention score, every next-token score
SVD, low rank, Eckart–Young	Unit 5	LoRA: fine-tune with a thin slice (§10)
backprop: one backward sweep, two jobs per weight	Unit 7, Unit 15	the 6 in $C\approx6ND$ (§8)
Adam	Unit 11	the optimiser of every training run (§8)
softmax, surprise, cross-entropy, KL	Unit 14	next-token probabilities, the loss, temperature, the leash (§1, §2, §4, §12)
prediction − truth	Unit 15	the blame at every output, masked in SFT (§9)
chain of guesses, perplexity, subwords	Unit 16	the chain rule of the loop, the die (§1, §2)
temperature, greedy, beam	Unit 17	how the next token is picked (§4, §5)
attention, mask, rotary positions	Unit 18	the engine; the cache; long contexts (§3, §6, §7)

Where this goes next. In this unit a model wrote one token at a time, each one picked from a probability list. Unit 20 asks the same question for pictures: how can a machine create an image that has never existed? It will generate a whole picture at once — by starting from pure noise and removing it, step by small step.

The realization

$\begin{gathered}\text{train: }\min\,-\tfrac1n\textstyle\sum\ln P(w_t\mid w_{\lt t})\\ \text{tune: }\max\,\mathbb E[r]-\beta\mathrm{KL}(\pi\|\pi_{\text{ref}})\end{gathered}$

An LLM is a next-token guesser trained on the average surprise of trillions of tokens, written out one token at a time with a sampler, made fast with a cache and long with clocks, and turned into an assistant by copying good answers and then chasing people's preferences on a KL leash.

In one sentence: Inside every chatbot is one loop — a probability for every next token, a sampler that picks one, a notebook of the past — trained to be unsurprised by human text and then taught, with a thin low-rank patch, a reward from people's choices and a leash, to be helpful.

Practice arena — sixteen problems, solved in full

Sixteen problems, in the order of the unit: next-token probabilities, scoring a sentence and reading a perplexity backwards, counting a model's parameters and its experts, temperature and the two tail cuts, the cache's work and memory, clock speeds, a training budget, the loss mask, a LoRA adapter, a preference, the leash and DPO. The tags say which are easy and which are hard. Every number here was checked by machine.

Three habits do most of the work. Work in logs: products of probabilities become sums, and powers become multiplications. Write the shapes and counts first: $d\times d$ , $2dr$ , $2\cdot L\cdot d\cdot b$ . And check that every list of probabilities adds up to 1 before you use it.

Problem 1easynext token

After "the train is", a model gives four tokens the scores late 2, early 0, full 1, banana −2. (a) Find the four probabilities. (b) Which token does greedy decoding write? (c) The model adds 3 to every score. What is the probability of "late" now?

What this tests. One move of the write loop (§1): scores → softmax → pick. Plan. Exponentiate, add, divide. For (c), remember what softmax ignores.

Show the full solution

Step 1 — (a) exponentials.

e^{2}\approx7.389

e^{0}=1

e^{1}\approx2.718

e^{-2}\approx0.135

; total

\approx11.243

Step 2 — (a) divide. late

7.389/11.243\approx0.6572

, early

\approx0.0889

, full

\approx0.2418

, banana

\approx0.0120

. They add up to 1.

Step 3 — (b). Greedy takes the largest probability: "late".

Step 4 — (c). Adding 3 multiplies every exponential, and the total, by

e^{3}

; it cancels. "late" stays at

0.6572

answers at a glance: (a) $\approx(0.6572,\ 0.0889,\ 0.2418,\ 0.0120)$ . (b) late. (c) still $0.6572$ .

Remember

Softmax sees only the gaps between scores. Shifting every score by the same amount changes nothing.

Problem 2easyscoring a sentence

A model reads a four-token sentence and gives the true tokens the probabilities 0.5, 0.2, 0.9 and 0.4. Find (a) the probability of the sentence, (b) its log-likelihood, (c) the loss in nats per token, (d) the perplexity, (e) the loss in bits per token.

What this tests. The five ways to report the same thing (§2). Plan. Multiply; take logs; average and flip the sign; exponentiate; divide by $\ln2$ .

Show the full solution

Step 1 — (a).

0.5\times0.2\times0.9\times0.4=0.036

Step 2 — (b).

\ln0.5+\ln0.2+\ln0.9+\ln0.4\approx-0.6931-1.6094-0.1054-0.9163=-3.3242

(=\ln0.036)

Step 3 — (c).

3.3242/4\approx0.8311

nats per token.

Step 4 — (d).

e^{0.8311}\approx2.2957

, the same as

(1/0.036)^{1/4}

Step 5 — (e).

0.8311/0.6931\approx1.1990

bits per token.

answers at a glance: (a) 0.036. (b) $\approx-3.3242$ . (c) $\approx0.8311$ . (d) $\approx2.2957$ . (e) $\approx1.1990$ .

Remember

Perplexity is one over the geometric mean of the probabilities: the one probability that, used for every token, gives the same product.

Problem 3mediumreverse-engineer

A three-token sentence has perplexity exactly 2.5. The model gave the first two true tokens 0.5 and 0.8. What probability did it give the third?

What this tests. Reading perplexity backwards (§2). Plan. Perplexity $=(\text{product})^{-1/n}$ , so the product is $\text{perplexity}^{-n}$ .

Show the full solution

Step 1 — the product.

2.5=(p_1p_2p_3)^{-1/3}

gives

p_1p_2p_3=2.5^{-3}=1/15.625=0.064

Step 2 — divide out the known ones.

p_3=0.064/(0.5\times0.8)=0.064/0.4=0.16

Step 3 — check.

(0.5\times0.8\times0.16)^{-1/3}=0.064^{-1/3}=2.5

. ✓

answer at a glance: $p_3=0.16$ .

Remember

Perplexity $\mathrm{PP}$ over $n$ tokens means the sentence probability is $\mathrm{PP}^{-n}$ .

Problem 4mediumcounting parameters

GPT-2 medium has $d=1\,024$ , $L=24$ blocks, a vocabulary of $V=50\,257$ and a learned position table for 1 024 positions. Using $12d^2+13d$ per block and a final layer norm of $2d$ , find (a) the numbers in one block, (b) the total with a tied output layer, (c) the total without tying.

What this tests. Where the parameters live (§3). Plan. Block, times $L$ ; then the tables; then decide how many token tables there are.

Show the full solution

Step 1 — (a).

12\times1\,024^2=12\,582\,912

and

13\times1\,024=13\,312

12\,596\,224

per block.

Step 2 — the blocks.

24\times12\,596\,224=302\,309\,376

Step 3 — the tables. Tokens:

50\,257\times1\,024=51\,463\,168

. Positions:

1\,024\times1\,024=1\,048\,576

. Final layer norm:

2\,048

Step 4 — (b).

302\,309\,376+51\,463\,168+1\,048\,576+2\,048=354\,823\,168

, about 355 million.

Step 5 — (c). A separate output table adds another

51\,463\,168

406\,286\,336

answers at a glance: (a) 12 596 224. (b) 354 823 168. (c) 406 286 336.

Remember

Parameters $\approx12Ld^2+Vd$ ; tying the output to the input table saves one whole $V\times d$ table.

Problem 5mediummixture of experts

A mixture-of-experts layer with $d=2\,048$ has 16 experts, each a feed-forward room of $8d^2$ numbers, and a router of $d\times16$ numbers that keeps the top 2. For one token the router's first four scores are $(1.2,\ 3.0,\ 0.4,\ 2.3)$ , and all the others are below 1. Find (a) which experts run and their weights, (b) the numbers stored in the experts, (c) the expert numbers used for this token, (d) the router's numbers.

What this tests. The router and "stored against used" (§3). Plan. Pick the two biggest scores and softmax just those two; then count.

Show the full solution

Step 1 — (a). The two biggest scores are 3.0 (expert 2) and 2.3 (expert 4). Softmax of the two:

e^{3}/(e^{3}+e^{2.3})=1/(1+e^{-0.7})\approx0.6682

, and

0.3318

Step 2 — (b). One expert:

8\times2\,048^2=33\,554\,432

. Sixteen:

536\,870\,912

Step 3 — (c). Two experts run:

67\,108\,864

, one eighth of the stored experts.

Step 4 — (d).

2\,048\times16=32\,768

answers at a glance: (a) experts 2 and 4, weights $\approx0.6682$ and $0.3318$ . (b) 536 870 912. (c) 67 108 864. (d) 32 768.

Remember

With top-2 routing over $E$ experts, memory grows with $E$ but the work per token only with 2.

Problem 6mediumtemperature

Scores $(3,\ 1,\ 0)$ . Find (a) the probabilities at $T=1$ , (b) at $T=2$ . (c) At what temperature is the first token exactly twice as likely as the second?

What this tests. Temperature divides the scores (§4), and the ratio rule $p_i/p_j=e^{(z_i-z_j)/T}$ . Plan. For (c) set the ratio to 2 and solve for $T$ .

Show the full solution

Step 1 — (a).

e^{3}\approx20.086

e^{1}\approx2.718

e^{0}=1

; total

23.804

\approx(0.8438,\ 0.1142,\ 0.0420)

Step 2 — (b). Scores

\div2

(1.5,\ 0.5,\ 0)

e^{1.5}\approx4.482

e^{0.5}\approx1.649

, 1; total

7.130

\approx(0.6285,\ 0.2312,\ 0.1402)

Step 3 — (c). The gap is

3-1=2

, so

e^{2/T}=2

2/T=\ln2

T=2/\ln2\approx2.885

answers at a glance: (a) $\approx(0.8438,\ 0.1142,\ 0.0420)$ . (b) $\approx(0.6285,\ 0.2312,\ 0.1402)$ . (c) $T\approx2.885$ .

Remember

Temperature raises every ratio to the power $1/T$ . To hit a target ratio, solve $e^{\text{gap}/T}=\text{ratio}$ .

Problem 7mediumtop-k and top-p

List A: $(0.35,\ 0.25,\ 0.20,\ 0.10,\ 0.06,\ 0.04)$ . List B: $(0.85,\ 0.07,\ 0.05,\ 0.03)$ . (a) Top-k 2 on A: the renormalised list. (b) Top-p 0.75 on A: how many tokens, and the renormalised list. (c) Top-p 0.8 on B. (d) Top-k 2 on B.

What this tests. The two cuts, on a flat-ish list and a peaked one (§5). Plan. Running totals for top-p; divide the kept ones by their sum.

Show the full solution

Step 1 — (a). Keep 0.35 and 0.25 (sum 0.6):

0.35/0.6\approx0.5833

0.25/0.6\approx0.4167

Step 2 — (b). Running totals 0.35, 0.60, 0.80: the total first reaches 0.75 at the third token. Keep 3 (sum 0.8):

(0.4375,\ 0.3125,\ 0.25)

Step 3 — (c). The first token alone is 0.85 ≥ 0.8: keep 1, with probability 1.

Step 4 — (d). Keep 0.85 and 0.07 (sum 0.92):

\approx(0.9239,\ 0.0761)

answers at a glance: (a) $\approx(0.5833,\ 0.4167)$ . (b) 3 tokens, $(0.4375,\ 0.3125,\ 0.25)$ . (c) 1 token, probability 1. (d) $\approx(0.9239,\ 0.0761)$ .

Remember

Top-p keeps many tokens when the model is unsure and few when it is sure; top-k keeps the same number either way.

Problem 8mediumKV cache memory

A model has $L=40$ layers, $d=5\,120$ , 40 attention heads and 16-bit numbers (2 bytes). (a) How many bytes of keys and values does it keep per token with multi-head attention? (b) For 8 192 tokens, in GiB? (c) With grouped-query attention using 8 key-value heads? (d) How much memory does the switch save?

What this tests. The cache bill $2\cdot L\cdot n_{kv}\cdot d_{\text{head}}\cdot b$ (§6). Plan. With all 40 heads, $n_{kv}d_{\text{head}}=d$ .

Show the full solution

Step 1 — (a).

2\times40\times5\,120\times2=819\,200

bytes (800 KiB) per token.

Step 2 — (b).

819\,200\times8\,192=6\,710\,886\,400

bytes

=6\,710\,886\,400/2^{30}=6.25

GiB.

Step 3 — (c). 8 key-value heads instead of 40:

6.25\times8/40=1.25

GiB.

Step 4 — (d).

6.25-1.25=5

GiB saved, for every conversation of that length.

answers at a glance: (a) 819 200 bytes. (b) 6.25 GiB. (c) 1.25 GiB. (d) 5 GiB.

Remember

The cache grows with layers × key-value heads × head size × bytes × tokens; sharing key-value heads divides it directly.

Problem 9mediumwork without a cache

(a) Show that writing $n$ tokens without a cache computes $n(n+1)/2$ rows of keys and values in each layer. (b) Apply it to $n=4\,096$ , and compare with the cache. (c) What is the shortest text for which the work without a cache passes one million rows?

What this tests. The triangle against the line (§6), and turning a formula around. Plan. Run $t$ processes $t$ tokens; add them up; then solve an inequality by trying values near the square root.

Show the full solution

Step 1 — (a). Run

t

reads all

t

tokens and recomputes a row for each:

1+2+\dots+n

. Pair the first and last terms, the second and second-last, …: each pair makes

n+1

, and there are

n/2

pairs, so the total is

n(n+1)/2

Step 2 — (b).

4\,096\times4\,097/2=8\,390\,656

rows without the cache;

4\,096

with it:

(n+1)/2=2\,048.5

times more.

Step 3 — (c). We need

n(n+1)/2\gt10^{6}

, so

n

is about

\sqrt{2\times10^{6}}\approx1\,414

. Check:

1\,413\times1\,414/2=998\,991

(not yet) and

1\,414\times1\,415/2=1\,000\,405

(passed).

answers at a glance: (a) $n(n+1)/2$ . (b) 8 390 656 against 4 096, 2 048.5 times more. (c) $n=1\,414$ .

Remember

Without a cache the work grows like $n^2/2$ ; with it, like $n$ . The longer the text, the bigger the win.

Problem 10hardrotary clocks

Rotary positions with $d=16$ and base 10 000. (a) List the eight speeds $\theta_i=10\,000^{-2i/16}$ . (b) How many positions does the slowest hand need for one full turn? (c) The model was trained on 4 096 tokens. What is the largest gap angle its slowest hand saw? (d) It must now read 16 384 tokens. With interpolation, what is the scale, and what gap angle does the slowest hand show for a gap of 12 000?

What this tests. The clock speeds and interpolation (§7). Plan. $10\,000^{-i/8}=10^{-i/2}$ ; a full turn is $2\pi/\theta$ ; angles are gap $\times\,\theta$ (times the scale).

Show the full solution

Step 1 — (a).

10^{-i/2}

for

i=0,\dots,7

1,\ 0.3162,\ 0.1,\ 0.03162,\ 0.01,\ 0.003162,\ 0.001,\ 0.0003162

Step 2 — (b).

2\pi/0.0003162\approx19\,869

positions.

Step 3 — (c).

4\,096\times0.0003162\approx1.295

radians.

Step 4 — (d). Scale

4\,096/16\,384=0.25

. The gap 12 000 acts like

3\,000

: angle

3\,000\times0.0003162\approx0.949

radians, inside the trained range. (Without interpolation it would be

3.795

radians, far outside.)

answers at a glance: (a) $10^{-i/2}$ : 1 … 0.0003162. (b) ≈ 19 869. (c) ≈ 1.295 rad. (d) scale 0.25; ≈ 0.949 rad (not 3.795).

Remember

The slow hands decide how far a model can see. Squeezing positions keeps their angles inside what training showed them.

Problem 11hardcompute budget

You have a training budget of $C=6\times10^{22}$ operations. (a) With $C=6ND$ and the rule $D=20N$ , find $N$ and $D$ . (b) Use the fitted law $L=1.69+406.4/N^{0.34}+410.7/D^{0.28}$ to find its loss. (c) A colleague prefers a 100-billion-parameter model on the same budget. How many tokens can it read, and what loss does the law predict?

What this tests. $6ND$ , the 20-to-1 rule, and why balance wins (§8). Plan. $C=120N^2$ ; then plug into the law, term by term.

Show the full solution

Step 1 — (a).

N=\sqrt{C/120}=\sqrt{5\times10^{20}}\approx2.236\times10^{10}

(22.4 billion);

D=20N\approx4.472\times10^{11}

(447 billion tokens).

Step 2 — (b).

406.4/N^{0.34}

and

410.7/D^{0.28}

add to about 0.348, so

L\approx2.0376

Step 3 — (c).

D=C/(6N)=6\times10^{22}/(6\times10^{11})=10^{11}

: only 100 billion tokens, 1 per parameter. The law gives

L\approx2.1056

— worse, although the model is more than four times bigger.

answers at a glance: (a) $N\approx22.4$ billion, $D\approx447$ billion. (b) $\approx2.0376$ . (c) $10^{11}$ tokens; $L\approx2.1056$ .

Remember

For a fixed budget, a smaller model that reads more usually beats a bigger model that reads less.

Problem 12easythe loss mask

In one fine-tuning chat, the user's three tokens get the probabilities 0.05, 0.3 and 0.6, and the assistant's four answer tokens get 0.7, 0.9, 0.95 and 0.6. (a) What is the SFT loss (with the mask)? (b) What would the loss be over all seven tokens? (c) Which single token contributes the most surprise to (b)?

What this tests. Only the answer is graded (§9). Plan. Surprise $-\ln p$ for each; average the ones that count.

Show the full solution

Step 1 — the answer's surprises.

-\ln0.7\approx0.3567

-\ln0.9\approx0.1054

-\ln0.95\approx0.0513

-\ln0.6\approx0.5108

; sum

\approx1.0242

Step 2 — (a).

1.0242/4\approx0.2560

Step 3 — (b). The user's:

-\ln0.05\approx2.9957

-\ln0.3\approx1.2040

-\ln0.6\approx0.5108

. All seven:

(1.0242+4.7105)/7\approx0.8192

Step 4 — (c). The user's first token, 0.05: surprise

2.9957

— a token the model is never asked to predict in SFT.

answers at a glance: (a) $\approx0.2560$ . (b) $\approx0.8192$ . (c) the user's first token (2.9957).

Remember

The mask keeps the loss about the answer. Without it, the hardest-to-guess user tokens would dominate.

Problem 13mediumLoRA

(a) Compute $BA$ for $B=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}$ and $A=\begin{pmatrix}1&2&0\\0&1&-1\end{pmatrix}$ , and find its rank. (b) For a $d\times d$ matrix, for which ranks $r$ does LoRA store fewer numbers than the full change? (c) A model has 80 layers with $d=8\,192$ . Adapters of rank 16 go on the query and value matrices of every layer. How many numbers are trained?

What this tests. $\operatorname{rank}(BA)\le r$ and the count $2dr$ (§10). Plan. Multiply; look for a row that is a combination of the others; then compare $2dr$ with $d^2$ .

Show the full solution

Step 1 — (a).

BA=\begin{pmatrix}1&2&0\\0&1&-1\\1&3&-1\end{pmatrix}

. Row 3 = row 1 + row 2, and rows 1 and 2 are not multiples of each other: rank 2 (=

r

Step 2 — (b).

2dr\lt d^2

exactly when

r\lt d/2

. Here

d=3

r=2

12\gt9

— this tiny adapter is not worth it. At

d=4\,096

any

r\lt2\,048

saves.

Step 3 — (c). One adapter:

2\times8\,192\times16=262\,144

. Two per layer, 80 layers:

262\,144\times160=41\,943\,040

answers at a glance: (a) rows (1, 2, 0), (0, 1, −1), (1, 3, −1); rank 2. (b) $r\lt d/2$ . (c) 41 943 040.

Remember

A rank- $r$ adapter on a $d\times d$ matrix costs $2dr$ numbers; the saving is huge only because $r$ is tiny compared with $d$ .

Problem 14mediumBradley–Terry

A reward model scores the answer a person preferred at $r_w=0.8$ and the rejected one at $r_l=1.2$ . (a) What chance did it give the person's choice? (b) What is the loss? (c) With step size 0.5, find the two new rewards. (d) What is the new chance?

What this tests. The preference loss and its nudge $1-\sigma$ (§11). Plan. $\sigma$ of the gap; minus its log; then move both rewards by $\eta(1-\sigma)$ .

Show the full solution

Step 1 — (a).

\sigma(0.8-1.2)=\sigma(-0.4)=1/(1+e^{0.4})\approx0.4013

: the model had it slightly backwards.

Step 2 — (b).

-\ln0.4013\approx0.9130

Step 3 — (c). Nudge

0.5\times(1-0.4013)\approx0.2993

r_w\to1.0993

r_l\to0.9007

Step 4 — (d). New gap

\approx0.1987

\sigma(0.1987)\approx0.5495

. One step turned the order around.

answers at a glance: (a) $\approx0.4013$ . (b) $\approx0.9130$ . (c) $r_w\approx1.0993$ , $r_l\approx0.9007$ . (d) $\approx0.5495$ .

Remember

The nudge is the probability the model gave to the choice that did not happen: large when it was wrong, small when it was right.

Problem 15hardthe leash

Three answers: reference $\pi_{\text{ref}}=(0.6,\ 0.3,\ 0.1)$ , rewards $(0,\ 1,\ 3)$ . (a) Find $\pi^{*}$ for $\beta=1$ . (b) Find the expected reward, the KL and the objective, and check that the objective equals $\beta\ln Z$ . (c) Find $\pi^{*}$ for $\beta=2$ . Which answer's share changes most between the two?

What this tests. The closed form $\pi^{*}\propto\pi_{\text{ref}}e^{r/\beta}$ (§12). Plan. Boost each answer, add the boosts to get $Z$ , divide.

Show the full solution

Step 1 — (a). Boosts:

0.6

0.3e\approx0.8155

0.1e^{3}\approx2.0086

Z\approx3.4240

\pi^{*}\approx(0.1752,\ 0.2382,\ 0.5866)

Step 2 — (b).

\mathbb E[r]\approx0.2382\times1+0.5866\times3\approx1.9980

\mathrm{KL}=\sum\pi^{*}\ln(\pi^{*}/\pi_{\text{ref}})\approx0.7672

. Objective

\approx1.9980-0.7672=1.2308

, and

\ln Z=\ln3.4240\approx1.2308

. ✓

Step 3 — (c). Boosts at

\beta=2

0.6

0.3e^{0.5}\approx0.4946

0.1e^{1.5}\approx0.4482

Z\approx1.5428

\pi^{*}\approx(0.3889,\ 0.3206,\ 0.2905)

Step 4 — which moved most. The third answer: 0.5866 at

\beta=1

against 0.2905 at

\beta=2

. A shorter string holds the high-reward, low-reference answer back.

answers at a glance: (a) $\approx(0.1752,\ 0.2382,\ 0.5866)$ . (b) $\approx1.9980$ , $0.7672$ , $1.2308=\ln Z$ . (c) $\approx(0.3889,\ 0.3206,\ 0.2905)$ ; the third answer.

Remember

$\pi^{*}$ is softmax of $r/\beta$ with the reference as a head start; the best value of the objective is $\beta\ln Z$ .

Problem 16mediumDPO

With $\beta=0.2$ , the reference gives the preferred answer 0.1 and the rejected answer 0.3. After training, the policy gives them 0.2 and 0.15. (a) Find the two log-ratios. (b) Find the margin and the DPO loss. (c) What was the loss before training? (d) How strongly does the next step push, $\beta\,\sigma(-\text{margin})$ ?

What this tests. DPO's loss from probabilities (§13). Plan. $\ln(\pi/\pi_{\text{ref}})$ for each; subtract; multiply by $\beta$ ; then $-\ln\sigma$ .

Show the full solution

Step 1 — (a). Preferred:

\ln(0.2/0.1)=\ln2\approx0.6931

. Rejected:

\ln(0.15/0.3)=\ln0.5\approx-0.6931

Step 2 — (b). Gap

\approx1.3863

, margin

0.2\times1.3863\approx0.2773

. Loss

-\ln\sigma(0.2773)\approx0.5641

Step 3 — (c). Before training

\pi=\pi_{\text{ref}}

: margin 0, loss

\ln2\approx0.6931

Step 4 — (d).

\sigma(-0.2773)\approx0.4311

, so the push is

0.2\times0.4311\approx0.0862

: the preferred answer's log-probability up, the rejected one's down.

answers at a glance: (a) $\approx0.6931$ and $\approx-0.6931$ . (b) margin $\approx0.2773$ , loss $\approx0.5641$ . (c) $\ln2\approx0.6931$ . (d) $\approx0.0862$ .

Remember

DPO compares how far each answer moved from the reference. The absolute probabilities do not matter — only the two log-ratios.

Next up

Unit 20 · From Noise to Pictures: VAEs and Diffusion →

An LLM writes one token at a time from a probability list. How can a machine draw a whole picture that has never existed? Squeeze pictures into a small cloud of codes, or bury them in noise and learn to dig them back out, one small step at a time.

← Unit 18 · Attention and Transformers · All units