Unit 14 · Thinking in Probabilities — The Math Behind the Machine

1

Why a machine needs probabilities

Imagine this

It is the monsoon. Two weather forecasters work in your town.

The first one says "rain" every single morning. The second one says things like "70% chance of rain" or "20% chance of rain".

You check the second one's record. On the days she said 70%, it rained on about 7 days out of 10. Now her numbers mean something. A farmer can decide whether to sow today. You can decide whether to carry an umbrella. The first forecaster is never quite wrong, but he is never useful.

Every unit so far gave crisp answers. Solve $A\mathbf x=\mathbf b$ : here is $\mathbf x$ . Project onto a line: here is the shadow. Unit 13's support vector machine says " $+1$ " or " $-1$ ".

But look at how the SVM says it. A point deep inside the blue group gets " $+1$ ". A point that only just grazes the street also gets " $+1$ ". Same answer, same confident voice. The machine cannot say "probably blue, but I am not sure".

The world is full of such "probably". Will it rain? Is this patient sick? Is this photo a cat? The honest answer is a number between 0 and 1. So we need a language for uncertainty. That language is probability, and this whole unit teaches it.

Start with one word. A random variable is a rule that turns an uncertain outcome into a number. Toss a coin and write 1 for heads, 0 for tails. Roll two dice and write their sum. Measure tomorrow's rainfall in millimetres. Each time, you cannot know the number in advance. But you can say how likely each number is.

Here is a machine that makes a random variable. It is called a Galton board. A bead drops in at the top and meets a peg. It bounces left or right, like a coin toss. Then it meets another peg, and another — twelve rows of them. At the bottom it falls into a slot. The slot number is simply how many times it bounced right.

Where will one bead land? Nobody can say. But drop two thousand, and something surprising happens.

Try: Drop 1 bead a few times — each lands somewhere new. Then drop 2000 and watch a bell rise out of the chaos, hugging the dashed outline. Now push the peg bias to the right: the whole bell slides, but it keeps its shape.

rows of pegs12

peg bias0.5

—

Why a bell? Count the routes. To reach slot 0 a bead must bounce left twelve times in a row: there is only one such route. To reach the middle slot, 6, it needs six lefts and six rights in any order, and there are 924 different orders. Out of all $2^{12}=4096$ routes, 924 end in the middle. So about 23% of the beads land there, and almost none at the edges.

No bead knows about bells. Each one just flips a coin at each peg. The shape comes from counting. That is the first big idea of this unit: a single outcome is unpredictable, but the pattern of many outcomes is not. The pattern is called a distribution, and it is the thing we can know.

The realization

$P(X=k)=\frac{\text{beads in slot }k}{\text{all the beads}}$

A random variable $X$ turns an uncertain outcome into a number. You cannot know the next value of $X$ . You can know its distribution: how the chances are spread over the possible values. The shape of the pile is the knowledge.

Pause & predict

A forecaster said "70% chance of rain" on 100 different days. When is she doing her job well?

The road ahead. The unit has four acts.

Uncertainty has a shape (§1–§3): distributions, their balance point and spread, and Bayes' rule for turning evidence around.
The Gaussian (§4–§6): the bell curve, why it keeps appearing, what it looks like in many dimensions (Unit 12's ellipse!), and how a computer draws samples from it.
Learning = the least surprising world (§7–§9): likelihood, fitting by maximum likelihood, and the discovery that least squares was maximum likelihood all along.
Scores into probabilities (§10–§13): softmax, surprise and entropy, cross-entropy and KL, and a classifier that trains itself with the cleanest gradient in machine learning.

In one sentence: A random variable turns an uncertain outcome into a number, and although one outcome cannot be predicted, the distribution of many outcomes can — the shape of the pile is the knowledge.

2

Distributions: where the probability lives

Imagine this

You are playing Ludo with two dice. You need a total of 7 to capture a piece. Your cousin needs a 12.

Who should be more hopeful? A 12 needs double six: one way. A 7 can come as 1+6, 2+5, 3+4, 4+3, 5+2 or 6+1: six ways. You are six times as likely to get your number.

Listing every total with its chance is exactly what a distribution is.

Two dice give 36 equally likely pairs. Count how many pairs make each total:

total	2	3	4	5	6	7	8	9	10	11	12
ways	1	2	3	4	5	6	5	4	3	2	1
chance	1/36	2/36	3/36	4/36	5/36	6/36	5/36	4/36	3/36	2/36	1/36

This list is the probability mass function, or pmf, written $p(x)=P(X=x)$ . "Mass" is a good word: think of each total carrying a small weight of probability. All the weights add up to 1, because some total always happens.

The balance point. Put the weights on a see-saw plank, each at its own total. Where must the pivot go for the plank to balance? At 7. That balance point is the expected value, or mean:

\begin{aligned}\mathbb E[X]&=\sum_x x\,p(x)\\ &=2\cdot\tfrac1{36}+3\cdot\tfrac2{36}+\dots+12\cdot\tfrac1{36}\\ &=7.\end{aligned}

For one die, every face has weight $\tfrac16$ , and the balance point is $\tfrac{1+2+3+4+5+6}{6}=3.5$ . No die ever shows 3.5. The expected value is not a value you expect to see. It is where the weights balance — the long-run average.

The spread. The mean says where the pile sits. We also want how wide it is. Take each value's distance from the mean, square it, and average with the same weights. That is the variance:

\operatorname{Var}[X]=\mathbb E\big[(X-\mu)^2\big]=\sum_x (x-\mu)^2\,p(x).

One die: $\operatorname{Var}=\tfrac{35}{12}\approx2.917$ . Two dice: $\tfrac{35}{6}\approx5.833$ — exactly double, because the two dice wobble independently and independent spreads add. The square root, $\sigma=\sqrt{\operatorname{Var}}$ , is the standard deviation. It is back in the same units as the totals: about 2.42 for two dice.

Try: Drag the pivot left and right and watch the plank tip. Find where it levels. Then switch to 3 dice: the pile gets rounder and wider. Press ▶ roll to throw real dice and watch the rough outline of the rolls creep onto the exact bars.

drag the gold pivot

—

When the values are not countable. Now think of a school bus. It is usually about 5 minutes late. What is the chance it is late by exactly 5 minutes — not 5.01, not 4.999, but exactly 5.000000…? There are endlessly many possible times, so the chance of any single one is zero.

So for a quantity that can take any value — a time, a height, a rainfall — we stop asking about single values. We ask about windows: "late by between 4 and 6 minutes". The answer is an area under a curve called the probability density, $p(x)$ . The whole area under the curve is 1.

Here is the part that surprises everyone. The height of a density is not a probability. It is probability per minute. If a bus is always within a few seconds of 5 minutes late, all the area is squeezed into a tiny window, and the curve must shoot up to keep the area 1. A bell with a spread of 0.1 minutes peaks at about $3.989$ — far above 1, and perfectly legal.

Try: Drag the two window handles to shade "between 4 and 6 minutes". Then slide the spread down to 0.1 and watch the peak shoot past 1 — the readout of total area never moves. Finally press exactly 5: a window of zero width holds zero probability.

drag the window handles

spread σ (minutes)1

—

Watch out — the trap

"The density at 5 is 3.99, so the chance of 5 is 3.99." No: a chance can never be more than 1. A density height only turns into a chance after you multiply by a width. A height of 3.99 over a window 0.01 minutes wide gives a chance of about 0.04.

The realization

$\begin{aligned}&\text{discrete: }P(X=x)=p(x),\\ &\qquad\textstyle\sum_x p(x)=1;\\ &\text{continuous: }P(a\le X\le b)\\ &\qquad=\int_a^b p(x)\,dx,\ \ \textstyle\int p=1.\end{aligned}$

Probability lives in a mass for countable outcomes and in an area for smooth ones. The mean is the balance point of that mass. The variance is the average squared distance from it.

Pause & predict

A game pays you the total of two dice in rupees. Over many games, what do you earn per game on average, and what is the spread (variance) of one game's payout?

Pause & predict

A density curve reaches a height of 3.99 at $x=5$ . Is something wrong?

If you want the algebra · 2 proofs, step by step

Prove it · the mean and variance of one die

Claim. A fair die has $\mathbb E[X]=\tfrac72=3.5$ and $\operatorname{Var}[X]=\tfrac{35}{12}\approx2.917$ . Two independent dice have mean 7 and variance $\tfrac{35}6$ .

1

Each face has weight

\tfrac16

:

\begin{aligned}\mathbb E[X]&=\tfrac16(1+2+\dots+6)\\ &=\tfrac{21}{6}=3.5.\end{aligned}

2

The squared distances from 3.5 are

6.25,\,2.25,\,0.25,\,0.25,\,2.25,\,6.25

, which add to

17.5

. Average them:

\operatorname{Var}[X]=\tfrac{17.5}{6}=\tfrac{35}{12}\approx2.917.

3

For a sum, means always add:

3.5+3.5=7

. Variances add when the two parts are independent (the cross term

2\,\mathbb E[(X_1-\mu_1)(X_2-\mu_2)]

is zero):

\begin{aligned}&\operatorname{Var}[X_1+X_2]\\ &\quad=\tfrac{35}{12}+\tfrac{35}{12}=\tfrac{35}{6}\approx5.833.\end{aligned}

∎ Check against the table:

\sum (x-7)^2p(x)=\tfrac{2(25\cdot1+16\cdot2+9\cdot3+4\cdot4+1\cdot5)}{36}=\tfrac{210}{36}=\tfrac{35}6

.

Prove it · the shortcut Var = E[X²] − (E X)²

Claim. $\operatorname{Var}[X]=\mathbb E[X^2]-\mu^2$ , where $\mu=\mathbb E[X]$ . It saves subtracting $\mu$ from every value.

1

Open the square inside the average:

\begin{aligned}&\mathbb E[(X-\mu)^2]\\ &\quad=\mathbb E[X^2-2\mu X+\mu^2].\end{aligned}

2

An average of a sum is the sum of the averages, and constants come out:

\begin{aligned}&\mathbb E[X^2]-2\mu\,\mathbb E[X]+\mu^2\\ &\quad=\mathbb E[X^2]-2\mu^2+\mu^2\\ &\quad=\mathbb E[X^2]-\mu^2.\end{aligned}

∎ One die:

\mathbb E[X^2]=\tfrac{1+4+9+16+25+36}{6}=\tfrac{91}6

, so

\operatorname{Var}=\tfrac{91}6-\tfrac{49}4=\tfrac{182-147}{12}=\tfrac{35}{12}

✓.

In one sentence: A distribution says where the probability lives — as masses that add to 1 or as an area of 1 under a density — and its mean is the balance point while its variance is the average squared distance from it.

3

Bayes: turning evidence around

Imagine this

It is dengue season. You feel a little tired, and the clinic runs a quick blood test. It comes back positive.

The doctor says the test catches 99% of people who have dengue, and only 5% of healthy people get a false alarm. Your heart sinks: surely you are 95% likely to be sick?

Not even close. Once you count properly, the chance is about 17%. This section shows why, and it is one of the most useful ideas in this course.

Say 1% of the people who come in actually have dengue. Picture 10 000 people walking into the clinic.

100 of them are sick. The test catches 99% of them: 99 test positive.
9 900 are healthy. 5% get a false alarm: 495 test positive.

Now you are one of the positives. There are $99+495=594$ positives, and only 99 of them are sick. Your chance of being sick is

\frac{99}{594}=\frac16\approx0.167.

What went wrong with "95%"? The healthy crowd is 99 times bigger than the sick crowd. Even a small false-alarm rate on a huge crowd makes more alarms (495) than the real cases (99). The test is good. It is the starting chance that is small.

Now the same thing in symbols. Read the bar " $\mid$ " as "given". $P(+\mid\text{sick})$ is "the chance of a positive, given that you are sick" — that is the 99%. We want the reverse, $P(\text{sick}\mid+)$ . Bayes' rule turns the question around:

\begin{aligned}P(\text{sick}\mid +)&=\frac{P(+\mid\text{sick})\,P(\text{sick})}{P(+)}\\ &=\frac{0.99\times0.01}{0.0099+0.0495}=\frac16.\end{aligned}

Each piece has a name. $P(\text{sick})=0.01$ is the prior: where you stood before the evidence. $P(+\mid\text{sick})$ is the likelihood: how well "sick" explains the evidence. The answer $P(\text{sick}\mid+)$ is the posterior: where you stand after. The bottom line $P(+)$ just makes the answers add up to 1.

A second test. Suppose you take another, independent test and it is positive again. Today's posterior becomes tomorrow's prior. Start from $\tfrac16$ :

\frac{0.99\times\tfrac16}{0.99\times\tfrac16+0.05\times\tfrac56}\approx0.798.

Two positives take you from 1% to about 80%. Evidence adds up — but it always starts from where you stood.

Try: Look at the zoom: most of the positives are grey (healthy). Now slide the false-alarm rate down to 1% and watch the posterior jump. Then reset and press test again: the positives become the new crowd, and the answer climbs to about 0.8.

how common (prior)1%

catches the sick99%

false-alarm rate5%

—

Watch out — the flip that fools doctors

$P(+\mid\text{sick})$ and $P(\text{sick}\mid+)$ sound alike but are different numbers: 0.99 and 0.167 here. "Most sick people test positive" does not mean "most people who test positive are sick". To go from one to the other you must bring in the prior.

The realization

$\begin{aligned}\underbrace{P(H\mid E)}_{\text{posterior}}&=\frac{\overbrace{P(E\mid H)}^{\text{likelihood}}\;\overbrace{P(H)}^{\text{prior}}}{P(E)}\\ &\propto\text{likelihood}\times\text{prior}\end{aligned}$

Bayes' rule turns "how likely is the evidence, if the idea is true?" into "how likely is the idea, now that I have seen the evidence?". The prior is where you stood before. The posterior is where you stand after, and it becomes the prior for the next piece of evidence.

Pause & predict

Same clinic: 1% are sick, the test catches 99% of them, and gives a false alarm to 5% of the healthy. Out of 10 000 people, how many test positive, and what share of those are sick?

Pause & predict

After one positive test your chance is $\tfrac16$ . A second, independent test is also positive. What do you use as the prior for the second update?

If you want the algebra · 2 proofs, step by step

Prove it · Bayes' rule from one overlap counted two ways

Claim. $P(H\mid E)=\dfrac{P(E\mid H)\,P(H)}{P(E)}$ , with $P(E)=P(E\mid H)P(H)+P(E\mid\text{not }H)P(\text{not }H)$ .

1

"Given

E

" means: keep only the cases where

E

happened, and ask what share of them also have

H

:

P(H\mid E)=\frac{P(H\text{ and }E)}{P(E)}.

In the clinic: of the 594 positives, the 99 who are also sick.

2

The same overlap can be counted from the other side — first pick the

H

cases, then the share of them with

E

:

\begin{aligned}&P(H\text{ and }E)\\ &\quad=P(E\mid H)\,P(H).\end{aligned}

100 sick people × 99% = 99.

3

Put step 2 into step 1. For the bottom, every

E

case either has

H

or does not:

\begin{aligned}P(H\mid E)&=\frac{P(E\mid H)\,P(H)}{P(E)},\\ P(E)&=P(E\mid H)\,P(H)\\ &\quad+P(E\mid\text{not }H)\\ &\qquad\times P(\text{not }H).\end{aligned}

∎ Clinic:

\dfrac{0.99\cdot0.01}{0.99\cdot0.01+0.05\cdot0.99}=\dfrac{0.0099}{0.0594}=\dfrac16

.

Prove it · evidence multiplies the odds

Claim. In odds form, each independent test multiplies the odds of "sick" by the same factor $\dfrac{P(+\mid\text{sick})}{P(+\mid\text{healthy})}=\dfrac{0.99}{0.05}=19.8$ .

1

Write Bayes for "sick" and for "healthy" and divide; the bottom

P(+)

cancels:

\begin{aligned}&\frac{P(\text{sick}\mid+)}{P(\text{healthy}\mid+)}\\ &\quad=\frac{P(+\mid\text{sick})}{P(+\mid\text{healthy})}\\ &\qquad\times\frac{P(\text{sick})}{P(\text{healthy})}.\end{aligned}

2

Start at odds

1:99

. One positive:

19.8:99=1:5

, which is a chance of

\tfrac16

. A second positive multiplies again:

19.8:5=3.96:1

, a chance of

\tfrac{3.96}{4.96}\approx0.798

. ∎ The prior sets the starting odds; each piece of evidence multiplies them. That is why one test cannot overcome a very small prior, and two can.

In one sentence: Bayes' rule turns the evidence around — posterior ∝ likelihood × prior — and because the prior is where you stood before the evidence, a rare disease stays fairly unlikely even after a good test says yes.

4

The bell curve, and why it keeps showing up

Imagine this

Line up all the students of a big school by height. Most are somewhere in the middle. A few are very tall, a few are very short, and nobody is three metres tall.

Draw how many students have each height and you get a hill with gentle slopes on both sides: a bell.

Marks in a board result, errors when you weigh rice on a shop scale, the time a train is late — the same bell shows up again and again. Why?

Think about what makes your height. Hundreds of small things: many genes, food at age five, sleep, a growth spurt in class 8. Each one nudges you a little up or a little down, and mostly independently of the others.

That is the Galton board again. Each peg is one small nudge left or right, and the pile at the bottom is a bell. The same thing happens whenever many small, independent nudges add up. It does not matter what each nudge looks like. Their sum comes out bell-shaped. This fact has a grand name, the central limit theorem, and it is why the bell is everywhere.

The bell has a formula. It looks scary, so let us read it aloud, one step at a time:

p(x)=\frac{1}{\sigma\sqrt{2\pi}}\;\exp\!\Big(-\tfrac12\Big(\frac{x-\mu}{\sigma}\Big)^{2}\Big)

Reading the bell, left to right inside the bracket:

$x-\mu$ : how far $x$ is from the middle $\mu$ .
$\dfrac{x-\mu}{\sigma}$ : that distance, measured in $\sigma$ 's. This number is called the z-score.
Square it: left or right does not matter, only how far.
$\exp(-\tfrac12\,\cdot)$ : at the middle this is 1; far away it dies off very fast.
$\dfrac1{\sigma\sqrt{2\pi}}$ : a fixed number that makes the total area exactly 1.

Two numbers set the whole bell: the middle $\mu$ and the spread $\sigma$ . We write it $\mathcal N(\mu,\sigma^2)$ .

Because the formula only cares about the z-score, every bell is the same bell, stretched and shifted. So one rule works for all of them, the 68–95–99.7 rule: about 68% of the area lies within $1\sigma$ of the middle, about 95% within $2\sigma$ , and about 99.7% within $3\sigma$ .

Example. In a class, heights have $\mu=160$ cm and $\sigma=8$ cm. A student 172 cm tall has $z=\tfrac{172-160}{8}=1.5$ : one and a half spreads above the middle. About 95% of the class is between $160-16=144$ cm and $160+16=176$ cm.

Try: Press ±2σ: 95.4% of the area, whatever $\mu$ and $\sigma$ are. Change $\sigma$ and check — the shaded share never moves. Then drag the probe to 172 cm and read its z-score.

drag the probe

middle μ (cm)160

spread σ (cm)8

—

Now watch the central limit theorem happen. Take a lopsided die: it lands on 1 very often (40%), on 6 often (30%), and rarely on the middle faces. Its pmf looks nothing like a bell — it is a U. Roll it $n$ times and take the average. The widget below draws the exact distribution of that average.

The realization

$\begin{gathered}\mathcal N(\mu,\sigma^2):\ \ p(x)=\frac{1}{\sigma\sqrt{2\pi}}\,e^{-z^2/2},\\ z=\frac{x-\mu}{\sigma}\end{gathered}$

The bell only asks one question: how many $\sigma$ 's from the middle are you? That is why 68–95–99.7 holds for every bell. And it appears whenever many small independent nudges add up — the average of $n$ rolls has spread $\sigma/\sqrt n$ and a bell's shape.

Pause & predict

Board marks in a subject follow a bell with $\mu=60$ and $\sigma=10$ . About what share of students scored between 40 and 80?

Pause & predict

A die is badly lopsided — its own pmf is U-shaped. What does the distribution of the average of 30 rolls look like?

In one sentence: The bell $\mathcal N(\mu,\sigma^2)$ measures everything in $\sigma$ 's from the middle, so 68–95–99.7 holds for every bell, and it shows up whenever many small independent nudges add up.

5

The Gaussian in many dimensions

Imagine this

Now measure two things for every student: height and weight. Plot each student as a dot.

The dots do not make a round blob. Tall students tend to be heavier, so the cloud is a tilted oval. A 180 cm student who weighs 75 kg is ordinary. A 150 cm student who weighs 75 kg is unusual — even though both are the same "distance" from the average in plain centimetres and kilograms.

We need a bell that knows the two measurements move together.

In two dimensions the middle becomes a mean vector $\boldsymbol\mu$ , and the spread becomes the covariance matrix $\Sigma$ — the same matrix you built in Unit 12. Its diagonal holds each measurement's own variance. Its off-diagonal says how much they move together.

The formula is the 1-D bell with one change. Instead of "distance divided by $\sigma$ , squared", we use "distance measured with $\Sigma^{-1}$ ":

\begin{aligned}p(\mathbf x)&=\frac{1}{2\pi\sqrt{\det\Sigma}}\\ &\quad\times\exp\!\Big(-\tfrac12\,(\mathbf x-\boldsymbol\mu)^{\!\top}\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\Big).\end{aligned}

The number $d^2=(\mathbf x-\boldsymbol\mu)^{\!\top}\Sigma^{-1}(\mathbf x-\boldsymbol\mu)$ is a squared distance measured in the data's own ruler. It is called the Mahalanobis distance (after P. C. Mahalanobis, who founded the Indian Statistical Institute). Along the long axis of the cloud the ruler is long, so points count as close. Across the short axis the ruler is short, so the same step counts as far.

The picture. Where is the density equal to some fixed value? Where $d^2$ is fixed. And $\mathbf x^{\!\top}\Sigma^{-1}\mathbf x=\text{constant}$ is an ellipse — the covariance ellipse of Unit 12. It was a contour of a probability mountain all along. Its axes point along the eigenvectors of $\Sigma$ (Unit 4), and their lengths grow like $\sqrt\lambda$ .

Worked example. Take $\boldsymbol\mu=\mathbf 0$ and

\Sigma=\begin{bmatrix}2&1\\1&2\end{bmatrix}.

Its eigenvalues are 3 and 1, with eigenvectors $(1,1)/\sqrt2$ and $(1,-1)/\sqrt2$ . So the ellipse is tilted at 45°, and the long axis is $\sqrt3\approx1.73$ times the short one. $\det\Sigma=4-1=3$ , so the peak of the mountain is $\dfrac{1}{2\pi\sqrt3}\approx0.0919$ .

Now two points, both at ordinary distance $\sqrt2\approx1.414$ from the middle: $(1,1)$ and $(1,-1)$ . Using $\Sigma^{-1}=\tfrac13\begin{bmatrix}2&-1\\-1&2\end{bmatrix}$ : the point $(1,1)$ has $d^2=\tfrac23$ , so $d\approx0.816$ . The point $(1,-1)$ has $d^2=2$ , so $d\approx1.414$ . The first lies along the long axis — ordinary. The second cuts across the short axis — unusual.

Two special cases. If the off-diagonal is zero (correlation $\rho=0$ ), the ellipse's axes line up with the coordinate axes, and if also the two spreads are equal, it is a circle. Then the mountain is simply one bell in $x_1$ times one bell in $x_2$ : the two measurements are independent. And whatever $\Sigma$ is, every straight vertical slice through the mountain is itself a 1-D bell.

Try: Press Σ = [[2,1],[1,2]], then drag the probe to (1, 1) and to (1, −1): the same plain distance, very different Mahalanobis distances. Turn $\rho$ to 0 and the ellipse snaps to the axes. Switch on slice — every slice is a bell. Drag the gold mean to move the whole mountain.

drag the probe or the gold mean · drag elsewhere to orbit

spread σ₁1.41

spread σ₂1.41

correlation ρ0.5

—

The realization

$\begin{gathered}p(\mathbf x)\propto\exp\!\big(-\tfrac12\,d^2\big),\\ d^2=(\mathbf x-\boldsymbol\mu)^{\!\top}\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\end{gathered}$

The many-dimensional bell is the 1-D bell with the distance measured in the data's own ruler. Its contours are ellipses — Unit 12's covariance ellipse was a Gaussian's contour — with axes along the eigenvectors of $\Sigma$ and lengths $\propto\sqrt\lambda$ .

Pause & predict

With $\boldsymbol\mu=\mathbf 0$ and $\Sigma=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ , which point is more unusual: $(1,1)$ or $(1,-1)$ ?

Pause & predict

The correlation is $\rho=0$ but $\sigma_1=2$ and $\sigma_2=1$ . What do the contours look like?

If you want the algebra · 2 proofs, step by step

Prove it · the contours are ellipses along the eigenvectors

Claim. For a symmetric positive-definite $\Sigma=Q\Lambda Q^{\!\top}$ , the set $\mathbf x^{\!\top}\Sigma^{-1}\mathbf x=c^2$ is an ellipse whose axes point along the columns of $Q$ (the eigenvectors), with half-lengths $c\sqrt{\lambda_1}$ and $c\sqrt{\lambda_2}$ .

1

The spectral theorem (Unit 4) gives

\Sigma^{-1}=Q\Lambda^{-1}Q^{\!\top}

. Turn to the eigenvector axes:

\mathbf u=Q^{\!\top}\mathbf x

. Then

\begin{aligned}\mathbf x^{\!\top}\Sigma^{-1}\mathbf x&=\mathbf u^{\!\top}\Lambda^{-1}\mathbf u\\ &=\frac{u_1^2}{\lambda_1}+\frac{u_2^2}{\lambda_2}.\end{aligned}

Q^{\!\top}

is a rotation: it only turns the picture, it does not bend it.

2

Setting this to

c^2

gives

\frac{u_1^2}{(c\sqrt{\lambda_1})^2}+\frac{u_2^2}{(c\sqrt{\lambda_2})^2}=1,

the textbook ellipse with half-axes

c\sqrt{\lambda_1}

and

c\sqrt{\lambda_2}

along the

u_1,u_2

directions — the eigenvectors. ∎ For

\Sigma=\begin{bmatrix}2&1\\1&2\end{bmatrix}

: half-axes

c\sqrt3

along

(1,1)/\sqrt2

and

c

along

(1,-1)/\sqrt2

.

Prove it · the height of the peak

Claim. The 2-D Gaussian peaks at $\dfrac1{2\pi\sqrt{\det\Sigma}}$ . For $\det\Sigma=3$ that is $\approx0.0919$ .

1

In the eigen-coordinates of the previous proof, the density splits into two independent 1-D bells with variances

\lambda_1

and

\lambda_2

:

p=\frac{e^{-u_1^2/2\lambda_1}}{\sqrt{2\pi\lambda_1}}\cdot\frac{e^{-u_2^2/2\lambda_2}}{\sqrt{2\pi\lambda_2}}.

A rotation does not change areas, so this still integrates to 1.

2

At the middle both exponentials are 1, and

\lambda_1\lambda_2=\det\Sigma

(Unit 4: the determinant is the product of the eigenvalues):

\begin{aligned}p_{\max}&=\frac{1}{2\pi\sqrt{\lambda_1\lambda_2}}\\ &=\frac1{2\pi\sqrt{\det\Sigma}}\\ &=\frac1{2\pi\sqrt3}\approx0.0919.\end{aligned}

∎ A bigger determinant means a cloud spread over more area, so the same total volume 1 makes a lower mountain.

In one sentence: A many-dimensional Gaussian is a bell whose distance is measured with $\Sigma^{-1}$ , so its contours are ellipses along the eigenvectors of $\Sigma$ — the covariance ellipse of Unit 12 is a contour of this mountain.

6

Making Gaussian data: stretch a round cloud

Imagine this

You are making a roti. The dough starts as a round ball. Roll it one way and it stretches into an oval. Push it at an angle and the oval tilts.

Every grain of flour moves with the dough. Nothing is added or taken away — the round shape is simply stretched and sheared.

A computer makes tilted Gaussian clouds exactly like this.

Computers are good at one kind of randomness: the round cloud, $\mathcal N(\mathbf 0,I)$ . Each coordinate is an independent standard bell, and the cloud looks the same in every direction.

To get a cloud with mean $\boldsymbol\mu$ and covariance $\Sigma$ , take a round point $\mathbf z$ and push it through a matrix:

\mathbf x=\boldsymbol\mu+L\mathbf z,\qquad\text{where } LL^{\!\top}=\Sigma.

Why does this work? Stretching a cloud by $L$ turns its covariance $I$ into $L\,I\,L^{\!\top}=LL^{\!\top}$ . So we need a "square root" of $\Sigma$ . The Cholesky factor from Unit 4 is exactly that: a lower-triangular $L$ with $LL^{\!\top}=\Sigma$ .

Worked example. For

\Sigma=\begin{bmatrix}4&2\\2&2\end{bmatrix},\quad L=\begin{bmatrix}2&0\\1&1\end{bmatrix}.

Check: $LL^{\!\top}=\begin{bmatrix}2\cdot2&2\cdot1\\1\cdot2&1+1\end{bmatrix}=\begin{bmatrix}4&2\\2&2\end{bmatrix}$ ✓. Read $L$ as a recipe: $x_1=2z_1$ (stretch the first coordinate by 2), and $x_2=z_1+z_2$ (let the second one lean on the first). The lean is what tilts the cloud.

Hold on to this trick. In Unit 20 a picture-making network will need to sample from a Gaussian and still learn through the sampling. It will write $\mathbf x=\boldsymbol\mu+L\mathbf z$ and call it the reparameterisation trick.

Try: Press ▶ stretch to watch the round cloud become the tilted one. Then set $\ell_{21}=0$ : the lean is gone and the ellipse stands straight. Switch to set Σ and type in a covariance — $L$ is computed for you.

ℓ₁₁2

ℓ₂₁ (the lean)1

ℓ₂₂1

—

The realization

$\begin{gathered}\mathbf z\sim\mathcal N(\mathbf 0,I)\\ \Longrightarrow\ \boldsymbol\mu+L\mathbf z\sim\mathcal N(\boldsymbol\mu,\,LL^{\!\top})\end{gathered}$

Every Gaussian is a round cloud, stretched. To sample from $\mathcal N(\boldsymbol\mu,\Sigma)$ , find a square root $L$ of $\Sigma$ (Cholesky), stretch round samples by it, and shift them by $\boldsymbol\mu$ .

Pause & predict

You want samples with covariance $\Sigma=\begin{bmatrix}9&3\\3&2\end{bmatrix}$ . Which $L$ should stretch the round cloud?

If you want the algebra · 2 proofs, step by step

Prove it · stretching by L makes covariance LLᵀ

Claim. If $\mathbf z$ has mean $\mathbf 0$ and covariance $I$ , then $\mathbf x=\boldsymbol\mu+L\mathbf z$ has mean $\boldsymbol\mu$ and covariance $LL^{\!\top}$ .

1

Mean:

\mathbb E[\mathbf x]=\boldsymbol\mu+L\,\mathbb E[\mathbf z]=\boldsymbol\mu

.

2

Covariance is the average of (deviation)(deviation)

^{\!\top}

. The deviation is

\mathbf x-\boldsymbol\mu=L\mathbf z

:

\begin{aligned}\operatorname{Cov}[\mathbf x]&=\mathbb E\big[L\mathbf z\,(L\mathbf z)^{\!\top}\big]\\ &=L\,\mathbb E[\mathbf z\mathbf z^{\!\top}]\,L^{\!\top}\\ &=L\,I\,L^{\!\top}=LL^{\!\top}.\end{aligned}

∎

L

is a fixed matrix, so it moves outside the average. And a sum of Gaussians is Gaussian, so

\mathbf x

is not just any cloud with this covariance — it is exactly

\mathcal N(\boldsymbol\mu,LL^{\!\top})

.

Prove it · Cholesky for a 2 × 2 covariance

Claim. $\begin{bmatrix}a&b\\b&c\end{bmatrix}=LL^{\!\top}$ with $L=\begin{bmatrix}\ell_{11}&0\\\ell_{21}&\ell_{22}\end{bmatrix}$ , $\ell_{11}=\sqrt a$ , $\ell_{21}=b/\ell_{11}$ , $\ell_{22}=\sqrt{c-\ell_{21}^2}$ .

1

Multiply out:

LL^{\!\top}=\begin{bmatrix}\ell_{11}^2&\ell_{11}\ell_{21}\\\ell_{11}\ell_{21}&\ell_{21}^2+\ell_{22}^2\end{bmatrix}.

2

Match entry by entry, top-left first:

\ell_{11}^2=a

, then

\ell_{11}\ell_{21}=b

, then

\ell_{21}^2+\ell_{22}^2=c

. Each equation has one new unknown. ∎

\Sigma=\begin{bmatrix}4&2\\2&2\end{bmatrix}

:

\ell_{11}=2

,

\ell_{21}=1

,

\ell_{22}=\sqrt{2-1}=1

. The square root inside

\ell_{22}

is real exactly when

ac>b^2

— when

\Sigma

is positive definite.

In one sentence: To draw from $\mathcal N(\boldsymbol\mu,\Sigma)$ , stretch a round cloud by a square root $L$ of $\Sigma$ and shift it — $\mathbf x=\boldsymbol\mu+L\mathbf z$ has covariance $LL^{\!\top}=\Sigma$ .

7

Likelihood: turning the question around

Imagine this

Before a gully-cricket match, your friend always tosses his own lucky coin — and he always seems to win. You borrow it and toss it 10 times. It shows 7 heads.

Is the coin fair? You cannot open it up and look. All you have is what you saw. So you ask a different question: which coin would most easily have produced 7 heads in 10?

Say the coin shows heads with chance $p$ . The chance of one particular run with 7 heads and 3 tails is

p^7(1-p)^3.

This one formula can be read in two opposite ways.

Probability fixes the coin and asks about the data: "if $p=0.5$ , how likely are 7 heads?"
Likelihood fixes the data and asks about the coin: "given 7 heads, how well does each $p$ explain it?"

Written as a function of the coin, it is the likelihood $L(p)=p^7(1-p)^3$ . Try a few coins:

coin $p$	0.3	0.5 (fair)	0.7	0.9
$L(p)=p^7(1-p)^3$	0.0000750	0.000977	0.00222	0.000478

The fair coin explains your 7 heads less than half as well as a coin with $p=0.7$ . The best explanation of all is $p=0.7=\tfrac7{10}$ — the share of heads you saw. Choosing the parameter that makes the data most likely is called maximum likelihood, and the winner is written $\hat p$ .

Why we take logs. Look at the numbers: 0.00222 for just ten tosses. For a thousand tosses the likelihood is around $10^{-265}$ , smaller than a computer can store. The fix is to take the logarithm:

\log L(p)=7\log p+3\log(1-p).

Two good things happen. A product of many tiny numbers becomes a sum of ordinary numbers. And because $\log$ only ever goes up, the peak stays in the same place. So we always maximise the log-likelihood instead.

More data, more confidence. With 70 heads in 100 tosses the best coin is still 0.7, but the likelihood curve becomes a sharp spike. Coins like 0.6 or 0.8 now explain the data far worse. The width of the peak tells you how sure the data lets you be.

Try: Drag the white marker across both panels and read how well each coin explains 7 heads in 10. Then press 70 of 100: same peak, much sharper. Press ▶ toss 10 more a few times with the mystery coin and watch the peak settle.

drag the marker

tosses n10

heads h7

—

The realization

$\begin{gathered}L(\theta)=P(\text{data}\mid\theta),\\ \hat\theta=\arg\max_\theta L(\theta)\\ =\arg\max_\theta\,\log L(\theta)\end{gathered}$

Probability goes from the model to the data. Likelihood goes back from the data to the model. Learning by maximum likelihood means choosing the model under which what you saw is least surprising. For a coin, that is simply the share of heads.

Pause & predict

A coin shows 3 heads in 12 tosses. What is the maximum-likelihood coin, and does taking the log of the likelihood move it?

If you want the algebra · 1 proof, step by step

Prove it · the best coin is the share of heads

Claim. With $h$ heads in $n$ tosses, $\log L(p)=h\log p+(n-h)\log(1-p)$ is largest at $\hat p=h/n$ .

1

At a peak the slope is zero. Differentiate:

\frac{d}{dp}\log L=\frac hp-\frac{n-h}{1-p}.

2

Set it to zero and cross-multiply:

\begin{aligned}&h(1-p)=(n-h)p\\ &\Rightarrow\;h=np\;\Rightarrow\;\hat p=\frac hn.\end{aligned}

3

It is a maximum, not a minimum: the second derivative

-\tfrac h{p^2}-\tfrac{n-h}{(1-p)^2}

is negative everywhere. ∎ At

\hat p

the second derivative is

-\tfrac{n}{\hat p(1-\hat p)}

: it grows with

n

. A more curved peak is a sharper peak — more tosses, more confidence.

In one sentence: Likelihood reads $P(\text{data}\mid\theta)$ as a function of the model, maximum likelihood picks the model that makes the data least surprising, and taking logs turns tiny products into sums without moving the peak.

8

Fitting a Gaussian by maximum likelihood

Imagine this

You time how long the chai stall near your college takes to serve a glass, on eight mornings: 2, 4, 4, 4, 5, 5, 7 and 9 minutes.

You want one bell curve that describes this stall. Where should its middle be, and how wide should it be? Try a few bells and ask of each: how well does it explain these eight mornings?

A bell $\mathcal N(\mu,\sigma^2)$ gives each data point a height. The likelihood of the bell is the product of the eight heights. Picture three bad bells:

Too narrow. The bell is a tall spike at 5. The points 2 and 9 sit far out on its tails, with heights almost zero. One near-zero factor ruins the whole product.
Too wide. Every point is covered, but the bell is low everywhere, so every factor is small.
Off-centre. Centred at 3, the points 7 and 9 are pushed onto a tail.

The best bell sits in the middle and is exactly as wide as the data. Maximum likelihood gives a clean answer (the drawer shows why):

\hat\mu=\frac1N\sum_i x_i,\qquad \hat\sigma^2=\frac1N\sum_i(x_i-\hat\mu)^2.

For the chai stall: $\hat\mu=\tfrac{2+4+4+4+5+5+7+9}{8}=\tfrac{40}{8}=5$ minutes. The squared distances from 5 are $9,1,1,1,0,0,4,16$ , which add to 32, so $\hat\sigma^2=\tfrac{32}8=4$ and $\hat\sigma=2$ minutes.

Maximum likelihood gives back the two most ordinary summaries there are: the average and the average squared spread. Nothing mysterious — but now we know why they are the right summaries for a bell.

A small catch. Many calculators divide by $N-1=7$ instead, giving $\tfrac{32}{7}\approx4.571$ . Why? The spread is measured around $\hat\mu$ , which was chosen to sit as close to these very points as possible. Measured from their own centre, the points look a little tighter than they really are. Dividing by $N-1$ corrects that. It is the same "divide by $N$ or $N-1$ " note you met in Unit 12. For large $N$ the difference vanishes.

Try: Press too narrow, too wide and off-centre and watch the bell on the right fail in three different ways, low on the landscape each time. Then press ▶ climb: a ball rolls uphill (gradient ascent, Unit 9) and stops at the peak, $(5,\,2)$ .

drag the ball · drag elsewhere to orbit

middle μ3

spread σ4

—

The realization

$\hat\mu=\bar x,\qquad\hat\sigma^2=\frac1N\sum_i(x_i-\bar x)^2$

The bell that makes your data least surprising sits at the average and is as wide as the average squared spread. Fitting a model is climbing a likelihood landscape to its peak.

Pause & predict

Five measurements: 1, 3, 4, 6, 6. What are the maximum-likelihood $\hat\mu$ and $\hat\sigma^2$ ?

If you want the algebra · 2 proofs, step by step

Prove it · the best middle is the average

Claim. For data $x_1,\dots,x_N$ and a bell $\mathcal N(\mu,\sigma^2)$ , the log-likelihood is largest over $\mu$ at $\hat\mu=\bar x$ .

1

Take the log of the product of

N

bell heights:

\begin{aligned}\ell(\mu,\sigma)&=-N\log\sigma\\ &\quad-\frac{1}{2\sigma^2}\sum_i(x_i-\mu)^2\\ &\quad-\frac N2\log2\pi.\end{aligned}

2

Only the middle term has

\mu

. Its slope is

\begin{aligned}&\frac{\partial\ell}{\partial\mu}=\frac1{\sigma^2}\sum_i(x_i-\mu)=0\\ &\Rightarrow\;\hat\mu=\frac1N\sum_ix_i.\end{aligned}

∎ Whatever

\sigma

is, the best middle is the average. The chai times give

\hat\mu=5

.

Prove it · the best spread divides by N

Claim. With $\mu=\hat\mu$ , the log-likelihood is largest at $\hat\sigma^2=\tfrac1N\sum_i(x_i-\hat\mu)^2$ .

1

Write

S=\sum_i(x_i-\hat\mu)^2

. Then

\ell=-N\log\sigma-\dfrac{S}{2\sigma^2}+\text{const}

, and

\frac{\partial\ell}{\partial\sigma}=-\frac N\sigma+\frac{S}{\sigma^3}.

2

Set to zero:

N\sigma^2=S

, so

\hat\sigma^2=S/N

. ∎ Chai times:

S=32

,

N=8

,

\hat\sigma^2=4

. The two terms pull against each other:

-N\log\sigma

wants a narrow bell (tall peaks),

-S/2\sigma^2

wants a wide one (no point on a tail). The balance is at the data's own spread.

In one sentence: Maximum likelihood fits a bell by placing it at the average and making it as wide as the average squared spread — a climb to the top of the likelihood landscape, with a small downward bias that dividing by $N-1$ corrects.

9

Least squares falls out

Imagine this

An auto fare should be a fixed start plus a rate per kilometre. But real fares wobble: traffic, a detour, rounding to the nearest ten.

You note down four rides and want the best straight line through them. In Unit 3 you would have said "minimise the squared errors" and projected. But why squared? Why not the plain size of the errors, or their cubes?

Probability gives the answer — and it is a beautiful one.

Say what you believe about the wobble. The simplest belief: the true fare lies on the line, plus a random nudge from a bell,

y_i=wx_i+b+\varepsilon_i,\qquad\varepsilon_i\sim\mathcal N(0,\sigma^2).

Then each data point $y_i$ is a draw from a bell centred on the line's prediction $\hat y_i=wx_i+b$ . The likelihood of a line is the product of the heights of the data points on their bells. Take $-\log$ of it:

\begin{aligned}-\log L(w,b)&=\frac{1}{2\sigma^2}\sum_i\big(y_i-\hat y_i\big)^2\\ &\quad+N\log\big(\sigma\sqrt{2\pi}\big).\end{aligned}

The second part does not depend on the line at all. So making the data most likely is the same as making the sum of squared errors smallest. The projection of Unit 3 was maximum likelihood under bell-shaped noise all along.

Worked example. Four rides: $x=0,1,2,3$ and $y=1,3,2,5$ . The averages are $\bar x=1.5$ and $\bar y=2.75$ . The slope is

\begin{gathered}w=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}=\frac{5.5}{5}=1.1,\\ b=\bar y-w\bar x=2.75-1.65=1.1.\end{gathered}

The predictions are $1.1,2.2,3.3,4.4$ , the errors $-0.1,\;0.8,\;-1.3,\;0.6$ , and the sum of their squares is $0.01+0.64+1.69+0.36=2.70$ .

A different belief, a different loss. Suppose instead the nudges follow a Laplace distribution, a pointy tent $\tfrac1{2s}e^{-|\varepsilon|/s}$ with heavier tails than the bell. The same steps give

-\log L=\frac1s\sum_i\big|y_i-\hat y_i\big|+\text{const}.

Now the best line minimises the sum of absolute errors. Squares punish a big error enormously (an error of 10 costs 100); absolute values punish it only 10. So the Laplace fit shrugs off an outlier. Add a mistyped ride at $(1,\,8)$ : the least-squares line tilts from slope 1.1 to about 0.654, while the Laplace line stays exactly where it was, $w=\tfrac43$ , $b=1$ .

Try: Drag the slope and start sliders: when a bell's centre moves away from its point, the bead sinks down the bell's side. Press ▶ best fit — the log-likelihood and the squared error move in lockstep. Then add the outlier and flip between Gaussian and Laplace noise.

drag the picture to orbit

slope w0.5

start b2

—

The realization

$\begin{gathered}\text{bell noise}\;\Rightarrow\;\min\sum_i(y_i-\hat y_i)^2,\\ \text{Laplace noise}\;\Rightarrow\;\min\sum_i|y_i-\hat y_i|\end{gathered}$

A loss function is a belief about noise in disguise. Squared error is maximum likelihood with bell-shaped noise — that is why least squares works so well, and why it is thrown off by outliers that a bell says should never happen.

Pause & predict

For $x=0,1,2,3$ and $y=1,3,2,5$ with bell-shaped noise, what is the maximum-likelihood line?

Pause & predict

Your sensor sometimes produces wild readings. Which noise model, and therefore which loss, protects the fit?

If you want the algebra · 2 proofs, step by step

Prove it · −log of the Gaussian likelihood is the squared error

Claim. If $y_i=wx_i+b+\varepsilon_i$ with independent $\varepsilon_i\sim\mathcal N(0,\sigma^2)$ , then $-\log L(w,b)=\dfrac1{2\sigma^2}\sum_i(y_i-\hat y_i)^2+N\log(\sigma\sqrt{2\pi})$ .

1

Each

y_i

is a bell draw centred on

\hat y_i=wx_i+b

. Independence lets us multiply:

\begin{aligned}L&=\prod_i\frac1{\sigma\sqrt{2\pi}}\\ &\qquad\times\exp\!\Big(-\frac{(y_i-\hat y_i)^2}{2\sigma^2}\Big).\end{aligned}

2

The log of a product is a sum, and

-\log e^{-a}=a

:

\begin{aligned}-\log L&=\sum_i\frac{(y_i-\hat y_i)^2}{2\sigma^2}\\ &\quad+N\log(\sigma\sqrt{2\pi}).\end{aligned}

∎ The last term has no

w

or

b

, and

\tfrac1{2\sigma^2}

is a positive constant, so the best line is the one with the smallest

\sum(y_i-\hat y_i)^2

— least squares.

Prove it · Laplace noise gives absolute error

Claim. With $\varepsilon_i$ drawn from the Laplace density $\tfrac1{2s}e^{-|\varepsilon|/s}$ , $-\log L=\tfrac1s\sum_i|y_i-\hat y_i|+N\log2s$ .

1

Same steps, a different curve:

\begin{aligned}&-\log\prod_i\frac{e^{-|y_i-\hat y_i|/s}}{2s}\\ &\quad=\frac1s\sum_i|y_i-\hat y_i|\\ &\qquad+N\log2s.\end{aligned}

∎ So the most likely line minimises the total absolute error. For a single constant (no slope) this picks the median instead of the mean — which is why it ignores a wild value.

In one sentence: If the noise is a bell, the most likely line is the least-squares line — Unit 3's projection was maximum likelihood all along — and a pointier, heavier-tailed noise turns the loss into absolute error, which shrugs off outliers.

10

Softmax: turning scores into probabilities

Imagine this

Three children helped at a family wedding and earned points: Asha 2, Bittu 1, Chintu 0. You have one box of laddoos to share.

You want three things. Everyone gets some, even Chintu. More points means more laddoos. And the whole box is given out, nothing left over.

Softmax is a recipe that does exactly this, and it is how almost every modern classifier turns its raw scores into probabilities.

A classifier looks at a photo and produces one raw score per class: cat 2, dog 1, rabbit 0. The scores can be any numbers, even negative. We want probabilities: positive, in the same order, adding to 1. Two steps:

Exponentiate. $e^{z}$ is always positive and keeps the order. Each extra point of score multiplies the share by $e\approx2.718$ . So $e^2=7.389$ , $e^1=2.718$ , $e^0=1$ .
Divide by the total. The total is $7.389+2.718+1=11.107$ . Dividing gives $0.665$ , $0.245$ , $0.090$ . They add to 1.

\operatorname{softmax}(\mathbf z)_k=\frac{e^{z_k}}{\sum_j e^{z_j}}.

Only differences matter. Add 10 to every score. Every $e^{z}$ gets multiplied by the same $e^{10}$ , which cancels when you divide. Nothing changes. Real code uses this: it subtracts the largest score first, so it never has to compute something like $e^{1000}$ , which no computer can hold.

The temperature dial. Divide the scores by a number $T$ before the softmax. A small $T$ stretches the differences; a big $T$ squashes them.

temperature	scores used	cat	dog	rabbit	character
$T\to0$	huge gaps	1	0	0	winner takes all (argmax)
$T=0.5$	4, 2, 0	0.867	0.117	0.016	confident
$T=1$	2, 1, 0	0.665	0.245	0.090	the plain softmax
$T=2$	1, 0.5, 0	0.506	0.307	0.186	relaxed
$T\to\infty$	all near 0	⅓	⅓	⅓	no opinion (uniform)

This is exactly the "temperature" setting on a chatbot. A language model picks each next word from a softmax, and turning $T$ down makes it predictable, turning it up makes it adventurous. You will meet this dial again in Unit 19.

Two classes: the sigmoid. With only two scores $z_1,z_2$ , divide top and bottom by $e^{z_1}$ :

\begin{aligned}\frac{e^{z_1}}{e^{z_1}+e^{z_2}}&=\frac{1}{1+e^{-(z_1-z_2)}}\\ &=\sigma(z_1-z_2).\end{aligned}

That S-shaped curve $\sigma$ is the sigmoid (or logistic function). It finally gives Unit 13's raw score a meaning. The SVM said $f(\mathbf x)=1.5$ , "positive side". Pass it through the sigmoid: $\sigma(1.5)\approx0.818$ , "about 82% sure". A point on the boundary, $f=0$ , gets exactly 50%.

Try: Drag the dog's score up past the cat's and watch the shares swap. Slide the temperature to 0.1 (winner takes all) and to 10 (almost ⅓ each). Then tick subtract the max: the scores shift, the probabilities do not move at all.

drag a score bar up or down

temperature T1

—

Where do the answers live? Three probabilities that add to 1 are a point in 3-D space, and all such points lie on one flat triangle with corners $(1,0,0)$ , $(0,1,0)$ and $(0,0,1)$ . It is called the probability simplex. A corner is total certainty; the centre is "no idea". Every softmax output is a point on this triangle, and the temperature moves it.

Try: Press ▶ sweep T: the glowing point starts near the cat corner (T small) and slides to the centre (T large), drawing its path. Change the scores in the softmax machine above — the path here redraws.

drag the picture to orbit

temperature T1

—

The realization

$\begin{gathered}q_k=\frac{e^{z_k/T}}{\sum_j e^{z_j/T}},\\ \text{two classes: } q_1=\sigma(z_1-z_2)\end{gathered}$

Exponentiate, then share out. Softmax turns any scores into a point on the probability triangle; only the differences between scores matter; and the temperature slides the answer between "winner takes all" and "no opinion".

Pause & predict

The scores $(2,1,0)$ give probabilities $(0.665,0.245,0.090)$ . What do the scores $(12,11,10)$ give?

Pause & predict

You lower the temperature from 1 to 0.5 for the scores $(2,1,0)$ . What happens to the top probability?

If you want the algebra · 2 proofs, step by step

Prove it · adding a constant to every score changes nothing

Claim. $\operatorname{softmax}(\mathbf z+c\mathbf 1)=\operatorname{softmax}(\mathbf z)$ for any number $c$ .

1

e^{z_k+c}=e^{c}e^{z_k}

, so the common factor

e^c

appears on top and in every term of the bottom:

\frac{e^ce^{z_k}}{\sum_je^ce^{z_j}}=\frac{e^{z_k}}{\sum_je^{z_j}}.

∎ With

c=-\max_jz_j

, every exponent is

\le0

, so no

e^{z}

can overflow. That is the "subtract the max" line in every softmax routine.

Prove it · two-class softmax is the sigmoid

Claim. $\dfrac{e^{z_1}}{e^{z_1}+e^{z_2}}=\sigma(z_1-z_2)$ , where $\sigma(t)=\dfrac1{1+e^{-t}}$ .

1

Divide top and bottom by

e^{z_1}

:

\begin{aligned}\frac{1}{1+e^{z_2-z_1}}&=\frac1{1+e^{-(z_1-z_2)}}\\ &=\sigma(z_1-z_2).\end{aligned}

∎ Only the difference

z_1-z_2

survives, as the shift rule promised. An SVM score

f(\mathbf x)

can play that difference:

\sigma(f)

is 0.5 on the boundary and approaches 1 deep on the positive side.

In one sentence: Softmax exponentiates the scores and shares them out so they become probabilities — only score differences matter, temperature slides from winner-takes-all to uniform, and with two classes it is the sigmoid of the score difference.

11

Surprise and entropy

Imagine this

Three things happen tomorrow. The sun rises. The toss before the match comes up heads. Your neighbour wins the state lottery.

The sunrise surprises nobody. The toss surprises you a little — it could easily have gone the other way. The lottery win is shocking.

The rarer the event, the bigger the surprise. Can we put a number on surprise?

Yes. The surprise of an outcome with probability $p$ is

\text{surprise}=-\log_2 p.

Certain events ( $p=1$ ) carry zero surprise. A fair toss ( $p=\tfrac12$ ) carries 1. An event with $p=\tfrac14$ carries 2, and a one-in-1024 event carries 10. Why a log? Because when two independent things happen, their chances multiply, and we want their surprises to add. The log turns multiplying into adding.

With $\log_2$ the unit is the bit: the number of fair yes/no questions the news is worth. With the natural log $\ln$ the unit is the nat. They differ only by a fixed factor: 1 nat $\approx1.443$ bits. Machine-learning code usually uses nats.

Now average the surprise over all outcomes, weighting each by how often it happens. That average is the entropy:

H(p)=-\sum_k p_k\log_2 p_k.

A fair coin: $\tfrac12\cdot1+\tfrac12\cdot1=1$ bit.
A coin that shows heads 90% of the time: $0.9\cdot0.152+0.1\cdot3.32\approx0.469$ bits. Usually no surprise, occasionally a big one; on average, less than half a bit.
Four equally likely outcomes: 2 bits (two yes/no questions pin it down).
Something certain: 0 bits.

Entropy measures how uncertain a distribution is. It is largest when every outcome is equally likely, and zero when one outcome is certain.

Try: Drag the coin marker to 0.9: the curve reads 0.469 bits. Then drag the glowing point on the dome toward a corner and watch the height drain to zero; at the centre it is $\log_2 3\approx1.585$ bits, the most uncertain three outcomes can be.

drag the coin marker or the point on the dome

—

The realization

$\begin{gathered}\text{surprise}(x)=-\log p(x),\\ H(p)=\mathbb E\big[-\log p(X)\big]\\ =-\sum_k p_k\log p_k\end{gathered}$

Surprise is minus the log of the chance. Entropy is the average surprise — the true uncertainty of a distribution: zero when you are certain, largest when every outcome is equally likely.

Pause & predict

Which has more entropy: a fair coin, or a coin that shows heads 90% of the time?

In one sentence: The surprise of an outcome is $-\log p$ , and entropy — the average surprise — measures how uncertain a distribution is, from 0 for a sure thing up to its maximum when all outcomes are equally likely.

12

Cross-entropy and KL: paying for the wrong belief

Imagine this

You move to Mumbai in July, believing "90% of days are dry". You leave your umbrella at home most mornings.

In truth, July is about half rain, half dry. Every rainy day shocks you — you had given it only a 10% chance. You are not unlucky. You are paying for a wrong belief, and you pay in surprise.

Call the truth $p$ and your belief $q$ . Each day the weather comes from $p$ , but your surprise is measured with $q$ : $-\log q(\text{what happened})$ . Average it over the true weather and you get the cross-entropy:

H(p,q)=-\sum_k p_k\log q_k.

Mumbai in July, in bits: truth $p=(0.5,0.5)$ for (rain, dry), belief $q=(0.1,0.9)$ . Your average surprise is $0.5\cdot3.32+0.5\cdot0.152\approx1.737$ bits. Someone who knew the truth would average only $H(p)=1$ bit. The extra 0.737 bits a day is the price of your wrong belief. It has a name, the KL divergence (after Kullback and Leibler):

\begin{gathered}H(p,q)=H(p)+\mathrm{KL}(p\,\|\,q),\\ \mathrm{KL}(p\,\|\,q)=\sum_k p_k\log\frac{p_k}{q_k}.\end{gathered}

Three facts about KL:

Never negative. A wrong belief can never make you less surprised on average than the truth.
Zero only when $q=p$ . It is a way of measuring how far a belief is from the truth.
Not symmetric. Swap the roles: $\mathrm{KL}(q\,\|\,p)\approx0.531$ bits, not 0.737. "Believing 90/10 when the truth is 50/50" costs more than "believing 50/50 when the truth is 90/10". So KL is not a true distance — the direction matters.

Why the direction matters when you fit. Suppose the truth $p$ has two humps, and you must describe it with one bell $q$ . If you minimise $\mathrm{KL}(p\,\|\,q)$ , every place where $p$ has mass but $q$ has almost none is punished hard, so $q$ spreads out to cover both humps. If you minimise $\mathrm{KL}(q\,\|\,p)$ , every place where $q$ puts mass but $p$ has none is punished, so $q$ locks onto one hump and ignores the other. Two humps at $\pm2$ with spread 0.6: the first fit is a wide bell at 0 with spread about 2.09; the second is a narrow bell at about $\pm2$ with spread about 0.6.

Try: On bars, press Mumbai in July, then drag the orange bars toward the blue ones and watch the red KL slice shrink to nothing. On fit one bell, press both fit buttons and compare: one bell stretches over both humps, the other picks a side.

drag a bar up or down

outcomes2

—

The realization

$\begin{aligned}H(p,q)&=\underbrace{H(p)}_{\text{the world's own uncertainty}}\\ &\quad+\underbrace{\mathrm{KL}(p\,\|\,q)}_{\text{price of your wrong belief}\;\ge\;0}\end{aligned}$

Cross-entropy is your average surprise when the truth is $p$ and you believe $q$ . It splits into the surprise nobody can avoid, $H(p)$ , plus the extra you pay for being wrong, $\mathrm{KL}$ . KL is never negative, zero only for a perfect belief, and not symmetric.

Pause & predict

The truth is $p=(0.5,0.5)$ and your belief is $q=(0.9,0.1)$ . You compute $\mathrm{KL}(p\,\|\,q)\approx0.737$ bits. What is $\mathrm{KL}(q\,\|\,p)$ ?

If you want the algebra · 2 proofs, step by step

Prove it · cross-entropy = entropy + KL

Claim. $H(p,q)=H(p)+\mathrm{KL}(p\,\|\,q)$ .

1

Split the log with

\log q_k=\log p_k-\log\dfrac{p_k}{q_k}

:

\begin{aligned}&-\sum_kp_k\log q_k\\ &\quad=-\sum_kp_k\log p_k\\ &\qquad+\sum_kp_k\log\frac{p_k}{q_k}.\end{aligned}

∎ Mumbai in July:

1.737=1+0.737

bits. When we train a model

q

on data from

p

,

H(p)

is fixed, so minimising cross-entropy and minimising KL are the same job.

Prove it · KL is never negative (in outline)

Claim. $\mathrm{KL}(p\,\|\,q)\ge0$ , with equality only when $q=p$ .

1

The log is a curve that bends downward, so the log of an average is at least the average of the logs (Jensen's inequality). Apply it with weights

p_k

to the numbers

q_k/p_k

:

\begin{aligned}-\mathrm{KL}&=\sum_kp_k\log\frac{q_k}{p_k}\\ &\le\log\sum_kp_k\frac{q_k}{p_k}\\ &=\log\sum_kq_k=\log1=0.\end{aligned}

2

Equality in Jensen needs all the numbers

q_k/p_k

to be equal, and since both add to 1, that means

q_k=p_k

for every

k

. ∎ So a wrong belief always costs extra surprise on average, and only the truth itself costs nothing extra.

In one sentence: Cross-entropy is the average surprise of believing $q$ when the truth is $p$ , it equals the truth's own entropy plus $\mathrm{KL}(p\,\|\,q)\ge0$ , and because KL is lopsided, fitting in one direction covers every hump while the other direction locks onto one.

13

Cross-entropy is the classifier's likelihood

Imagine this

A student guesses answers with confidence levels. On one question she writes "70% sure it's (a)", and the answer is (a). Good — little surprise.

On another she writes "1% sure it's (c)" — and the answer is (c). She should be very surprised, and a good teacher would mark that down hard: she was confidently wrong.

Training a classifier is teaching it to be less surprised by the right answers.

A classifier outputs a belief $\mathbf q=\operatorname{softmax}(\mathbf z)$ . The truth for one example is a certainty: "it is class 1", written as a one-hot vector $\mathbf y=(1,0,0)$ . The cross-entropy between them has only one term left, because the other $y_k$ are zero:

H(\mathbf y,\mathbf q)=-\sum_k y_k\log q_k=-\log q_{\text{correct}}.

That is just the surprise of the right answer. And it is also the negative log-likelihood from §7: "how likely did the model say the true label was?". So minimising cross-entropy is maximum likelihood for a classifier. Every idea of Act III carries over.

Worked example. Scores $(2,1,0)$ , the true class is the first. Then $\mathbf q=(0.665,0.245,0.090)$ and

\text{loss}=-\ln0.665\approx0.408\text{ nats}.

The cleanest gradient in machine learning. How should each score change to lower the loss? The answer (the drawer derives it) is astonishingly simple:

\begin{aligned}\frac{\partial\,\text{loss}}{\partial\mathbf z}&=\mathbf q-\mathbf y\\ &=(0.665-1,\;0.245-0,\;0.090-0)\\ &=(-0.335,\;0.245,\;0.090).\end{aligned}

Prediction minus truth. The correct class gets a negative gradient, so a step downhill raises its score. Each wrong class gets a positive gradient equal to the probability it stole, so it is pushed down by exactly that much. When the prediction is perfect, $\mathbf q=\mathbf y$ and the gradient is zero.

A whole classifier. For a 2-D point $\mathbf x$ , let each class have its own score line, $z_k=\mathbf w_k\cdot\mathbf x+b_k$ . Softmax turns the three scores into probabilities. The loss over the data is the average cross-entropy, and the gradient for $\mathbf w_k$ is $(q_k-y_k)\,\mathbf x$ . Walk downhill with gradient descent (Unit 9). This is softmax regression, and the step " $\mathbf q-\mathbf y$ " is the first move of every backward pass in Unit 15.

Try: Press ▶ train and watch the fog of the untrained model sharpen into three regions, with soft edges where the classes meet. Drag the white probe into a border zone: its three probabilities share out. Then turn the prediction temperature down to 0.3 — the same model, far more sure of itself.

drag the white probe

step size η0.5

prediction temperature1

—

The realization

$\begin{gathered}\text{loss}=-\log q_{\text{correct}},\\ \frac{\partial\,\text{loss}}{\partial\mathbf z}=\mathbf q-\mathbf y\end{gathered}$

With a one-hot truth, cross-entropy is the surprise of the right answer — the negative log-likelihood. Its gradient with respect to the scores is prediction minus truth, the simplest signal a learner could ask for.

Pause & predict

A model gives the true class probability 0.9 on one example and 0.1 on another. What are the two cross-entropy losses (in nats)?

Pause & predict

Scores $(2,1,0)$ give $\mathbf q=(0.665,0.245,0.090)$ , and the true class is the second. What is the gradient of the loss with respect to the scores?

If you want the algebra · 1 proof, step by step

Prove it · the gradient of −log softmax is q − y

Claim. For $\text{loss}=-\log q_c$ with $\mathbf q=\operatorname{softmax}(\mathbf z)$ and true class $c$ , $\dfrac{\partial\,\text{loss}}{\partial z_k}=q_k-y_k$ .

1

Write the loss without the fraction:

\begin{aligned}-\log q_c&=-z_c\\ &\quad+\log\sum_je^{z_j}.\end{aligned}

2

Differentiate with respect to

z_k

. The first term gives

-1

if

k=c

and 0 otherwise — that is

-y_k

. The second gives

\frac{e^{z_k}}{\sum_je^{z_j}}=q_k.

3

Add them:

\dfrac{\partial\,\text{loss}}{\partial z_k}=q_k-y_k

. ∎ Scores

(2,1,0)

, class 1:

(0.665-1,\;0.245,\;0.090)=(-0.335,\;0.245,\;0.090)

. The entries always add to 0: raising one score is paid for by the others. And for a linear model

z_k=\mathbf w_k\cdot\mathbf x+b_k

, the chain rule gives

\partial\,\text{loss}/\partial\mathbf w_k=(q_k-y_k)\,\mathbf x

.

In one sentence: For a classifier, cross-entropy with the one-hot truth is $-\log q_{\text{correct}}$ , the negative log-likelihood, and its gradient with respect to the scores is simply prediction minus truth, $\mathbf q-\mathbf y$ .

14

What to carry forward

Thirteen sections, one shift in thinking: from crisp answers to honest maybes. Here is the kit you now carry.

Distributions

\mathbb E[X]=\sum x\,p(x),\quad\operatorname{Var}=\mathbb E[(X-\mu)^2]

Mass for counts, area for smooth quantities. The mean balances, the variance spreads.

Bayes

P(H\mid E)\propto P(E\mid H)\,P(H)

Posterior ∝ likelihood × prior. The prior is where you stood before the evidence.

The Gaussian and its ellipse

p(\mathbf x)\propto e^{-\frac12(\mathbf x-\boldsymbol\mu)^{\!\top}\Sigma^{-1}(\mathbf x-\boldsymbol\mu)}

Contours are Unit 12's ellipse, axes are Unit 4's eigenvectors, samples are $\boldsymbol\mu+L\mathbf z$ .

Likelihood and MLE

\hat\theta=\arg\max\log P(\text{data}\mid\theta)

A coin: the share of heads. A bell: the average and the average squared spread.

Least squares = Gaussian MLE

-\log L=\tfrac1{2\sigma^2}\textstyle\sum(y_i-\hat y_i)^2+\text{const}

A loss is a noise belief in disguise. Laplace noise gives absolute error.

Softmax and temperature

q_k=\frac{e^{z_k/T}}{\sum_je^{z_j/T}}

Only differences matter. Two classes: $\sigma(z_1-z_2)$ .

Entropy, cross-entropy, KL

H(p,q)=H(p)+\mathrm{KL}(p\,\|\,q)

Average surprise; the price of a wrong belief; never negative, not symmetric.

The training signal

\frac{\partial(-\log q_y)}{\partial\mathbf z}=\mathbf q-\mathbf y

Prediction minus truth — the first step of every backward pass.

The thread to Units 15–20

Unit 15 wires softmax and cross-entropy onto the end of a whole network, and " $\mathbf q-\mathbf y$ " is where its backward pass starts. Unit 16 predicts the next word with a softmax over the whole vocabulary. Unit 19 samples from that softmax with a temperature dial. Unit 20 builds pictures from Gaussian noise, using the stretch $\boldsymbol\mu+L\mathbf z$ (the reparameterisation trick) and a KL penalty to keep its beliefs honest.

The one thing to remember

Learning is choosing the world that makes your data least surprising. Squared error, cross-entropy and maximum likelihood are three names for that one idea.

In one sentence: Uncertainty has a shape, the Gaussian's shape is the covariance ellipse, fitting a model means maximising the likelihood of the data, and softmax plus cross-entropy turn that idea into a classifier whose gradient is prediction minus truth.

15

Practice arena — fourteen problems, solved in full

Fourteen problems, easy to hard: a loaded die, two rounds of Bayes, a bell read with z-scores, a covariance matrix taken apart, a Cholesky factor, a coin and a bell fitted by maximum likelihood, a line fitted through its likelihood, softmax at two temperatures, the sigmoid, entropy, cross-entropy and both KLs, one training step of a classifier, and a question about which loss belongs to which noise. Every number was checked by machine.

Three habits do most of the work. Count before you divide: for Bayes, imagine 10 000 people and count them. Take the log early: products of chances become sums you can differentiate. And check that probabilities add to 1 — it catches most slips.

Problem 1easypmf · mean · variance

A loaded die shows 6 with probability $\tfrac12$ and each of the faces 1–5 with probability $\tfrac1{10}$ . (a) Check that it is a valid pmf. (b) Find $\mathbb E[X]$ . (c) Find $\operatorname{Var}[X]$ and $\sigma$ . (d) Compare with a fair die.

What this tests. The balance point and the spread computed from a table. Plan. Add the weights; weighted average of $x$ ; then the shortcut $\operatorname{Var}=\mathbb E[X^2]-\mu^2$ .

Show the full solution

Step 1 — valid (a).

5\times\tfrac1{10}+\tfrac12=1

✓, and every weight is between 0 and 1.

Step 2 — the mean (b).

\mathbb E[X]=\tfrac{1+2+3+4+5}{10}+6\cdot\tfrac12=1.5+3=4.5

.

Step 3 — the variance (c).

\mathbb E[X^2]=\tfrac{1+4+9+16+25}{10}+36\cdot\tfrac12=5.5+18=23.5

, so

\operatorname{Var}=23.5-4.5^2=23.5-20.25=3.25

and

\sigma=\sqrt{3.25}\approx1.803

.

Step 4 — compare (d). A fair die has mean 3.5 and variance

\tfrac{35}{12}\approx2.917

. The loading moves the balance point up by 1 and makes the outcomes a little more spread out.

answers at a glance: (a) weights add to 1 ✓; (b) $\mathbb E[X]=4.5$ ; (c) $\operatorname{Var}=3.25$ , $\sigma\approx1.803$ ; (d) fair die: 3.5 and 2.917.

Problem 2easyBayes · two tests

A disease affects 2% of people. A test catches 90% of the sick and gives a false alarm to 10% of the healthy. (a) What fraction of all people test positive? (b) You test positive. What is the chance you are sick? (c) A second, independent test is also positive. Now what? (d) If instead the first test had been negative, what is the chance you are sick?

What this tests. Bayes' rule with natural counts, and chaining an update. Plan. Imagine 10 000 people and count. For (c), yesterday's posterior is today's prior.

Show the full solution

Step 1 — count (a). Of 10 000 people, 200 are sick and 180 of them test positive; 9 800 are healthy and 980 of them get a false alarm. Positives:

180+980=1160

, which is

11.6\%

.

Step 2 — one positive (b).

P(\text{sick}\mid+)=\tfrac{180}{1160}\approx0.155

. Only about 1 in 6.4 positives is sick: the false alarms on the big healthy crowd swamp the real cases.

Step 3 — a second positive (c). New prior

0.155

:

\begin{gathered}\frac{0.9\times0.155}{0.9\times0.155+0.1\times0.845}\\ \approx0.623.\end{gathered}

In counts: of the 1160 positives, 180 sick → 162 positive again; 980 healthy → 98 positive again;

\tfrac{162}{260}\approx0.623

✓.

Step 4 — a negative (d). Negatives: 20 sick (missed) and 8 820 healthy.

P(\text{sick}\mid-)=\tfrac{20}{8840}\approx0.0023

— a negative is very reassuring.

answers at a glance: (a) 11.6% test positive; (b) $\approx0.155$ ; (c) $\approx0.623$ ; (d) $\approx0.0023$ .

Remember

A positive test on a rare condition is weak evidence on its own; two independent positives are strong. And the same test can be far more convincing when it says "no" than when it says "yes".

Problem 3easythe bell · z-scores

Marks in a subject follow $\mathcal N(60,10^2)$ . (a) Roughly what share scored between 50 and 70? Between 40 and 80? (b) Roughly what share scored above 80? (c) A student scored 75. What is her z-score? (d) Another subject has $\mu=70$ , $\sigma=5$ and her friend scored 80 there. Who did better relative to their class?

What this tests. The 68–95–99.7 rule and comparing across different bells with z-scores. Plan. Turn every mark into "how many $\sigma$ 's from the middle".

Show the full solution

Step 1 — the rule (a). 50 to 70 is

\pm1\sigma

: about 68% (68.3%). 40 to 80 is

\pm2\sigma

: about 95% (95.4%).

Step 2 — the tail (b). Outside

\pm2\sigma

is about 4.6%, split equally between the two tails by symmetry: about 2.3% scored above 80.

Step 3 — z (c).

z=\tfrac{75-60}{10}=1.5

.

Step 4 — compare (d). Her friend:

z=\tfrac{80-70}{5}=2

. The friend is 2 spreads above the class middle, she is 1.5, so the friend did better relative to their class — even though 80 − 70 = 10 is less than 75 − 60 = 15.

answers at a glance: (a) ≈ 68% and ≈ 95%; (b) ≈ 2.3%; (c) $z=1.5$ ; (d) the friend ( $z=2$ against $1.5$ ).

Problem 4medium2-D Gaussian · Mahalanobis

A 2-D Gaussian has $\boldsymbol\mu=\mathbf 0$ and $\Sigma=\begin{bmatrix}5&2\\2&2\end{bmatrix}$ . (a) Find the eigenvalues and unit eigenvectors of $\Sigma$ . (b) Describe the contour ellipse. (c) Find $\det\Sigma$ and the peak density. (d) Find the Mahalanobis distance of $(2,1)$ and of $(1,-2)$ from the mean, and compare with their ordinary distances.

What this tests. Reading a Gaussian's shape from $\Sigma$ (Unit 4 meets §5). Plan. Trace and determinant give the eigenvalues; eigenvectors give the axes; $\Sigma^{-1}$ gives the ruler.

Show the full solution

Step 1 — eigenvalues (a). Trace

7

, determinant

10-4=6

. So

\lambda^2-7\lambda+6=0

,

\lambda=6

or

1

.

Step 2 — eigenvectors (a). For

\lambda=6

:

(5-6)v_1+2v_2=0\Rightarrow v=(2,1)/\sqrt5

. For

\lambda=1

:

4v_1+2v_2=0\Rightarrow v=(1,-2)/\sqrt5

. They are at right angles, as the spectral theorem promises.

Step 3 — the ellipse (b). The long axis points along

(2,1)

, the short axis along

(1,-2)

. Half-lengths are in the ratio

\sqrt6:1\approx2.45:1

.

Step 4 — the peak (c).

\det\Sigma=6

, so

p_{\max}=\dfrac1{2\pi\sqrt6}\approx0.0650

.

Step 5 — the ruler (d).

\Sigma^{-1}=\tfrac16\begin{bmatrix}2&-2\\-2&5\end{bmatrix}

.

\begin{aligned}(2,1):\ d^2&=\tfrac{8-8+5}{6}=\tfrac56,\\ d&\approx0.913;\\ (1,-2):\ d^2&=\tfrac{2+8+20}{6}=5,\\ d&\approx2.236.\end{aligned}

Both points have ordinary distance

\sqrt5\approx2.236

. The first lies on the long axis (

d^2=5/6

, i.e.

\|\mathbf x\|^2/\lambda_1

); the second on the short axis (

d^2=5/1

).

answers at a glance: (a) $\lambda=6$ along $(2,1)/\sqrt5$ , $\lambda=1$ along $(1,-2)/\sqrt5$ ; (b) ellipse tilted along $(2,1)$ , axes in ratio $\sqrt6:1$ ; (c) $\det=6$ , peak $\approx0.0650$ ; (d) Mahalanobis $0.913$ and $2.236$ , ordinary distance $2.236$ for both.

Remember

Along an eigenvector, the Mahalanobis distance is the ordinary distance divided by $\sqrt\lambda$ : long axes forgive, short axes do not.

Problem 5mediumsampling · Cholesky

You need samples from $\mathcal N(\boldsymbol\mu,\Sigma)$ with $\boldsymbol\mu=(1,1)$ and $\Sigma=\begin{bmatrix}9&6\\6&5\end{bmatrix}$ . (a) Find the Cholesky factor $L$ . (b) Check $LL^{\!\top}=\Sigma$ . (c) The computer gives the round sample $\mathbf z=(1,-1)$ . What sample $\mathbf x$ do you produce? (d) What is the correlation between $x_1$ and $x_2$ ?

What this tests. The stretch recipe $\mathbf x=\boldsymbol\mu+L\mathbf z$ . Plan. Fill $L$ entry by entry, top-left first.

Show the full solution

Step 1 — L (a).

\ell_{11}=\sqrt9=3

;

\ell_{21}=6/3=2

;

\ell_{22}=\sqrt{5-2^2}=1

. So

L=\begin{bmatrix}3&0\\2&1\end{bmatrix}

.

Step 2 — check (b).

LL^{\!\top}=\begin{bmatrix}9&6\\6&4+1\end{bmatrix}=\Sigma

✓.

Step 3 — one sample (c).

L\mathbf z=(3\cdot1,\;2\cdot1+1\cdot(-1))=(3,1)

, so

\mathbf x=(1,1)+(3,1)=(4,2)

.

Step 4 — correlation (d).

\rho=\dfrac{6}{\sqrt9\,\sqrt5}=\dfrac{6}{3\sqrt5}=\dfrac2{\sqrt5}\approx0.894

: strongly tied together, a narrow tilted cloud.

answers at a glance: (a) $L=\begin{bmatrix}3&0\\2&1\end{bmatrix}$ ; (b) ✓; (c) $\mathbf x=(4,2)$ ; (d) $\rho\approx0.894$ .

Problem 6easylikelihood · coin

A coin shows 3 heads in 12 tosses. (a) Write $L(p)$ and $\log L(p)$ . (b) Find $\hat p$ . (c) Compute $\log L$ at $\hat p$ and at $p=0.5$ (natural log). (d) How many times better does $\hat p$ explain the data than a fair coin?

What this tests. Maximum likelihood and comparing models by their likelihood. Plan. Differentiate the log; compare in logs and exponentiate the difference.

Show the full solution

Step 1 — the functions (a).

L(p)=p^3(1-p)^9

and

\log L=3\log p+9\log(1-p)

.

Step 2 — the peak (b).

\tfrac3p-\tfrac9{1-p}=0\Rightarrow3(1-p)=9p\Rightarrow\hat p=\tfrac3{12}=0.25

.

Step 3 — two log-likelihoods (c).

\log L(0.25)=3\ln0.25+9\ln0.75\approx-4.159-2.589=-6.748

.

\log L(0.5)=12\ln0.5\approx-8.318

.

Step 4 — the ratio (d).

e^{-6.748-(-8.318)}=e^{1.570}\approx4.81

. The best coin explains the tosses about 4.8 times better than a fair one.

answers at a glance: (a) $p^3(1-p)^9$ ; (b) $\hat p=0.25$ ; (c) $-6.748$ and $-8.318$ ; (d) about 4.81 times.

Problem 7easyGaussian MLE · N vs N − 1

Measurements: 1, 3, 4, 6, 6. (a) Find the maximum-likelihood $\hat\mu$ and $\hat\sigma^2$ . (b) Find the "divide by $N-1$ " variance. (c) Which one is too small on average, and why?

What this tests. The Gaussian MLE formulas and the reason for the $N-1$ correction. Plan. Average; squared distances from the average; divide two ways.

Show the full solution

Step 1 — the middle (a).

\hat\mu=\tfrac{1+3+4+6+6}{5}=\tfrac{20}5=4

.

Step 2 — the spread (a). Squared distances:

9,1,0,4,4

, total 18.

\hat\sigma^2=\tfrac{18}{5}=3.6

,

\hat\sigma\approx1.897

.

Step 3 — N − 1 (b).

\tfrac{18}{4}=4.5

.

Step 4 — which is biased (c). The MLE, 3.6. It measures the spread around

\hat\mu=4

, which was chosen to be as close to these very points as possible, so the points look slightly tighter than they would around the true middle. Dividing by

N-1

undoes that on average.

answers at a glance: (a) $\hat\mu=4$ , $\hat\sigma^2=3.6$ ; (b) $4.5$ ; (c) the MLE is slightly too small on average.

Problem 8mediumleast squares via likelihood

Three points: $(1,2)$ , $(2,3)$ , $(3,5)$ . Assume $y=wx+b+\varepsilon$ with $\varepsilon\sim\mathcal N(0,\sigma^2)$ . (a) Why does maximising the likelihood mean minimising $\sum(y_i-\hat y_i)^2$ ? (b) Find $w$ and $b$ . (c) Find the squared error. (d) The maximum-likelihood noise variance is $\hat\sigma^2=\tfrac1N\sum(y_i-\hat y_i)^2$ . Compute it.

What this tests. The chain "bell noise → −log L → squared error → the least-squares formulas". Plan. Write $-\log L$ ; use the centred slope formula.

Show the full solution

Step 1 — why (a).

-\log L=\tfrac1{2\sigma^2}\sum(y_i-\hat y_i)^2+N\log(\sigma\sqrt{2\pi})

. For any fixed

\sigma

the last term is fixed and

\tfrac1{2\sigma^2}>0

, so the line that makes

-\log L

smallest is the one with the smallest squared error.

Step 2 — the line (b).

\bar x=2

,

\bar y=\tfrac{10}3

. Deviations:

x

:

-1,0,1

;

y

:

-\tfrac43,-\tfrac13,\tfrac53

.

\begin{aligned}w&=\frac{\tfrac43+0+\tfrac53}{1+0+1}=\frac32,\\ b&=\frac{10}3-\frac32\cdot2=\frac13.\end{aligned}

Step 3 — the error (c). Predictions

\tfrac{11}6,\tfrac{10}3,\tfrac{29}6

; errors

\tfrac16,-\tfrac13,\tfrac16

; squared error

\tfrac1{36}+\tfrac4{36}+\tfrac1{36}=\tfrac16\approx0.167

.

Step 4 — the noise (d).

\hat\sigma^2=\tfrac{1/6}{3}=\tfrac1{18}\approx0.0556

.

answers at a glance: (a) the other term of $-\log L$ does not depend on the line; (b) $w=1.5$ , $b=\tfrac13$ ; (c) $\tfrac16\approx0.167$ ; (d) $\hat\sigma^2=\tfrac1{18}\approx0.0556$ .

Problem 9easysoftmax · temperature

Scores $(3,1,1)$ . (a) Compute the softmax at $T=1$ . (b) At $T=2$ . (c) What happens as $T\to0$ and as $T\to\infty$ ? (d) What does softmax give for $(103,101,101)$ ?

What this tests. The recipe, the dial and shift invariance. Plan. Exponentiate, add, divide — after dividing the scores by $T$ .

Show the full solution

Step 1 — T = 1 (a).

e^3=20.086

,

e^1=2.718

,

e^1=2.718

; total

25.522

. Probabilities

(0.787,\,0.107,\,0.107)

.

Step 2 — T = 2 (b). Scores become

(1.5,0.5,0.5)

:

4.482,\,1.649,\,1.649

, total

7.779

. Probabilities

(0.576,\,0.212,\,0.212)

— more relaxed.

Step 3 — the ends (c).

T\to0

: all the probability goes to the first class,

(1,0,0)

.

T\to\infty

:

(\tfrac13,\tfrac13,\tfrac13)

. The two tied classes stay tied at every temperature.

Step 4 — shift (d). Adding 100 to every score changes nothing:

(0.787,\,0.107,\,0.107)

. Computing

e^{103}

directly would be foolish; subtract the max first.

answers at a glance: (a) $(0.787,0.107,0.107)$ ; (b) $(0.576,0.212,0.212)$ ; (c) $(1,0,0)$ and $(\tfrac13,\tfrac13,\tfrac13)$ ; (d) same as (a).

Problem 10easytwo classes · sigmoid

A two-class model gives scores $z_1=2.5$ , $z_2=1$ . (a) Compute $q_1$ with softmax. (b) Compute $\sigma(z_1-z_2)$ and compare. (c) An SVM gives a new point $f(\mathbf x)=-1.5$ . What probability of class $+1$ does $\sigma(f)$ suggest? (d) What does a point on the SVM's boundary get?

What this tests. That two-class softmax is the sigmoid of the score difference. Plan. Compute both ways; then read an SVM score as a difference.

Show the full solution

Step 1 — softmax (a).

e^{2.5}=12.182

,

e^{1}=2.718

:

q_1=\tfrac{12.182}{14.901}\approx0.818

.

Step 2 — sigmoid (b).

\sigma(1.5)=\tfrac1{1+e^{-1.5}}=\tfrac1{1+0.223}\approx0.818

. The same number.

Step 3 — the SVM score (c).

\sigma(-1.5)\approx0.182

: about an 18% chance of class

+1

, 82% of class

-1

.

Step 4 — the boundary (d).

\sigma(0)=0.5

: exactly undecided.

answers at a glance: (a) and (b) $0.818$ ; (c) $0.182$ ; (d) $0.5$ .

Remember

In practice the SVM score is first scaled and shifted ( $\sigma(af+b)$ , fitted on held-out data) before it is trusted as a probability — but the idea is exactly this.

Problem 11easysurprise · entropy

A four-outcome distribution is $(\tfrac12,\tfrac14,\tfrac18,\tfrac18)$ . (a) Find the surprise of each outcome in bits. (b) Find the entropy in bits and in nats. (c) Compare with four equally likely outcomes.

What this tests. Surprise, average surprise, and the units. Plan. $-\log_2$ of each chance; weight by the chance; multiply by $\ln2$ for nats.

Show the full solution

Step 1 — surprises (a).

1,\,2,\,3,\,3

bits.

Step 2 — entropy (b).

\tfrac12\cdot1+\tfrac14\cdot2+\tfrac18\cdot3+\tfrac18\cdot3=0.5+0.5+0.375+0.375=1.75

bits

=1.75\ln2\approx1.213

nats.

Step 3 — compare (c). Four equal outcomes: 2 bits. The lopsided one is more predictable, so it carries less uncertainty. A neat reading: ask "is it the first?" (yes half the time), then "the second?", then "the third?" — on average 1.75 questions.

answers at a glance: (a) 1, 2, 3, 3 bits; (b) 1.75 bits ≈ 1.213 nats; (c) less than the 2 bits of the uniform.

Problem 12mediumcross-entropy · both KLs

The truth is $p=(0.25,0.75)$ and a model believes $q=(0.5,0.5)$ . In bits: (a) find $H(p)$ ; (b) find $H(p,q)$ ; (c) find $\mathrm{KL}(p\,\|\,q)$ two ways; (d) find $\mathrm{KL}(q\,\|\,p)$ .

What this tests. The identity $H(p,q)=H(p)+\mathrm{KL}$ and the lopsidedness of KL. Plan. Compute each sum directly, then check the identity.

Show the full solution

Step 1 — entropy (a).

H(p)=0.25\cdot2+0.75\log_2\tfrac43=0.5+0.311=0.811

.

Step 2 — cross-entropy (b).

H(p,q)=-(0.25+0.75)\log_20.5=1

.

Step 3 — KL two ways (c). Directly:

0.25\log_2\tfrac{0.25}{0.5}+0.75\log_2\tfrac{0.75}{0.5}=-0.25+0.439=0.189

. From the identity:

1-0.811=0.189

✓.

Step 4 — the other direction (d).

0.5\log_2\tfrac{0.5}{0.25}+0.5\log_2\tfrac{0.5}{0.75}=0.5-0.292=0.208

. Not the same as 0.189.

answers at a glance: (a) 0.811; (b) 1; (c) 0.189; (d) 0.208 bits.

Problem 13hardcross-entropy loss · q − y

A three-class model gives scores $\mathbf z=(0,1,3)$ . The true class is the second. (a) Compute $\mathbf q$ . (b) Compute the loss in nats. (c) Compute the gradient with respect to $\mathbf z$ , and say what one step of size 1 does to each score. (d) Take that step and recompute the loss.

What this tests. The loss of a confidently wrong model, and one step of the cleanest gradient in the course. Plan. Softmax; $-\ln q_2$ ; $\mathbf q-\mathbf y$ ; $\mathbf z\leftarrow\mathbf z-\eta(\mathbf q-\mathbf y)$ .

Show the full solution

Step 1 — the belief (a).

e^0=1

,

e^1=2.718

,

e^3=20.086

; total

23.804

.

\mathbf q=(0.042,\,0.114,\,0.844)

. The model is fairly sure of the third class.

Step 2 — the loss (b).

-\ln0.114\approx2.170

nats. Large: the truth got only 11% belief.

Step 3 — the gradient (c).

\mathbf y=(0,1,0)

:

\begin{gathered}\mathbf q-\mathbf y\\ =(0.042,\;-0.886,\;0.844).\end{gathered}

A step downhill adds

0.886

to the true score, takes

0.844

off the third (the class that stole the most), and

0.042

off the first.

Step 4 — after the step (d).

\mathbf z=(-0.042,\,1.886,\,2.156)

, so

\mathbf q=(0.059,\,0.407,\,0.534)

and the loss falls to

-\ln0.407\approx0.899

.

answers at a glance: (a) $(0.042,0.114,0.844)$ ; (b) $2.170$ nats; (c) $(0.042,-0.886,0.844)$ ; (d) loss $\approx0.899$ .

Remember

The gradient's entries always add to zero, and each wrong class is pushed down by exactly the probability it wrongly held. That is the whole training signal of a classifier.

Problem 14mediumwhich loss from which noise

Five recorded auto fares for the same short ride, in tens of rupees: 1, 2, 3, 4 and — one mistyped — 100. You model them as one true fare $c$ plus noise. (a) With bell-shaped noise, which loss do you minimise, and what is $\hat c$ ? (b) With Laplace noise? (c) Which estimate would you trust, and why?

What this tests. That a loss is a noise belief, with a definite numerical answer for each. Plan. Bell → squared error → mean; Laplace → absolute error → median.

Show the full solution

Step 1 — bell noise (a).

-\log L

is the squared error

\sum(y_i-c)^2

, smallest at the mean:

\hat c=\tfrac{1+2+3+4+100}{5}=22

.

Step 2 — Laplace noise (b).

-\log L

is the absolute error

\sum|y_i-c|

. Its slope is (number of points below

c

) − (number above), which is zero at the median:

\hat c=3

.

Step 3 — judge (c). The median, 3: four of the five fares agree with it. The mean, 22, is nowhere near any real fare — one wild value dragged it, because a bell says such a value is practically impossible and bends everything to explain it.

answers at a glance: (a) squared error, $\hat c=22$ (the mean); (b) absolute error, $\hat c=3$ (the median); (c) trust the median.

Remember

Choosing a loss is choosing a noise model. If wild values can happen, say so with a heavy-tailed noise — and the loss will follow.

Next up

Unit 15 · The Network, Whole →

Every tool of this unit — softmax, cross-entropy, likelihood — now gets wired into a full network. You will build one yourself and watch the forward pass carry an input to a probability, and the backward pass carry "prediction minus truth" back through every layer, node by node.

← Unit 13 · Support Vector Machines · All units