Unit 8 · Taylor & MacLaurin Series — The Math Behind the Machine

1

Every smooth curve is secretly a polynomial — up close

Imagine this

A master forger wants to copy a signature. She cannot see all of it at once, but she can study it closely at one point. First she puts a dot in the right place. Then she draws a short straight stroke at the right angle. Then she bends the stroke to match the curve. Each extra piece of information — position, slope, bend, the change in the bend — makes the copy hug the original for longer. The Taylor polynomial is that forger: it copies a function's value and its first few derivatives at one point, and out comes a polynomial that is almost indistinguishable from the function nearby.

You built that impostor in Unit 6. Anchor at a point $a$ , copy the derivatives, and divide each by $k!$ so the copies do not disturb each other:

P_{n}(x)=f(a)+f'(a)(x-a)+\frac{f''(a)}{2!}(x-a)^2+\cdots+\frac{f^{(n)}(a)}{n!}(x-a)^n

Two questions were left open, and they are the whole of this unit. First: why does this work at all? Matching derivatives at one point is a plausible idea, but plausible is not the same as proved — what guarantees that the copy is close to the original one step away from the anchor? Second: how big is the lie? If you cut the series after a few terms, you have made an error. A formula for that error — not an estimate, an exact expression — is the difference between an approximation you hope is good and one you know is good.

Both answers come from the same humble place: a flat spot on a hiking trail. From that flat spot we will climb a ladder of derivatives and arrive at Taylor's theorem with its remainder. Then we take the theorem into two variables, where the second-derivative term turns into a small matrix — and that matrix turns out to be the judge of hills and valleys.

Pause & predict

You approximate $\sqrt{4.2}$ with the tangent line of $\sqrt{x}$ at $x=4$ and get $2.05$ . The true value is $2.0494$ . What should the next term of the polynomial do?

In one sentence: A Taylor polynomial copies a function's derivatives at one point; this unit proves why that works and measures exactly how far the copy can be trusted.

2

The flat spot — Rolle's theorem

The mountain trail

You hike a smooth trail. You start at 1000 m and, hours later, you finish at 1000 m again. Somewhere in between — at the top of a climb, or the bottom of a dip, or on a level stretch — the ground under your feet was perfectly flat. Not approximately. Exactly. You may not know where, but you know it happened, because you cannot go up and come back down without, at some instant, going neither up nor down.

That is Rolle's theorem, and it is the seed from which everything in this unit grows:

Rolle's theorem

If $f$ is continuous on $[a,b]$ (no gaps), differentiable on $(a,b)$ (no sharp corners), and $f(a)=f(b)$ (same height at both ends), then there is at least one $c$ strictly between $a$ and $b$ with

f'(c)=0.

Each hypothesis is a real part of the promise. Drop continuous and the trail can jump over the flat spot. Drop differentiable and the trail can turn around at a sharp corner — a kink like $|x|$ has a lowest point but no tangent there, so no $c$ with $f'(c)=0$ exists. The lecture's warning is exactly this: Rolle's theorem is not valid for functions with a kink. Hunt for the flat spot yourself, and then try the trail that breaks the rule:

Try: press ▶ hunt for c and watch the tangent slide until it is parallel to the chord. Then open the broken tab: same heights at both ends, and no flat spot anywhere.

left end a-1.6

right end b1.8

—

The chord's slope is the average rate of change over $[a,b]$ ; the tangent's slope is the instantaneous rate at one point. The theorem says the two must agree somewhere strictly inside. In the Rolle tab the right end is pinned to the same height as the left, so the chord is level and the promised tangent is flat.

Pause & predict

$f(x)=|x|$ on $[-1,1]$ : continuous, $f(-1)=f(1)=1$ , and it clearly has a lowest point at $0$ . Why does Rolle's theorem not hand you a $c$ with $f'(c)=0$ ?

In one sentence: Same height at both ends plus a smooth trail guarantees a flat spot somewhere strictly in between — and a kink voids the guarantee.

3

Average equals instantaneous — the mean value theorem

The highway trip

You drive 200 km in 2 hours. Your average speed was 100 km/h. At some moment your speedometer must have read exactly 100 km/h — you cannot average 100 by always being faster, nor by always being slower. The mean value theorem is this sentence, written for any smooth function: somewhere in the interval, the instantaneous slope equals the average slope.

Mean value theorem

If $f$ is continuous on $[a,b]$ and differentiable on $(a,b)$ , there is a $c$ in $(a,b)$ with

f'(c)=\frac{f(b)-f(a)}{b-a}.

Rolle is the special case where the average slope is zero. The surprise is that the general case is also Rolle — tilted. Subtract the chord from the function and the two ends line up:

Prove it · the mean value theorem, from Rolle

Claim. If $f$ is continuous on $[a,b]$ and differentiable on $(a,b)$ , some $c\in(a,b)$ has $f'(c)=\dfrac{f(b)-f(a)}{b-a}$ .

1

Name the average slope and build the chord — the straight line through the two endpoints:

m=\frac{f(b)-f(a)}{b-a},\qquad \ell(x)=f(a)+m\,(x-a)

\ell

is a straight line with slope

m

that passes through

(a,f(a))

; by the choice of

m

it also passes through

(b,f(b))

.

2

Subtract the chord from the function:

g(x)=f(x)-\ell(x)=f(x)-f(a)-m\,(x-a)

g

is "how far

f

sits above its own chord". It inherits continuity and differentiability from

f

, because

\ell

is as smooth as anything can be.

3

Check the two ends:

g(a)=f(a)-f(a)-0=0,\qquad g(b)=f(b)-f(a)-m\,(b-a)=f(b)-f(a)-\big(f(b)-f(a)\big)=0

The chord meets the curve at both ends, so the gap is zero at both ends. That is exactly Rolle's "same height" hypothesis — for

g

.

4

Apply Rolle to

g

: some

c\in(a,b)

has

g'(c)=0

. But

g'(x)=f'(x)-m

, so

0=g'(c)=f'(c)-m\quad\Longrightarrow\quad f'(c)=m=\frac{f(b)-f(a)}{b-a}.

That is the claim. ∎ Tilting the picture by a straight line does not change where tangents are parallel to it. So the general theorem is the flat one in disguise.

Now rearrange the mean value theorem and read it again, slowly:

f(b)=f(a)+f'(c)\,(b-a)

This says: the value at $b$ equals the value at $a$ , plus a correction — the distance travelled times a slope taken somewhere in between. That is a first-order Taylor expansion with the error built in. The tangent-line estimate would use $f'(a)$ ; the theorem says that if you are willing to use the slope at a secret point $c$ instead, the estimate becomes exact. Taylor's theorem is what you get when you ask the same question with more derivatives: copy $f$ 's slope, bend, and higher derivatives at $a$ , and let one secret $c$ absorb everything you left out.

Pause & predict

$f(x)=x^3$ on $[0,2]$ . The mean value theorem promises a $c$ with $f'(c)=\dfrac{f(2)-f(0)}{2-0}=4$ . Where is it?

In one sentence: Subtract the chord and the mean value theorem is Rolle; rearranged, it is already a first-order Taylor formula with an exact error term.

4

The Rolle ladder — one flat spot per rung

Here is the idea that turns one flat spot into a whole theorem. Suppose a function $F$ is very flat at $a$ — its value and its first $n-1$ derivatives all vanish there — and suppose it also vanishes at some $b>a$ :

F(a)=F'(a)=F''(a)=\cdots=F^{(n-1)}(a)=0,\qquad F(b)=0.

Then $F$ and $F'$ and $F''$ … each get a Rolle argument in turn, and the flat spots nest inward like Russian dolls. The lecture calls the conclusion Proposition P: there is a $c$ in $(a,b)$ with $F^{(n)}(c)=0$ . Climb the ladder and watch it happen:

Try: press ▶ climb and watch the zeros nest: $a<c_3<c_2<c_1<b$ . Then move $b$ closer to $a$ and see every $c_k$ get squeezed toward $a$ .

target b2

rungs n3

—

Here $F(x)=f(x)-P(x)$ , where $P$ copies $f$ 's value and first $n-1$ derivatives at $a=0$ , and its last coefficient is chosen so that $F(b)=0$ . Every rung starts with two zeros and Rolle plants one more, one level down — until the bottom rung reports $F^{(n)}(c_n)=0$ . That single zero is where the remainder formula comes from.

Prove it · Proposition P — n flat spots, nested

Claim. If $F(a)=F'(a)=\cdots=F^{(n-1)}(a)=0$ and $F(b)=0$ , with $F$ $n$ -times differentiable on $(a,b)$ and the lower derivatives continuous on $[a,b]$ , then some $c\in(a,b)$ has $F^{(n)}(c)=0$ .

1

Rung 1.

F(a)=0=F(b)

: Rolle gives

c_1\in(a,b)

with

F'(c_1)=0

. Two zeros of

F

at the ends, one flat spot of

F

in between. Standard Rolle.

2

Rung 2. Now look at

F'

. It vanishes at

a

(given) and at

c_1

(rung 1). Rolle on

[a,c_1]

gives

c_2\in(a,c_1)

with

F''(c_2)=0

. The zero handed down from rung 1 plus the zero we were given at

a

are the two ends Rolle needs. The interval has shrunk:

c_2<c_1

.

3

Rung $k$ , in general. Suppose

F^{(k-1)}(c_{k-1})=0

with

c_{k-1}\in(a,b)

, and

k-1\le n-1

so that

F^{(k-1)}(a)=0

too. Rolle on

[a,c_{k-1}]

gives

c_k\in(a,c_{k-1})\quad\text{with}\quad F^{(k)}(c_k)=0.

Same move each time: the zero at

a

is free (it was assumed), the other zero is inherited from the rung above. That is why we needed

n-1

vanishing derivatives at

a

— one for each rung after the first.

4

After

n

rungs:

a<c_n<c_{n-1}<\cdots<c_1<b,\qquad F^{(n)}(c_n)=0.

Set

c=c_n

. Done. ∎ Each rung uses one Rolle and shrinks the interval toward

a

. The last rung needs no zero at

a

for

F^{(n)}

— it only needs the inherited

c_{n-1}

and the free zero of

F^{(n-1)}

at

a

.

Pause & predict

In the ladder, why must the derivatives of $F$ vanish at $a$ up to order $n-1$ — why not just $F(a)=0$ ?

In one sentence: A function that is

n

-fold flat at

a

and zero at

b

has its

n

-th derivative vanishing somewhere in between — one Rolle per rung, nesting inward.

5

Taylor's theorem, built by hand

Proposition P wants a function that is $n$ -fold flat at $a$ and zero at $b$ . We have a function $f$ and a polynomial we would like to be its impostor,

P(x)=\sum_{k=0}^{n}a_k\,(x-a)^k,

with the coefficients $a_0,\dots,a_n$ still to be chosen. So the plan is: build the gap $F(x)=f(x)-P(x)$ , choose the coefficients so that $F$ satisfies Proposition P, and read off what the last flat spot tells us. Three moves. Each one is short.

The first $n$ coefficients make $F$ flat at $a$ . The last coefficient makes $F$ zero at $b$ . Proposition P then delivers a $c$ — and that $c$ is the remainder.

Prove it · move 1 — flattening the gap at a fixes the first n coefficients

Claim. $F^{(k)}(a)=f^{(k)}(a)-k!\,a_k$ for $k=0,1,\dots,n$ . So $F^{(k)}(a)=0$ exactly when $a_k=\dfrac{f^{(k)}(a)}{k!}$ .

1

Differentiate one power

j

times and evaluate at

a

:

\frac{d^{\,j}}{dx^{j}}(x-a)^k\Big|_{x=a}=\begin{cases}0 & j<k\\[.2em] k! & j=k\\[.2em] 0 & j>k\end{cases}

Fewer differentiations than the power (

j<k

): a positive power of

(x-a)

survives, and it is

0

at

a

. Exactly

k

differentiations: the power is used up and leaves

k(k-1)\cdots1=k!

. More than

k

: a constant differentiated again is

0

. Only the term whose power matches the number of differentiations survives at

a

.

2

So the

k

-th derivative of

P

at

a

picks out one term:

P^{(k)}(a)=\sum_{j=0}^{n}a_j\,\frac{d^{\,k}}{dx^{k}}(x-a)^j\Big|_{a}=a_k\cdot k!

Every other term in the sum is killed by step 1 — either it still has a factor

(x-a)

, or it has already been differentiated to zero.

3

Therefore

F^{(k)}(a)=f^{(k)}(a)-P^{(k)}(a)=f^{(k)}(a)-k!\,a_k

, and demanding

F^{(k)}(a)=0

for

k=0,\dots,n-1

forces

a_k=\frac{f^{(k)}(a)}{k!},\qquad k=0,1,\dots,n-1.\

That is the claim. ∎ This is where the Taylor coefficients come from — not from a formula handed down, but from the demand that the gap be flat. The

k!

is the factorial left behind by differentiating

(x-a)^k

k

times.

Move 1 has spent $n$ of the $n+1$ coefficients. One is left, $a_n$ , and Proposition P still wants $F(b)=0$ . That is one equation in one unknown:

Prove it · move 2 — the last coefficient makes the gap vanish at b

Claim. With $a_0,\dots,a_{n-1}$ fixed by move 1, $F(b)=0$ holds exactly when $a_n=\dfrac{1}{(b-a)^n}\Big(f(b)-\displaystyle\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k\Big).$

1

Write out

F(b)

, separating the last term:

F(b)=f(b)-\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k-a_n(b-a)^n

The first

n

coefficients are already known from move 1; only

a_n

is free.

2

Set

F(b)=0

and solve for

a_n

:

\begin{aligned}a_n(b-a)^n&=f(b)-\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k\\[.35em] a_n&=\frac{1}{(b-a)^n}\Big(f(b)-\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k\Big).\end{aligned}

That is the claim. ∎ Divide by

(b-a)^n

, which is not zero because

b\ne a

. Read it in words:

a_n

is "the gap left by the first

n

terms, per unit of

(b-a)^n

". We do not yet know what

a_n

has to do with

f

's derivatives — that is move 3.

The gap function, pinned. For

f=e^x

,

a=0

,

n=3

,

b=2

: the gap

F=f-P

leaves

0

perfectly flat (value, slope and bend all zero), dips, and returns to zero at

b

. The three nested flat spots are exactly where the ladder puts them.

Now the payoff. $F$ satisfies everything Proposition P asks for: flat to order $n-1$ at $a$ (move 1) and zero at $b$ (move 2). So there is a $c\in(a,b)$ with $F^{(n)}(c)=0$ . Unpack that one equation and the remainder falls out:

Prove it · move 3 — Taylor's theorem with the remainder

Claim. There is a $c$ strictly between $a$ and $b$ such that $f(b)=\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k+\frac{f^{(n)}(c)}{n!}(b-a)^n.$

1

Differentiate

F=f-P

exactly

n

times. By step 1 of move 1, the

n

-th derivative of

P

is the constant

n!\,a_n

(every lower power has been differentiated away, and there are no higher ones):

F^{(n)}(x)=f^{(n)}(x)-n!\,a_n\qquad\text{for every }x.

A degree-

n

polynomial differentiated

n

times is a constant: its leading coefficient times

n!

.

2

Proposition P gives

c\in(a,b)

with

F^{(n)}(c)=0

. At that

c

:

f^{(n)}(c)-n!\,a_n=0\quad\Longrightarrow\quad a_n=\frac{f^{(n)}(c)}{n!}.

The last coefficient — which move 2 defined as "the gap per unit

(b-a)^n

" — is revealed to be the

n

-th derivative at a secret point, divided by

n!

. It has the same shape as every other coefficient; only the point where the derivative is taken has moved from

a

to

c

.

3

Substitute this

a_n

into move 2's equation

F(b)=0

, i.e.

f(b)=\sum_{k<n}\frac{f^{(k)}(a)}{k!}(b-a)^k+a_n(b-a)^n

:

f(b)=\underbrace{\sum_{k=0}^{n-1}\frac{f^{(k)}(a)}{k!}(b-a)^k}_{\text{Taylor polynomial }P_{n-1}(b)}+\underbrace{\frac{f^{(n)}(c)}{n!}(b-a)^n}_{\text{remainder }R_n}.

That is the claim. ∎ The polynomial part uses derivatives at

a

— things you can compute. The remainder uses one derivative at an unknown

c

— which you cannot compute, but can bound. For

n=1

this is the mean value theorem,

f(b)=f(a)+f'(c)(b-a)

; the whole construction is that theorem, repeated up a ladder.

Read it as a promise

Replace $b$ by any $x$ in the interval and the theorem reads: $f(x)=P_{n-1}(x)+R_n(x)$ , exactly, for every $x$ — the impostor plus a lie whose size is $\dfrac{f^{(n)}(c)}{n!}(x-a)^n$ for some $c$ between $a$ and $x$ . The lie has two parts you can see: $(x-a)^n$ , which is tiny close to the anchor and grows fast away from it; and $1/n!$ , which shrinks ferociously as you keep more terms. The one part you cannot see, $f^{(n)}(c)$ , is trapped between the smallest and largest values of $f^{(n)}$ on the interval. That is enough to put a number on it, and the next section does.

Pause & predict

In the construction, $a_n$ was first defined (move 2) as a quotient involving $f(b)$ , then revealed (move 3) to equal $f^{(n)}(c)/n!$ . What did the theorem actually prove?

Pause & predict

Take $f(x)=x^3$ , $a=0$ , $n=2$ (keep only $P_1$ ), and $b=2$ . Since $f(0)=f'(0)=0$ , $P_1\equiv0$ and the whole value $f(2)=8$ is remainder: $8=\dfrac{f''(c)}{2!}\cdot2^2$ . What is $c$ ?

In one sentence: Choose the polynomial so the gap is flat at

a

and zero at

b

; Proposition P then forces the last coefficient to be

f^{(n)}(c)/n!

, which is Taylor's remainder.

6

How big is the lie? — the remainder in your hands

You will never know $c$ . But you do not need to. If $|f^{(n)}|$ never exceeds some number $M$ on the stretch between $a$ and $x$ , then whatever $c$ is,

|R_n(x)|=\frac{|f^{(n)}(c)|}{n!}\,|x-a|^n\ \le\ \frac{M}{n!}\,|x-a|^n.

That is a guarantee you can compute before you trust an approximation — and it is usually pessimistic by only a little, because the true $f^{(n)}(c)$ tends to sit close to the values at the ends. The detective below does both jobs: it computes the guaranteed band, and then it hunts down the actual $c$ that makes the formula exact.

Try: load √4.2 (Problem A3), then slide the order from 2 to 3 (the tangent line becomes the parabola) and watch the gap collapse from $0.0006$ to $0.000015$ . Then push $x$ far from the anchor and watch the bound balloon.

order n2

evaluate at x1

—

Keep the terms up to degree $n-1$ ; the lie is exactly $\dfrac{f^{(n)}(c)}{n!}(x-a)^n$ for one secret $c$ between $a$ and $x$ . The detective solves for that $c$ numerically and marks it. The translucent band is the guarantee you can compute without knowing $c$ : the worst $|f^{(n)}|$ on the interval, times $|x-a|^n/n!$ .

Prove it · the remainder bound

Claim. If $|f^{(n)}(t)|\le M$ for every $t$ between $a$ and $x$ , then $|f(x)-P_{n-1}(x)|\le\dfrac{M}{n!}|x-a|^n$ .

1

Taylor's theorem (with

b=x

) gives a

c

between

a

and

x

with

f(x)-P_{n-1}(x)=\frac{f^{(n)}(c)}{n!}(x-a)^n.

This is an equality, not an estimate. Everything uncertain is inside

f^{(n)}(c)

.

2

Take absolute values;

c

is between

a

and

x

, so the hypothesis applies to it:

|f(x)-P_{n-1}(x)|=\frac{|f^{(n)}(c)|}{n!}|x-a|^n\le\frac{M}{n!}|x-a|^n.

That is the claim. ∎ The only information used about

c

is where it lives. Whatever it is, the derivative there is at most

M

in size.

Two examples worth memorising

Small angles. $\cos x=1-\dfrac{x^2}{2}+R_4$ , with $|R_4|\le\dfrac{x^4}{24}$ because every derivative of $\cos$ is bounded by $1$ . At $x=0.5$ the bound is $0.0026$ ; the actual error is $0.00258$ . (Why $n=4$ and not $3$ ? The $x^3$ term of $\cos$ is zero, so $1-x^2/2$ is really $P_3$ , and the first thing left out is order $4$ .) The number $e$ . $e=\sum_{k<n}\dfrac{1}{k!}+R_n$ with $|R_n|\le\dfrac{e}{n!}<\dfrac{3}{n!}$ . Ten terms ( $n=10$ ) give $3/10!\approx8\times10^{-7}$ : six correct decimals from a sum you could do by hand. Practice Problem 4 works both in full.

Pause & predict

You approximate $\sin x$ by $x$ (that is $P_2$ : the $x^2$ term is zero). For $x=0.3$ , what is the tightest guarantee the remainder bound gives?

In one sentence: You cannot see

c

, but

|f^{(n)}|\le M

on the interval turns the exact remainder into a computable guarantee,

|R_n|\le M|x-a|^n/n!

.

7

MacLaurin's series — and why the factorial always wins

Anchor the polynomial at $a=0$ and it is called a MacLaurin series. Two functions make the pattern unforgettable. For $e^x$ every derivative is $e^x$ again, so every derivative at $0$ is $1$ :

e^x=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\cdots=\sum_{k=0}^{\infty}\frac{x^k}{k!}

For $\cos x$ the derivatives at $0$ cycle through $1,\,0,\,-1,\,0,\,1,\dots$ , so only even powers survive and the signs alternate:

\cos x=1-\frac{x^2}{2!}+\frac{x^4}{4!}-\frac{x^6}{6!}+\cdots

The word "series" hides a question: does the infinite sum actually reach the function, or only get close and then wander off? The remainder answers it. For $e^x$ on any interval $|t|\le|x|$ , $|f^{(n)}(t)|=e^t\le e^{|x|}$ , so $|R_n|\le e^{|x|}\dfrac{|x|^n}{n!}$ . Whether that goes to zero is a race between a power $|x|^n$ and a factorial $n!$ — and the factorial always wins:

Try: push $x$ to 5. The terms grow for a while (the hump), then $n!$ overtakes $5^n$ and they crash to nothing. Compare with ln(1+x): past $x=1$ the bars never come down.

x2

terms kept4

—

Term $n$ has size $|x|^n/n!$ (times a bounded derivative). A power $|x|^n$ multiplies by the same $|x|$ every step; a factorial multiplies by $n$ , which eventually dwarfs any fixed $|x|$ . That race is the whole reason the exponential and trigonometric series converge everywhere — and $\ln(1+x)$ , whose terms shrink only like $|x|^n/n$ , loses the race the moment $|x|>1$ .

Prove it · the factorial beats every power

Claim. For any fixed $x$ , $\dfrac{|x|^n}{n!}\to0$ as $n\to\infty$ . Hence the MacLaurin series of $e^x$ , $\sin x$ and $\cos x$ converge to the function at every $x$ .

1

Look at how each term grows into the next:

\frac{|x|^{n+1}/(n+1)!}{|x|^n/n!}=\frac{|x|}{n+1}.

Going from term

n

to term

n+1

multiplies the power by

|x|

(the same number every time) and the factorial by

n+1

(a number that keeps growing).

2

Once

n+1\ge2|x|

, each step multiplies the term by at most

\tfrac12

. So beyond that point the terms shrink at least as fast as

\tfrac12,\tfrac14,\tfrac18,\dots

times a fixed number — and that goes to zero. Before the crossover the terms may grow (the "hump" in the widget: for

x=5

the largest term is at

n=4

or

5

); after it they are halved, at least, every step.

3

For

e^x

:

|R_n|\le e^{|x|}|x|^n/n!\to0

. For

\sin x

and

\cos x

: every derivative is bounded by

1

, so

|R_n|\le|x|^n/n!\to0

. In each case

P_{n-1}(x)\to f(x)

: the series converges to the function everywhere. ∎ Contrast

\ln(1+x)=x-\tfrac{x^2}{2}+\tfrac{x^3}{3}-\cdots

: its terms shrink only like

|x|^n/n

— no factorial in the denominator — and for

|x|>1

they grow. Taylor's remainder is what tells you the difference between "trust it everywhere" and "trust it only near the anchor". Unit 6 saw the cliff at

x=1

; now you know why it is there.

Pause & predict

For $e^x$ at $x=5$ , the terms $5^n/n!$ first grow, then shrink. At which $n$ is the largest term?

In one sentence: MacLaurin is Taylor at

0

; the terms of

e^x

,

\sin x

,

\cos x

may hump but always crash, because

n!

eventually multiplies faster than any fixed

|x|

can.

8

Two variables — walk along a line and it is one variable again

The hilly landscape

Now the function is a landscape, $z=f(x,y)$ . You stand at $P=(a,b)$ and want to know the height at a nearby point $S=(a+h,\,b+k)$ . Everything you have built so far is about one variable. So do the simplest possible thing: walk from $P$ to $S$ in a straight line, and record your height as you go. Height-versus-time on that walk is an ordinary one-variable function — and one-variable Taylor applies to it, ladder, remainder and all.

Parameterise the segment by a clock $t$ running from $0$ at $P$ to $1$ at $S$ : $x=a+th,\ y=b+tk$ . The height along the walk is

F(t)=f(a+th,\;b+tk),\qquad F(0)=f(a,b),\quad F(1)=f(a+h,b+k).

Two derivatives of $F$ are all we need, and the chain rule delivers both:

Try: load Problem A2 and read the gap between the true height and the quadratic clone at $S=(0.1,0.2)$ : it is $0.00044$ . Then double $h$ and $k$ — the gap grows about tenfold: eightfold from the third-order leftover, the rest from fourth-order crumbs.

drag the picture to orbit

base a0

base b0

step h0.5

step k0.6

—

The surface is $f(x,y)=e^x\sin y$ . Along the segment, $F(t)=f(a+th,\,b+tk)$ is an ordinary one-variable function of $t$ , so one-variable Taylor applies to it — and its second derivative is exactly $h^2f_{xx}+2hkf_{xy}+k^2f_{yy}$ . The glass surface is the quadratic clone $Q$ built from those three numbers.

Prove it · the first derivative along the walk

Claim. $F'(t)=h\,f_x+k\,f_y$ , the partials evaluated at the current point $(a+th,\,b+tk)$ .

1

F

is

f

composed with the straight-line map

t\mapsto(x(t),y(t))

. Unit 6's chain rule for a path says

F'(t)=f_x\,\frac{dx}{dt}+f_y\,\frac{dy}{dt}.

Two routes from

t

to

F

— through

x

and through

y

— multiply along each route and add across them (Unit 7's two rules).

2

On a straight line the velocities are constants:

dx/dt=h

,

dy/dt=k

. So

F'(t)=h\,f_x(a+th,\,b+tk)+k\,f_y(a+th,\,b+tk).

That is the claim. ∎ This is the directional derivative of Unit 6 in the direction

(h,k)

: steepness of the landscape as felt by a walker moving with velocity

(h,k)

.

Prove it · the second derivative along the walk

Claim. $F''(t)=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}$ , again at the current point.

1

F'=hf_x+kf_y

is itself a function of the position on the walk, so apply the same chain rule to it:

F''(t)=\frac{\partial F'}{\partial x}\frac{dx}{dt}+\frac{\partial F'}{\partial y}\frac{dy}{dt}=\frac{\partial(hf_x+kf_y)}{\partial x}\,h+\frac{\partial(hf_x+kf_y)}{\partial y}\,k.

Same two routes, same two constant velocities

h

and

k

.

2

Differentiate inside the brackets,

h

and

k

being constants:

\frac{\partial(hf_x+kf_y)}{\partial x}=hf_{xx}+kf_{yx},\qquad \frac{\partial(hf_x+kf_y)}{\partial y}=hf_{xy}+kf_{yy}.

f_{yx}

means "first

y

, then

x

". For a twice continuously differentiable

f

, Clairaut's theorem says the order does not matter:

f_{yx}=f_{xy}

.

3

Multiply out and collect, using

f_{yx}=f_{xy}

:

F''(t)=h\,(hf_{xx}+kf_{xy})+k\,(hf_{xy}+kf_{yy})=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}.

That is the claim. ∎ A quadratic form in

(h,k)

: the three second partials are its coefficients. This single line is the birthplace of the Hessian.

$F$ is an ordinary function on $[0,1]$ , continuous with a continuous derivative and twice differentiable inside, so Taylor's theorem with $n=2$ , $a=0$ , $b=1$ applies to it word for word. Translate back to $x$ and $y$ and the two-variable expansion appears:

Prove it · Taylor in two variables, with its remainder

Claim. For some $c\in(0,1)$ , $f(a+h,\,b+k)=f(a,b)+h\,f_x(a,b)+k\,f_y(a,b)+\tfrac12\big(h^2f_{xx}+2hkf_{xy}+k^2f_{yy}\big)\Big|_{(a+ch,\ b+ck)}.$

1

One-variable Taylor for

F

on

[0,1]

with

n=2

:

F(1)=F(0)+F'(0)\,(1-0)+\frac{F''(c)}{2!}\,(1-0)^2,\qquad c\in(0,1).

The theorem of the previous sections, applied to the walk. The remainder is evaluated at a secret clock-time

c

— i.e. at a secret point on the segment.

2

Substitute the three ingredients:

F(1)=f(a+h,b+k)

,

F(0)=f(a,b)

,

F'(0)=hf_x(a,b)+kf_y(a,b)

, and

F''(c)=(h^2f_{xx}+2hkf_{xy}+k^2f_{yy})

evaluated at

(a+ch,\,b+ck)

. That is the claim. ∎ The first-order part is Unit 6's tangent plane. The second-order part is new: the landscape's curvature, packed into three numbers and one quadratic form. Replace the secret point by

(a,b)

and you get the quadratic clone

Q

— the glass surface in the widget — with a third-order leftover, exactly as in one variable.

Pause & predict

On the widget's surface $f=e^x\sin y$ , start at $P=(0,0)$ and take the step $(h,k)=(0.1,0.2)$ . The quadratic clone gives $Q=0.22$ ; the truth is $0.21956$ , a gap of $4.4\times10^{-4}$ . Double the step to $(0.2,0.4)$ . Roughly what happens to the gap?

Pause & predict

Why does the cross term appear as $2hk\,f_{xy}$ — with a $2$ — while the pure terms are $h^2f_{xx}$ and $k^2f_{yy}$ ?

In one sentence: Along a straight line the landscape is a one-variable function

F(t)

; its second derivative

h^2f_{xx}+2hkf_{xy}+k^2f_{yy}

is where the Hessian is born.

9

At a flat point, the second-order term decides

Optimisation lives at flat points. A critical point $(a,b)$ is one where the gradient vanishes, $f_x(a,b)=f_y(a,b)=0$ — the tangent plane is level, and the first-order term of the expansion is zero. Every step you take from there changes the height only through the second-order term:

f(a+h,\,b+k)-f(a,b)=\tfrac12\,\underbrace{\big(h^2f_{xx}+2hkf_{xy}+k^2f_{yy}\big)\Big|_{(a+ch,\,b+ck)}}_{Q(c)}

Read the left side: it is how much higher or lower you are after the step. If it is positive for every small step, you were at the bottom of a bowl — a local minimum. Negative for every small step: the top of a dome — a local maximum. Positive for some steps and negative for others: a saddle. So the whole question is the sign of $Q$ .

The step, seen from above. From a flat point

P

, a step

(h,k)

to

S

changes the height by

\tfrac12Q

, where the quadratic form is evaluated at a secret point on the segment. Whether that is up or down is the question the Hessian will answer.

One subtlety before the judge takes the stand. $Q$ is evaluated at the secret point, not at $(a,b)$ itself. The lecture's resolution is exactly right: if $Q(0)$ — the form evaluated at $(a,b)$ — is not zero, then for small enough steps $Q(c)$ has the same sign as $Q(0)$ . So the sign of the change is decided by the three numbers $f_{xx},f_{xy},f_{yy}$ at the critical point. Here is why the "small enough" is not a loophole:

Prove it · the secret point cannot flip the sign

Claim. Suppose the second partials are continuous and $Q_0(h,k)=h^2f_{xx}(a,b)+2hkf_{xy}(a,b)+k^2f_{yy}(a,b)$ is positive for every direction $(h,k)\ne(0,0)$ . Then for all sufficiently small steps, $f(a+h,b+k)-f(a,b)>0$ . (The negative case is identical with signs flipped.)

1

Scale out the step size. Write

(h,k)=r(u,v)

with

u^2+v^2=1

and

r>0

. Then

Q_0(h,k)=r^2\,Q_0(u,v)

, and

Q_0(u,v)

is a continuous function on the unit circle, which is closed and bounded, so it attains a minimum

m

. Because

Q_0>0

at every point of the circle,

m>0

. A quadratic form's sign does not depend on the length of the step, only on its direction. The minimum over directions exists and is positive — that is the number the argument hangs on.

2

Compare the secret point with the anchor. The coefficients

f_{xx},f_{xy},f_{yy}

are continuous, so there is a radius

\rho

such that within distance

\rho

of

(a,b)

each coefficient differs from its value at

(a,b)

by less than

m/4

. For the form at the secret point this gives, for any unit direction,

\big|Q_{c}(u,v)-Q_0(u,v)\big|\le|u^2|\tfrac m4+|2uv|\tfrac m4+|v^2|\tfrac m4\le\tfrac m4(1+1+1)<m.

Using

u^2\le1

,

v^2\le1

,

|2uv|\le u^2+v^2=1

. The secret point lies on the segment, hence within

r<\rho

of the anchor, so the estimate applies to it.

3

Hence

Q_c(u,v)>Q_0(u,v)-m\ge m-m=0

for every direction, and so

f(a+h,b+k)-f(a,b)=\tfrac12\,r^2\,Q_c(u,v)>0\qquad\text{whenever }0<r<\rho.

That is the claim. ∎ "Sufficiently small" means

r<\rho

, a definite radius that depends only on how fast the second partials change. Inside that radius, the anchor's three numbers rule.

Pause & predict

At a critical point, why can the first-order term never help you decide between a minimum and a maximum?

In one sentence: At a flat point the height change after a small step is

\tfrac12Q

, and continuity lets the three second partials at the point decide its sign.

10

Taylor in matrix form — and how to remember it

Three numbers decided everything in the last section, so give them a home. Pack the second partials into a matrix and the walk's second derivative becomes a sandwich:

H=\nabla^2 f=\begin{bmatrix}f_{xx} & f_{xy}\\ f_{xy} & f_{yy}\end{bmatrix}

\mathbf h^{\top}H\,\mathbf h=\begin{bmatrix}h & k\end{bmatrix}\begin{bmatrix}f_{xx} & f_{xy}\\ f_{xy} & f_{yy}\end{bmatrix}\begin{bmatrix}h\\ k\end{bmatrix}=h^2f_{xx}+2hk\,f_{xy}+k^2f_{yy}.

(Multiply it out: the two diagonal entries give $h^2f_{xx}$ and $k^2f_{yy}$ ; the two off-diagonal entries each give $hk\,f_{xy}$ , and together they are the $2hkf_{xy}$ from the walk.) The matrix $H$ is the Hessian — every second partial derivative in one place, symmetric because $f_{xy}=f_{yx}$ . With it, the whole two-variable expansion of the previous sections — first-order term included — collapses into one line that looks exactly like the one-variable formula:

Taylor's theorem in matrix form

For $f:\mathbb R^n\to\mathbb R$ , a base point $\mathbf x_0$ and a step $\mathbf h$ , with $\nabla f$ the gradient (a column of the $n$ first partials) and $H$ the Hessian (the $n\times n$ matrix $H_{ij}=\partial^2 f/\partial x_i\partial x_j$ ):

f(\mathbf x_0+\mathbf h)=f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}\mathbf h+\tfrac12\,\mathbf h^{\top}H(\mathbf x_0+c\,\mathbf h)\,\mathbf h\qquad\text{for some }c\in(0,1),

and, evaluating $H$ at $\mathbf x_0$ instead, the quadratic clone $Q(\mathbf h)=f(\mathbf x_0)+\nabla f^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H\mathbf h$ with a third-order leftover.

How to remember it — the rank ladder

Write the one-variable formula and promote each derivative by one rank:

\begin{array}{rcccccc} \text{one variable:} & f(a) & + & f'(a)\,h & + & \tfrac12\,f''(a)\,h^2 \\[.5em] \text{many variables:} & f(\mathbf x_0) & + & \nabla f^{\top}\mathbf h & + & \tfrac12\,\mathbf h^{\top}H\,\mathbf h \end{array}

The value is a number in both worlds — nothing to promote. The first derivative becomes a list of $n$ slopes, the gradient; a list needs exactly one step to turn into a number, so it is dotted with $\mathbf h$ : one $h$ , one index. The second derivative becomes an $n\times n$ table of curvatures, the Hessian; a table needs two steps to become a number, one for each index, so $\mathbf h$ is fed in from both sides: the sandwich. The $\tfrac12$ is $1/2!$ and stays exactly where it was — the first-order term's $1/1!$ is invisible for the same reason it was invisible in one variable. Two habits make it stick: each derivative index eats one copy of $\mathbf h$ ( $\sum_i f_i h_i$ , then $\sum_{i}\sum_j f_{ij}h_ih_j$ ), and every term must be a plain number — check the shapes: $(1\times n)(n\times1)$ is a number, $(1\times n)(n\times n)(n\times1)$ is a number. If you ever write $H\mathbf h^2$ or $\nabla f\,\mathbf h$ without a transpose, the shapes refuse to multiply, and that refusal is the formula correcting you.

The rank ladder. Promote each derivative by one rank — number, list, table — and feed it as many copies of the step

\mathbf h

as it has indices. The shapes do the remembering for you.

Try: hover the two off-diagonal entries: they feed the same product $h_1h_2$ , which is where the $2$ in $2hk\,f_{xy}$ comes from. Switch to three variables and count — 9 entries, 6 distinct numbers, and every term still a plain number.

—

Row × matrix × column is a double sum: $\mathbf h^{\top}H\mathbf h=\sum_i\sum_j h_i\,H_{ij}\,h_j$ . Each entry of $H$ feeds one term, and each term uses the step component of its row and of its column. Because $H$ is symmetric, the mirror-image entries $H_{ij}$ and $H_{ji}$ feed the same product and merge — for two variables that merge is the familiar $h^2f_{xx}+2hkf_{xy}+k^2f_{yy}$ .

Prove it · the matrix form, from the line trick, for any number of variables

Claim. For $F(t)=f(\mathbf x_0+t\mathbf h)$ : $\ F'(t)=\nabla f^{\top}\mathbf h$ and $F''(t)=\mathbf h^{\top}H\mathbf h$ , both evaluated at $\mathbf x_0+t\mathbf h$ . Hence $f(\mathbf x_0+\mathbf h)=f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H(\mathbf x_0+c\mathbf h)\mathbf h$ for some $c\in(0,1)$ .

1

Along the walk the

i

-th coordinate is

x_i(t)=x_{0,i}+t\,h_i

, so

dx_i/dt=h_i

. The chain rule with

n

routes (Unit 7's add-across-paths, one path per coordinate):

F'(t)=\sum_{i=1}^{n}\frac{\partial f}{\partial x_i}\,\frac{dx_i}{dt}=\sum_{i=1}^{n}f_i\,h_i=\nabla f^{\top}\mathbf h.

A sum of (partial) × (step component) over one index is exactly a dot product — a row times a column.

2

Differentiate again. Each

f_i

is itself a function of position on the walk, so it gets its own chain rule with the same constant velocities:

F''(t)=\sum_{i=1}^{n}h_i\,\frac{d}{dt}f_i=\sum_{i=1}^{n}h_i\sum_{j=1}^{n}\frac{\partial f_i}{\partial x_j}\,h_j=\sum_{i=1}^{n}\sum_{j=1}^{n}h_i\,H_{ij}\,h_j=\mathbf h^{\top}H\,\mathbf h.

A double sum of (entry) × (left step component) × (right step component) is exactly a row × matrix × column. Here

H_{ij}=\partial^2 f/\partial x_j\partial x_i=\partial^2 f/\partial x_i\partial x_j

by Clairaut, which is why

H

is symmetric and why the two cross terms

h_1h_2H_{12}

and

h_2h_1H_{21}

merge into

2h_1h_2H_{12}

in two variables.

3

Apply one-variable Taylor with

n=2

to

F

on

[0,1]

:

F(1)=F(0)+F'(0)+\tfrac12F''(c)

for some

c\in(0,1)

. Substitute steps 1 and 2:

f(\mathbf x_0+\mathbf h)=f(\mathbf x_0)+\nabla f(\mathbf x_0)^{\top}\mathbf h+\tfrac12\,\mathbf h^{\top}H(\mathbf x_0+c\mathbf h)\,\mathbf h.

That is the claim. ∎ Nothing about the argument used

n=2

. The matrix form is not a new theorem — it is the one-variable theorem read along a line, with the chain rule turning derivatives into a gradient and a Hessian. Keep going and the third derivative would be an

n\times n\times n

cube of numbers eating three copies of

\mathbf h

; linear algebra stops being comfortable there, which is exactly why "second-order" is where most of optimisation stops too.

Pause & predict

You remember $f(a)+f'(a)h+\tfrac12f''(a)h^2$ and want its many-variable form. Which promotion is right — and why do the others fail before you even think about calculus?

In one sentence:

f(\mathbf x_0+\mathbf h)\approx f(\mathbf x_0)+\nabla f^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H\mathbf h

is the one-variable formula with each derivative promoted one rank and fed one copy of

\mathbf h

per index — and it falls straight out of the line trick.

11

The judge of extrema — the Hessian

Put the two previous sections together. At a critical point the gradient term of the matrix form vanishes, so the height change after a step is

f(\mathbf x_0+\mathbf h)-f(\mathbf x_0)=\tfrac12\,\mathbf h^{\top}H\,\mathbf h\quad(\text{at the secret point}),\qquad Q(h,k)=\mathbf h^{\top}H\mathbf h=h^2f_{xx}+2hkf_{xy}+k^2f_{yy}.

The question "bowl, dome or saddle?" is now the question "is $\mathbf h^{\top}H\mathbf h$ positive for all $\mathbf h$ , negative for all $\mathbf h$ , or both?" — in Unit 4's words, is $H$ positive-definite, negative-definite, or indefinite. A matrix has become a judge. Put it on the stand:

Try: start from the bowl, then raise $f_{xy}$ slowly. The bowl tilts and stretches; the moment $f_{xy}^2$ passes $f_{xx}f_{yy}$ , one direction gives way and the bowl tears open into a saddle.

drag the picture to orbit

f_xx2

f_xy0

f_yy2

—

The surface is $z=\tfrac12\,(h,k)\,H\,(h,k)^{\top}=\tfrac12(f_{xx}h^2+2f_{xy}hk+f_{yy}k^2)$ — the second-order term of Taylor at a flat point, with nothing else in the way. The three tabs are three ways of reading the same matrix: they always agree, and the widget shows why.

The judge's ruling rests on one algebraic identity, and it is worth seeing exactly how three coefficients get squeezed into two signs. The trick is completing the square — multiply $Q$ by $f_{xx}$ so that the $h$ -terms fold into a perfect square, and watch what is left over:

Prove it · the judge's identity

Claim. $f_{xx}\,Q=(h\,f_{xx}+k\,f_{xy})^2+\big(f_{xx}f_{yy}-f_{xy}^2\big)\,k^2$ . Writing $D=f_{xx}f_{yy}-f_{xy}^2$ (the discriminant, which is $\det H$ ): $\ f_{xx}Q=(hf_{xx}+kf_{xy})^2+Dk^2$ .

1

Multiply the form by

f_{xx}

:

f_{xx}Q=h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xx}f_{yy}.

Every term picks up one factor

f_{xx}

. The first two now look like the start of

(hf_{xx}+kf_{xy})^2

.

2

Expand the square that those two terms suggest:

(hf_{xx}+kf_{xy})^2=h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xy}^2.

It matches step 1 in the first two terms; the third term is

k^2f_{xy}^2

instead of

k^2f_{xx}f_{yy}

.

3

Subtract step 2 from step 1; only the

k^2

terms survive:

f_{xx}Q-(hf_{xx}+kf_{xy})^2=k^2f_{xx}f_{yy}-k^2f_{xy}^2=\big(f_{xx}f_{yy}-f_{xy}^2\big)k^2=Dk^2.

That is the claim. ∎ The right side is a square plus

D

times another square. Squares are never negative — so the sign of

f_{xx}Q

is controlled by the sign of

D

.

Prove it · the three verdicts

Claim. At a critical point: $D>0,\ f_{xx}>0\Rightarrow$ local minimum; $D>0,\ f_{xx}<0\Rightarrow$ local maximum; $D<0\Rightarrow$ saddle. $D=0$ : no verdict.

1

$D>0$ . Then

f_{xx}\ne0

(otherwise

D=-f_{xy}^2\le0

). By the identity,

f_{xx}Q=(\text{square})+D\,k^2\ge0

, with equality only if both

k=0

and

hf_{xx}=0

, i.e.

h=k=0

. So for every non-zero step,

f_{xx}Q>0

:

Q

has the sign of

f_{xx}

.

f_{xx}>0

:

Q>0

for all steps — every small step goes up — a bowl, a local minimum.

f_{xx}<0

:

Q<0

for all steps — a dome, a local maximum.

2

$D<0$ , with $f_{xx}\ne0$ . Step along the

x

-axis,

(h,k)=(1,0)

:

Q=f_{xx}

. Step along

(h,k)=(-f_{xy},\,f_{xx})

: the square in the identity vanishes (

-f_{xy}f_{xx}+f_{xx}f_{xy}=0

), leaving

f_{xx}Q=Df_{xx}^2<0

, so

Q

has the sign opposite to

f_{xx}

. Two directions, two opposite signs: up one way, down another. That is a saddle. (If

f_{xx}=0

but

f_{yy}\ne0

, run the same argument with the roles of

h

and

k

swapped; if both vanish,

D=-f_{xy}^2<0

means

f_{xy}\ne0

and

Q=2hkf_{xy}

already changes sign between

(1,1)

and

(1,-1)

.)

3

$D=0$ . The identity reads

f_{xx}Q=(hf_{xx}+kf_{xy})^2\ge0

:

Q

never changes sign, but it is zero along a whole direction, and there the second-order term says nothing — the decision passes to third- and fourth-order terms, which this test does not see. Practice Problem 8 shows

x^4+y^4

(a minimum) and

x^4-y^4

(a saddle) with the same zero Hessian. ∎ Inconclusive is an honest verdict, not a failure: the judge only hears second-order evidence.

Pause & predict

At a critical point you find $f_{xx}=-2,\ f_{yy}=-2,\ f_{xy}=1$ . What kind of point is it?

The discriminant is one reading of $H$ . Unit 4 taught two more, and the lecture connects all three. Sylvester's criterion: a symmetric matrix is positive-definite exactly when every leading principal minor — the determinants of the top-left $1\times1$ , $2\times2$ , … blocks — is positive. For our $2\times2$ Hessian the minors are $f_{xx}$ and $\det H=D$ , so Sylvester's condition is the minimum verdict, word for word. Eigenvalues: a symmetric matrix is positive-definite exactly when all its eigenvalues are positive. The compass below shows why that is the same statement again, and why the eigenvectors are the directions of greatest and least curvature:

Try: press ▶ sweep. On a saddle the needle crosses zero four times per turn; on a bowl it never does. Then drag $f_{xy}$ and watch the eigen-directions (the longest and shortest spokes) rotate.

f_xx2

f_xy0

f_yy-2

direction θ30°

—

Walk away from the flat point in direction $\theta$ with unit speed; the surface's height grows like $\tfrac12 Q(\theta)\,r^2$ where $Q(\theta)=\mathbf u^{\top}H\mathbf u$ for $\mathbf u=(\cos\theta,\sin\theta)$ . The largest and smallest values of $Q$ over all directions are the eigenvalues $\lambda_1,\lambda_2$ of $H$ , reached along its eigenvectors — Unit 4's spectral theorem, standing on a hilltop.

Prove it · eigenvalues are the extreme curvatures, and their signs are the verdict

Claim. Let $H$ be symmetric with eigenvalues $\lambda_1\ge\lambda_2$ and orthonormal eigenvectors $\mathbf e_1,\mathbf e_2$ . Then for every unit direction $\mathbf u$ , $\ \lambda_2\le\mathbf u^{\top}H\mathbf u\le\lambda_1$ , with equality along $\mathbf e_2$ and $\mathbf e_1$ . Consequently $H$ is positive-definite $\iff\lambda_1,\lambda_2>0$ , negative-definite $\iff$ both negative, indefinite $\iff$ opposite signs; and $D=\det H=\lambda_1\lambda_2$ , $f_{xx}+f_{yy}=\lambda_1+\lambda_2$ .

1

Unit 4's spectral theorem:

H=Q\Lambda Q^{\top}

with

Q=[\mathbf e_1\ \mathbf e_2]

orthogonal and

\Lambda=\mathrm{diag}(\lambda_1,\lambda_2)

. Write any direction in the eigenbasis,

\mathbf u=\alpha\mathbf e_1+\beta\mathbf e_2

, i.e.

Q^{\top}\mathbf u=(\alpha,\beta)^{\top}

. Then

\mathbf u^{\top}H\mathbf u=(Q^{\top}\mathbf u)^{\top}\Lambda(Q^{\top}\mathbf u)=\lambda_1\alpha^2+\lambda_2\beta^2.

In the eigenbasis the cross term disappears: the quadratic form is just two squares, each weighted by an eigenvalue.

2

For a unit direction

\alpha^2+\beta^2=1

, so

\lambda_1\alpha^2+\lambda_2\beta^2

is a weighted average of

\lambda_1

and

\lambda_2

with non-negative weights summing to

1

. It therefore lies between them:

\lambda_2\le\mathbf u^{\top}H\mathbf u\le\lambda_1,

with

\lambda_1

reached at

\alpha=1

(direction

\mathbf e_1

) and

\lambda_2

at

\beta=1

(direction

\mathbf e_2

). The compass rose is exactly this: the needle's reading swings between the two eigenvalues, touching them along the eigenvectors.

3

Signs.

Q>0

for every direction

\iff

the minimum reading

\lambda_2>0

\iff

both eigenvalues positive.

Q<0

for every direction

\iff\lambda_1<0

. Opposite signs

\iff

the needle crosses zero

\iff

saddle. Definiteness is a statement about the smallest (or largest) curvature, and those are eigenvalues.

4

Bridge to the discriminant:

\det H=\det(Q\Lambda Q^{\top})=\det\Lambda=\lambda_1\lambda_2=D

, and

\operatorname{tr}H=f_{xx}+f_{yy}=\lambda_1+\lambda_2

. So

D<0\iff

opposite signs

\iff

saddle; and

D>0

with

f_{xx}>0

forces

f_{yy}>f_{xy}^2/f_{xx}\ge0

, hence

\lambda_1+\lambda_2>0

with

\lambda_1\lambda_2>0

: both positive. The three readings of

H

— discriminant, Sylvester, eigenvalues — are one verdict. ∎ Which reading to use is a matter of convenience:

D

is fastest by hand; eigenvalues also tell you how bowl-shaped (their ratio is the condition number that will govern gradient descent in Unit 9); Sylvester scales to

n\times n

Hessians where "

D

" no longer exists.

Prove it · negative-definite means the minors alternate

Claim. $H$ is negative-definite $\iff$ $f_{xx}<0$ and $D>0$ — the leading minors alternate in sign, starting negative.

1

H

is negative-definite exactly when

-H

is positive-definite (multiply

\mathbf h^{\top}H\mathbf h<0

by

-1

). A dome flipped upside down is a bowl.

2

Apply Sylvester to

-H=\begin{bmatrix}-f_{xx}&-f_{xy}\\-f_{xy}&-f_{yy}\end{bmatrix}

: its minors are

-f_{xx}

and

\det(-H)=(-1)^2\det H=D

. Both must be positive:

f_{xx}<0

and

D>0

. That is the claim. ∎ Flipping the sign of a

k\times k

block multiplies its determinant by

(-1)^k

— that is why, for a larger Hessian, the minors of a negative-definite matrix alternate: negative for odd

k

, positive for even

k

. The eigenvalues are all negative, as the lecture says.

Pause & predict

A Hessian has eigenvalues $\lambda_1=7.24$ and $\lambda_2=2.76$ at a critical point (Practice Problem 7). Without computing anything else, what do you know?

Pause & predict

For $f=e^x\sin y$ at $(0,0)$ the Hessian is $\begin{bmatrix}0&1\\1&0\end{bmatrix}$ : both pure curvatures are zero. Is the origin flat, or a saddle?

In one sentence: The Hessian is the matrix of the walk's second derivative;

D=\det H

with the sign of

f_{xx}

, Sylvester's minors, and the eigenvalue signs are three readings of the same verdict — bowl, dome, or saddle.

12

The judge at work — six landscapes

The procedure is always the same three moves, and it is worth saying them plainly before the widget does them for you. One: solve $\nabla f=\mathbf 0$ — two equations, two unknowns — to find every flat point. Two: compute $f_{xx},f_{xy},f_{yy}$ and evaluate them at each flat point separately (for anything beyond a quadratic they change from point to point). Three: read $D$ and $f_{xx}$ and pronounce the verdict. The companion's two worked examples and the practice set's landscapes are all here; so are the two where the judge must stay silent.

Try: load x³ + y³ − 3xy + 1. Two flat points: one is a saddle, one a minimum — click each dot and compare their Hessians. Then load the two $x^4$ landscapes, where the Hessian is all zeros and has nothing to say.

drag the picture to orbit · tap a dot to select it

—

For each landscape the widget solves $\nabla f=\mathbf 0$ , evaluates the Hessian at every solution and applies the second-derivative test. The glass surface at the selected point is the quadratic clone $f(P)+\tfrac12\,\mathbf h^{\top}H\mathbf h$ — the local bowl, dome or saddle that the Hessian is.

Reading the two cubic landscapes

$f=x^3+y^3-3xy+1$ : $\nabla f=(3x^2-3y,\ 3y^2-3x)=\mathbf 0$ gives $y=x^2$ and $x=y^2$ , so $x=x^4$ , i.e. $x\in\{0,1\}$ . Two flat points. At $(0,0)$ : $f_{xx}=f_{yy}=0$ , $f_{xy}=-3$ , $D=-9$ — a saddle. At $(1,1)$ : $f_{xx}=f_{yy}=6$ , $D=36-9=27$ , $f_{xx}>0$ — a local minimum, value $0$ . Same function, same formula for $H$ , two different verdicts, because $f_{xx}=6x$ and $f_{yy}=6y$ are different numbers at the two points. Local minimum, too: walk to $(-2,0)$ and $f=-7$ , lower than $0$ . The judge only rules on the immediate neighbourhood.

And the two landscapes where it cannot rule. $g=x^4+y^4$ and $h=x^4-y^4$ both have the origin as their only flat point, and both have $f_{xx}=12x^2$ , $f_{yy}=\pm12y^2$ , $f_{xy}=0$ — all zero there, $D=0$ . The second-derivative test sees two identical (empty) Hessians. Yet $g\ge0$ everywhere with equality only at the origin (a minimum), while $h=t^4$ along the $x$ -axis and $-t^4$ along the $y$ -axis (a saddle). The difference is fourth order — invisible to a test that only hears second-order evidence. The lecture's instruction is the right one: when $D=0$ , another test is needed: look at the higher-order terms of the expansion, or argue about the function's shape directly.

Pause & predict

For $f=x^3-3xy+3y^2$ (Practice Problem 6) the flat points are $(0,0)$ and $(\tfrac12,\tfrac14)$ , with $f_{xx}=6x$ , $f_{xy}=-3$ , $f_{yy}=6$ . Which verdict goes where?

In one sentence: Solve

\nabla f=\mathbf 0

, evaluate

H

at each flat point, read

D

and

f_{xx}

— and when

D=0

, the second-order evidence has run out and you must look further.

13

Why a learning machine cares — curvature, conditioning, and the saddle problem

The lecture opened by saying the Hessian "plays a huge role in neural network cost function minimisation". Here is that role, in three sentences that the coming units will turn into whole stories.

Training is descending a landscape. The loss $L(\mathbf w)$ is a function of a million weights, and every step of gradient descent (Unit 9) is a step on that landscape. At the bottom of a valley the gradient vanishes — and the Hessian, now a million-by-million matrix of second partials, is exactly the judge of this unit: positive-definite means a true minimum, opposite-sign eigenvalues mean a saddle you can still slide off.

Curvature sets the speed limit. Near a minimum the loss is its own quadratic clone, $L\approx L_0+\tfrac12\mathbf h^{\top}H\mathbf h$ . The compass showed the curvature ranging from $\lambda_{\min}$ to $\lambda_{\max}$ . Gradient descent must take steps small enough for the steepest direction ( $\lambda_{\max}$ ) yet then crawls along the flattest ( $\lambda_{\min}$ ); the ratio $\lambda_{\max}/\lambda_{\min}$ , the condition number, is how elongated the valley is and how badly plain descent zig-zags. Momentum, RMSProp and Adam (Unit 11) are all ways of coping with a bad Hessian without ever computing it.

The clone suggests a jump. If you trusted the quadratic clone completely, you would not step downhill — you would jump straight to the clone's own bottom. That is Newton's method, and it falls out of this unit in two lines:

Prove it · Newton's step is the bottom of the quadratic clone

Claim. The quadratic clone of $f$ at $\mathbf x_0$ , $\ Q(\mathbf h)=f(\mathbf x_0)+\nabla f^{\top}\mathbf h+\tfrac12\mathbf h^{\top}H\mathbf h$ , is minimised (when $H$ is positive-definite) at $\mathbf h^{\star}=-H^{-1}\nabla f$ .

1

Differentiate

Q

with respect to

\mathbf h

, using Unit 7's cookbook (

\nabla_{\mathbf h}(\mathbf a^{\top}\mathbf h)=\mathbf a

,

\ \nabla_{\mathbf h}(\tfrac12\mathbf h^{\top}H\mathbf h)=H\mathbf h

for symmetric

H

):

\nabla_{\mathbf h}Q=\nabla f+H\mathbf h.

The clone is a quadratic, so its gradient is linear in

\mathbf h

.

2

Set it to zero and solve:

H\mathbf h^{\star}=-\nabla f

, so

\mathbf h^{\star}=-H^{-1}\nabla f

. The clone's Hessian is

H

itself, positive-definite, so this flat point of

Q

is its minimum — by this unit's own verdict. ∎ Gradient descent moves along

-\nabla f

, scaled by a step size you must guess. Newton moves along

-H^{-1}\nabla f

: the curvature rescales every direction — big steps where the valley is flat, small where it is steep. For a true quadratic it lands on the minimum in one jump. The price is

H

: for a million weights it is unaffordable, which is why the whole optimiser zoo exists.

Two ways down an elongated bowl. Contours of

\tfrac12(9u^2+v^2)

, condition number

\lambda_1/\lambda_2=9

. Gradient descent (orange, step size

0.2

) must keep its steps short for the steep axis, so it bounces between the walls and crawls along the valley; Newton's step (green) uses

H^{-1}

to rescale every direction and lands on the minimum at once. Unit 9 begins where the orange path does.

Pause & predict

Gradient descent takes the step $-\eta\nabla f$ ; Newton takes $-H^{-1}\nabla f$ . For a quadratic bowl with eigenvalues $\lambda_1\gg\lambda_2$ , why does Newton not zig-zag?

In one sentence: A loss surface near its bottom is its own quadratic clone; the Hessian's eigenvalues say how hard descent will be, and its inverse turns the clone's bottom into Newton's one-step jump.

14

What to carry forward

One flat spot on a trail generated this unit. Everything else was the same move — Rolle — applied again, then tilted, then stacked, then walked along a line.

Idea	The one-line version	Where it returns
Rolle	Same height at both ends + smooth ⇒ a flat spot between	every existence argument in calculus
Mean value theorem	$f(b)=f(a)+f'(c)(b-a)$ — Rolle, tilted	Unit 9 · why a descent step works
Taylor with remainder	$f(x)=P_{n-1}(x)+\dfrac{f^{(n)}(c)}{n!}(x-a)^n$ , $c$ between $a$ and $x$	every "local approximation" from here on
The bound	$\|R_n\|\le M\|x-a\|^n/n!$ — you never need $c$	numerical methods, error budgets
MacLaurin	Taylor at $0$ ; $e^x,\sin,\cos$ converge everywhere because $n!$ wins	how your computer evaluates functions
The line trick	$F(t)=f(a+th,b+tk)$ : two variables become one	directional derivatives; convexity proofs (Unit 14)
The Hessian	$H=\begin{bmatrix}f_{xx}&f_{xy}\\f_{xy}&f_{yy}\end{bmatrix}$ ; $Q=\mathbf h^{\top}H\mathbf h$	Units 9–11 · the shape of every loss valley
The verdict	$D>0,f_{xx}>0$ min · $D>0,f_{xx}<0$ max · $D<0$ saddle · $D=0$ silent	classifying critical points, always
Three readings	$D=\det H=\lambda_1\lambda_2$ ; Sylvester's minors; eigenvalue signs	Unit 14 · positive-definite kernels and convexity
Newton's step	$-H^{-1}\nabla f$ : jump to the clone's bottom	Units 10–11 · second-order methods and why we avoid them

The one thing to remember

Every smooth function is, up close, a polynomial — and Taylor's theorem tells you exactly how far "up close" reaches. The remainder $f^{(n)}(c)(x-a)^n/n!$ is not a hand-wave; it came from a ladder of flat spots. And in two variables, the second-order term of that polynomial is a matrix whose signs decide whether a flat point is the bottom of a bowl. When a training loop reports that it has found a minimum, this unit is how anyone could ever check.

15

Practice arena — the unit's problem set, solved in full

Twelve problems: three from Part A (Taylor) and three from Part B (the Hessian) of Prof. Saurabh's practice set, three from his companion guide, and three written for this arena — a remainder-bound workout (Problem 4), an explicit hunt for the secret $c$ (Problem 5), and a proof workout that derives the judge's identity and connects it to eigenvalues (Problem 12). Every number below is machine-verified.

Two habits do most of the work: for a Taylor polynomial, write the derivatives at the anchor in a column before touching the formula, and keep the $k!$ ; for a critical point, evaluate the Hessian at each point separately, then read $D$ first and $f_{xx}$ second.

Problem 1easy

Let $f(x)=\ln x$ . (a) Compute $f$ and its first four derivatives at $x=1$ . (b) Write the fourth-order Taylor polynomial $P_4(x)$ of $f$ about the point $a=1$ . (c) Use $P_4$ to estimate $\ln 1.1$ and compare with the true value.

What this tests. The recipe itself — differentiate, evaluate at the centre, divide by $k!$ , attach $(x-a)^k$ . Plan. Make a small table of derivatives first and evaluate every one at the centre before assembling anything; then look for the pattern in the coefficients, and finish by plugging in a nearby $x$ to see the polynomial earn its keep.

Show the full solution

(a) Step 1 — differentiate four times. Each derivative of

\ln x

is a power of

x

, so the pattern is easy to carry:

\begin{aligned} f&=\ln x, & f'&=x^{-1}, & f''&=-x^{-2},\\ f'''&=2x^{-3}, & f^{(4)}&=-6x^{-4}. \end{aligned}

At the centre

x=1

every power of

x

is

1

:

f(1)=0,\quad f'(1)=1,\quad f''(1)=-1,\quad f'''(1)=2,\quad f^{(4)}(1)=-6.

(b) Step 2 — divide by $k!$ and attach the powers of $(x-1)$ . The coefficient of

(x-1)^k

is

f^{(k)}(1)/k!

:

\begin{aligned} \frac{f'(1)}{1!}&=1, & \frac{f''(1)}{2!}&=-\frac12, & \frac{f'''(1)}{3!}&=\frac26=\frac13, & \frac{f^{(4)}(1)}{4!}&=-\frac{6}{24}=-\frac14, \end{aligned}

so

P_4(x)=(x-1)-\tfrac12(x-1)^2+\tfrac13(x-1)^3-\tfrac14(x-1)^4.

The coefficients are

(-1)^{k+1}/k

— the

k!

in the denominator has eaten the

(k-1)!

that the derivatives produced.

Step 3 — the quicker route. Put

u=x-1

. Then

\ln x=\ln(1+u)

, whose MacLaurin series is a standard one:

\ln(1+u)=u-\frac{u^2}{2}+\frac{u^3}{3}-\frac{u^4}{4}+\cdots

Truncate at degree 4 and write

u=x-1

back: the same

P_4

, in one line, with no differentiation at all. Expanding about

a

is always the same as expanding about

0

in the shifted variable

x-a

.

(c) Step 4 — check at $x=1.1$ , where

u=0.1

:

\begin{aligned} P_4(1.1)&=0.1-\frac{0.01}{2}+\frac{0.001}{3}-\frac{0.0001}{4}\\ &=0.1-0.005+0.000333-0.000025=0.095308. \end{aligned}

The true value is

\ln 1.1=0.095310

; the error is

1.8\times10^{-6}

, about the size of the first term we dropped,

u^5/5=2.0\times10^{-6}

. Four terms, six correct decimals.

(a) $f(1)=0,\ f'(1)=1,\ f''(1)=-1,\ f'''(1)=2,\ f^{(4)}(1)=-6$ . (b) $P_4(x)=(x-1)-\tfrac12(x-1)^2+\tfrac13(x-1)^3-\tfrac14(x-1)^4$ . (c) $P_4(1.1)=0.095308$ against $\ln1.1=0.095310$ , error $1.8\times10^{-6}$ .

Remember

The coefficient is $f^{(k)}(a)/k!$ , never $f^{(k)}(a)$ alone, and the variable is $(x-a)$ , never $x$ . Those two slips lose most of the marks on this type. When the function is a shifted standard one — $\ln(1+u)$ , $e^{u}$ , $\sqrt{1+u}$ — substitute first and copy the known series.

Problem 2easy

Let $f(x)=\sqrt{x}$ , expanded about the point $a=4$ . (a) Find the first-order (linear) polynomial $P_1(x)$ and the second-order (quadratic) polynomial $P_2(x)$ . (b) Use each to estimate $\sqrt{4.2}$ . (c) Explain why the quadratic estimate is the better one.

What this tests. That $P_1$ is the tangent line, that $P_2$ adds the curvature, and that the sign of $f''$ tells you on which side of the curve the tangent sits. Plan. Compute $f,f',f''$ at the centre; halve $f''$ when it becomes a coefficient; plug in the increment $x-4=0.2$ rather than $x$ itself; compare both estimates with the true value at the end.

Show the full solution

(a) Step 1 — derivatives at the centre. With

f=x^{1/2}

:

\begin{aligned} f(4)&=2, & f'(x)&=\tfrac12x^{-1/2} &&\Rightarrow f'(4)=\tfrac14,\\ && f''(x)&=-\tfrac14x^{-3/2} &&\Rightarrow f''(4)=-\tfrac14\cdot\tfrac18=-\tfrac1{32}. \end{aligned}

Step 2 — assemble. The quadratic coefficient is

f''(4)/2!=-\tfrac1{64}

, so

P_1(x)=2+\tfrac14(x-4),\qquad P_2(x)=2+\tfrac14(x-4)-\tfrac1{64}(x-4)^2.

(b) Step 3 — plug in the increment $h=x-4=0.2$ .

\begin{aligned} P_1(4.2)&=2+\tfrac14(0.2)=2.05,\\ P_2(4.2)&=2.05-\tfrac{1}{64}(0.2)^2=2.05-0.000625=2.049375. \end{aligned}

The true value is

\sqrt{4.2}=2.0493902

. So

P_1

is off by

+6.1\times10^{-4}

and

P_2

by

-1.5\times10^{-5}

— about forty times closer.

(c) Step 4 — why the quadratic wins.

P_1

is the tangent line at

x=4

: it matches the height and the slope there but knows nothing about bending. Since

f''(4)=-\tfrac1{32}<0

, the curve bends downward, so the tangent rides above it and drifts away — that is why

P_1

's error is positive. The remainder formula says how far: with

n=2

the error is

\tfrac12 f''(c)\,h^2

for some

c

between

4

and

4.2

, and

\tfrac12\cdot\tfrac1{32}\cdot0.04=6.25\times10^{-4}

— almost exactly the

6.1\times10^{-4}

we found.

P_2

puts that missing bend back in with the term

-\tfrac1{64}h^2

, so what remains is the third-order piece

\frac{f'''(4)}{3!}h^3=\frac{3/256}{6}\,(0.008)=\frac{0.008}{512}=1.56\times10^{-5},

which matches

P_2

's leftover error to two figures (and its sign explains why

P_2

now sits just below the curve).

Step 5 — the binomial shortcut. Factor out the centre and use

(1+t)^{1/2}=1+\tfrac12t-\tfrac18t^2+\cdots

with

t=h/4

:

\sqrt{4+h}=2\sqrt{1+\tfrac h4}=2\Big(1+\tfrac12\cdot\tfrac h4-\tfrac18\cdot\tfrac{h^2}{16}+\cdots\Big)=2+\tfrac h4-\tfrac{h^2}{64}+\cdots

Both polynomials fall out at once.

(a) $P_1(x)=2+\tfrac14(x-4)$ , $P_2(x)=2+\tfrac14(x-4)-\tfrac1{64}(x-4)^2$ . (b) $P_1(4.2)=2.05$ , $P_2(4.2)=2.049375$ , true $2.0493902$ ; errors $6.1\times10^{-4}$ and $1.5\times10^{-5}$ . (c) $f''<0$ : the tangent sits above a curve that bends down; the quadratic term restores the bend.

Remember

$f''(a)$ is not the coefficient — $f''(a)/2$ is. Writing $-\tfrac1{32}(x-4)^2$ instead of $-\tfrac1{64}(x-4)^2$ is the classic lost mark here. And always substitute the increment $x-a$ , not $x$ : $P_1(4.2)$ uses $0.2$ , not $4.2$ .

Problem 3medium

Let $f(x,y)=e^{x}\sin y$ . (a) Compute $\nabla f$ and the Hessian $H$ at $(0,0)$ . (b) Write the second-order Taylor polynomial $Q(x,y)$ of $f$ about $(0,0)$ . (c) Use $Q$ to approximate $f(0.1,0.2)$ , compare with the exact value, and say where the error comes from.

What this tests. The two-variable quadratic clone — five partial derivatives, the $\tfrac12$ in front and the $2$ on the cross term — and reading the Hessian as a shape. Plan. List all five partials before evaluating any; substitute the centre; assemble $Q$ in the form value + gradient term + half the quadratic form; then evaluate and compare, and name the order of the first neglected term.

Show the full solution

(a) Step 1 — the five partials. The exponential survives every

x

-derivative; the sine cycles under

y

-derivatives:

\begin{aligned} f_x&=e^{x}\sin y, & f_y&=e^{x}\cos y,\\ f_{xx}&=e^{x}\sin y, & f_{xy}&=e^{x}\cos y, & f_{yy}&=-e^{x}\sin y. \end{aligned}

At

(0,0)

,

e^{0}=1,\ \sin0=0,\ \cos0=1

, so

f=0,\ f_x=0,\ f_y=1,\ f_{xx}=0,\ f_{xy}=1,\ f_{yy}=0

:

\nabla f(0,0)=\begin{bmatrix}0\\1\end{bmatrix},\qquad H(0,0)=\begin{bmatrix}0&1\\1&0\end{bmatrix}.

(b) Step 2 — assemble the clone. With centre

(0,0)

the increments are

h=x,\ k=y

:

\begin{aligned} Q(x,y)&=f+\big(f_x\,x+f_y\,y\big)+\tfrac12\big(f_{xx}x^2+2f_{xy}xy+f_{yy}y^2\big)\\ &=0+(0\cdot x+1\cdot y)+\tfrac12\,(0+2\cdot1\cdot xy+0)\\ &=y+xy. \end{aligned}

(c) Step 3 — the numbers.

Q(0.1,0.2)=0.2+(0.1)(0.2)=0.22

. Exactly,

f(0.1,0.2)=e^{0.1}\sin0.2=1.105171\times0.198669=0.2195636,

so

Q

overshoots by

4.4\times10^{-4}

.

Step 4 — where the error comes from. Multiply the one-variable series and keep track of total degree:

\begin{aligned} e^{x}\sin y&=\Big(1+x+\tfrac{x^2}{2}+\tfrac{x^3}{6}+\cdots\Big)\Big(y-\tfrac{y^3}{6}+\cdots\Big)\\ &=y+xy+\underbrace{\tfrac12x^2y-\tfrac16y^3}_{\text{degree }3}+\cdots \end{aligned}

This is the product-of-series shortcut:

Q=y+xy

appears with no Hessian at all. The first neglected terms are cubic, and at

(0.1,0.2)

they give

0.001-0.001333=-3.3\times10^{-4}

; the quartic terms

\tfrac16x^3y-\tfrac16xy^3

add

-1.0\times10^{-4}

, and together they account for almost the whole gap of

-4.4\times10^{-4}

. Doubling the step to

(0.2,0.4)

gives

f=0.475637

against

Q=0.48

, a gap of

-4.4\times10^{-3}

— ten times larger for a step twice as long, exactly what a cubic-led error (

2^3=8

, plus the quartic share) predicts.

Step 5 — read the Hessian as a shape.

H=\begin{bmatrix}0&1\\1&0\end{bmatrix}

has trace

0

and determinant

-1

, so its eigenvalues solve

\lambda^2-1=0

:

\lambda=\pm1

. The pure curvatures

f_{xx},f_{yy}

vanish and all the bending lives in the cross term — the quadratic part

xy

is a saddle, rising along

y=x

and falling along

y=-x

. (The origin is not a critical point, since

f_y=1

, so this is a statement about the clone's shape, not a classification of an extremum.)

(a) $\nabla f(0,0)=(0,1)^{\top}$ , $H(0,0)=\begin{bmatrix}0&1\\1&0\end{bmatrix}$ . (b) $Q(x,y)=y+xy$ . (c) $Q(0.1,0.2)=0.22$ against $f=0.2195636$ ; error $4.4\times10^{-4}$ , third order, mostly $\tfrac12x^2y-\tfrac16y^3$ .

Remember

The quadratic term is $\tfrac12\big(f_{xx}h^2+2f_{xy}hk+f_{yy}k^2\big)$ — a $\tfrac12$ outside and a $2$ on the cross term. Dropping either is where this problem sheds marks. When both functions have known series, multiply them and truncate by total degree: it is faster and it also shows you the size of the next term.

Problem 4mediumadded — covers the remainder bound

Bound the lie. (a) The small-angle rule $\cos x\approx1-\tfrac{x^2}{2}$ is used at $x=0.5$ . Using the remainder formula, bound the error before computing anything; then compute the actual error and compare. (b) How many terms of the MacLaurin series of $e^{x}$ are needed at $x=1$ to guarantee an error below $10^{-6}$ ? Confirm by summing.

What this tests. Using $R_n=f^{(n)}(c)(x-a)^n/n!$ as a guarantee: choosing $n$ , bounding $|f^{(n)}(c)|$ over the whole interval, and reading "how many terms" off a factorial. Plan. First decide which $n$ the remainder formula should carry (count a zero coefficient as a kept term); replace the unknown $c$ by the worst case on the interval; only then compute the true value and check that it sits inside the bound.

Show the full solution

(a) Step 1 — pick $n$ honestly. The MacLaurin coefficients of

\cos

are

1,\ 0,\ -\tfrac12,\ 0,\ \tfrac1{24},\dots

because

f'''(0)=\sin0=0

. So

1-\tfrac{x^2}{2}

is not only

P_2

but also

P_3

: the degree-3 term is present and equals zero. The remainder formula therefore applies with

n=4

:

R_4(x)=\frac{f^{(4)}(c)}{4!}\,x^4=\frac{\cos c}{24}\,x^4,\qquad c\ \text{between}\ 0\ \text{and}\ x.

Step 2 — bound the unknown $c$ away. Whatever

c

is,

|\cos c|\le1

, so

|R_4(0.5)|\le\frac{0.5^4}{24}=\frac{0.0625}{24}=0.0026042.

That is a promise made before any cosine was evaluated: the rule cannot lie by more than

0.0026

.

Step 3 — the actual error.

\cos0.5=0.8775826

and

1-\tfrac{0.25}{2}=0.875

, so the error is

0.0025826

— inside the bound, and almost equal to it (ratio

0.99

). The bound is nearly tight because

\cos c\approx1

for every

c

in

(0,0.5)

: the next term of the series,

x^4/24

, is essentially the error. Had we lazily used

n=3

, the bound

|\sin c|\,0.5^3/6\le0.0208

would still be true, but eight times looser.

(b) Step 4 — write the bound. Keeping the terms

k=0,\dots,n-1

of

e^{x}=\sum x^k/k!

leaves the remainder

R_n=e^{c}x^n/n!

with

c\in(0,1)

. At

x=1

,

e^{c}\le e<3

, so

|R_n|\le\frac{e}{n!}<\frac{3}{n!}.

Step 5 — find the first $n$ that works.

\frac{3}{9!}=\frac{3}{362\,880}=8.27\times10^{-6}\ (\text{too big}),\qquad \frac{3}{10!}=8.27\times10^{-7}<10^{-6}.

So

n=10

: keep the terms

k=0,1,\dots,9

— ten terms, up to

x^9/9!

.

Step 6 — confirm by summing.

\sum_{k=0}^{9}\frac1{k!}=2.7182815\quad\text{against}\quad e=2.7182818,

an error of

3.0\times10^{-7}

, below

10^{-6}

as promised. And nine terms genuinely fail:

\sum_{k\le8}1/k!=2.7182788

, error

3.1\times10^{-6}

. The bound was honest, not lazy.

(a) $n=4$ : $|R_4|\le0.5^4/24=0.0026042$ ; actual error $\cos0.5-0.875=0.0025826$ , inside the bound. (b) $3/n!<10^{-6}$ first at $n=10$ : ten terms ( $k=0,\dots,9$ ); the sum is $2.7182815$ , error $3.0\times10^{-7}$ .

Remember

In the remainder formula $n$ is one more than the highest degree you kept — and a zero coefficient still counts as kept, which is why $1-x^2/2$ earns the $x^4$ bound. Two habits lose marks: bounding $f^{(n)}(c)$ by its value at $a$ instead of its maximum over the interval, and confusing "number of terms" with "degree".

Problem 5easyadded — covers the secret c

Find the secret $c$ . Take $f(x)=x^3$ on $[0,2]$ , anchored at $a=0$ . Taylor's theorem promises a $c\in(0,2)$ with $f(2)=P_{n-1}(2)+\dfrac{f^{(n)}(c)}{n!}\,2^n$ . (a) For $n=1$ (the mean value theorem), find $c$ . (b) For $n=2$ , find $c$ . (c) For $n=3$ , what happens?

What this tests. That the $c$ in the remainder is a definite number fixed by $f$ , $a$ , $x$ and $n$ — not a free parameter — and that it moves when $n$ moves. Plan. Write the theorem with the given $n$ , fill in every known value ( $f$ and its derivatives at the anchor, the power of $x-a$ ), and solve the one equation left for $c$ ; check it lands strictly inside the interval.

Show the full solution

(a) Step 1 — $n=1$ is the mean value theorem.

f(2)-f(0)=f'(c)(2-0)

with

f'(x)=3x^2

:

8-0=3c^2\cdot2=6c^2\ \Rightarrow\ c^2=\tfrac43\ \Rightarrow\ c=\tfrac{2}{\sqrt3}=1.1547.

The negative root

-1.1547

lies outside

[0,2]

; the positive one is inside, as promised. Check: the chord has slope

8/2=4

, and

f'(1.1547)=3\cdot\tfrac43=4

. ✓

(b) Step 2 — $n=2$ . Now

P_1(x)=f(0)+f'(0)\,x=0+0\cdot x=0

: at the anchor the cubic is flat, so its tangent line is the

x

-axis and the whole of

f(2)

is remainder. With

f''(x)=6x

:

8=\frac{f''(c)}{2!}\,2^2=\frac{6c}{2}\cdot4=12c\ \Rightarrow\ c=\tfrac23.

Again

c\in(0,2)

. The curvature that explains the climb to height

8

is the curvature measured at

x=\tfrac23

.

(c) Step 3 — $n=3$ .

f'''(x)\equiv6

, so

R_3(x)=\frac{f'''(c)}{3!}\,x^3=\frac66\,x^3=x^3=f(x)

for every

c

. The remainder is the entire function: a cubic is its own degree-3 Taylor polynomial, so

P_3=f

,

R_4\equiv0

and

f^{(4)}\equiv0

. The theorem is true for all

c

at once and stops telling you anything.

Step 4 — $c$ depends on $n$ and on $x$ . Redo (a) and (b) with a general endpoint

x

instead of

2

:

x^3=3c^2x

gives

c=x/\sqrt3

, and

x^3=3c\,x^2

gives

c=x/3

. At

x=2

these are

1.1547

and

0.6667

; at

x=1

they would be

0.5774

and

0.3333

. The secret point slides with the endpoint and jumps when the order changes.

(a) $c=2/\sqrt3=1.1547$ . (b) $c=2/3$ . (c) $f'''\equiv6$ makes $R_3=x^3=f(x)$ : every $c$ works, because a cubic equals its own $P_3$ . In general $c=x/\sqrt3$ for $n=1$ and $c=x/3$ for $n=2$ .

Remember

The theorem promises that some $c$ strictly between $a$ and $x$ makes the formula exact; it does not tell you which one, and it changes with $n$ and $x$ . Marks go when a student sets $c=a$ or $c=x$ "for simplicity", or forgets the $(x-a)^n$ factor next to $f^{(n)}(c)/n!$ .

Problem 6medium

Let $f(x,y)=x^3-3xy+3y^2$ . (a) Find all critical points. (b) Form the Hessian and classify each point with the second-derivative test. (c) State the local-minimum value, and confirm the classification by the eigenvalue route.

What this tests. The full routine — solve $\nabla f=0$ , build $H$ , read $D=f_{xx}f_{yy}-f_{xy}^2$ and then $f_{xx}$ — at two points where the Hessian is not constant. Plan. Use the simpler gradient equation to eliminate a variable; keep $H$ symbolic and only then substitute each point; write $D$ 's sign before $f_{xx}$ 's; finish with trace and determinant for the eigenvalues as an independent check.

Show the full solution

(a) Step 1 — solve $\nabla f=\mathbf 0$ .

\nabla f=\big(3x^2-3y,\ -3x+6y\big)

. The second component gives

x=2y

; substitute into the first:

3(2y)^2-3y=12y^2-3y=3y(4y-1)=0\ \Rightarrow\ y=0\ \text{or}\ y=\tfrac14.

Then

x=2y

gives the critical points

(0,0)

and

\big(\tfrac12,\tfrac14\big)

.

(b) Step 2 — the Hessian, kept symbolic.

f_{xx}=6x,\ f_{xy}=-3,\ f_{yy}=6

:

H=\begin{bmatrix}6x&-3\\-3&6\end{bmatrix},\qquad D=f_{xx}f_{yy}-f_{xy}^2=36x-9.

Step 3 — the verdicts. At

(0,0)

:

D=-9<0

⇒ saddle (no need to look at

f_{xx}

). At

\big(\tfrac12,\tfrac14\big)

:

D=36\cdot\tfrac12-9=9>0

and

f_{xx}=3>0

⇒ local minimum.

(c) Step 4 — the value.

f\big(\tfrac12,\tfrac14\big)=\tfrac18-3\cdot\tfrac12\cdot\tfrac14+3\cdot\tfrac1{16}=\tfrac{2}{16}-\tfrac{6}{16}+\tfrac{3}{16}=-\tfrac1{16}.

Step 5 — the eigenvalue route. At the minimum

H=\begin{bmatrix}3&-3\\-3&6\end{bmatrix}

has trace

9

and determinant

18-9=9

, so its eigenvalues solve

\lambda^2-9\lambda+9=0

:

\lambda=\frac{9\pm\sqrt{81-36}}{2}=\frac{9\pm\sqrt{45}}{2}=\frac{9\pm6.708}{2}=7.854,\ 1.146,

both positive ⇒ positive-definite ⇒ a bowl. At the origin

H=\begin{bmatrix}0&-3\\-3&6\end{bmatrix}

has determinant

-9

, so

\lambda=3\pm\sqrt{18}=7.243,\ -1.243

: opposite signs ⇒ indefinite ⇒ saddle. For a symmetric

2\times2

matrix

\det H=\lambda_1\lambda_2

, which is exactly why

D<0

forces a saddle.

(a) $(0,0)$ and $\big(\tfrac12,\tfrac14\big)$ . (b) $(0,0)$ : $D=-9$ , saddle; $\big(\tfrac12,\tfrac14\big)$ : $D=9,\ f_{xx}=3$ , local minimum. (c) $f_{\min}=-\tfrac1{16}$ ; eigenvalues there $(9\pm\sqrt{45})/2\approx7.854,\ 1.146$ .

Remember

Ask $D$ 's sign first, then $f_{xx}$ 's — a saddle needs no second question. The mark most often lost here: evaluating $H$ once and reusing it, when $f_{xx}=6x$ changes from point to point. And $f_{xx}=0$ at the origin does not make it "neither"; $D=-9$ already says saddle.

Problem 7medium

A model is trained by minimising the quadratic loss $f(x,y)=2x^2+2xy+3y^2-4x-2y$ . (a) Find the unique critical point. (b) Show in two ways — Sylvester's criterion and the eigenvalues — that the Hessian is positive-definite, and conclude the point is the global minimum. (c) State the minimum value.

What this tests. That "positive-definite Hessian" can be certified by leading minors or by eigenvalue signs, and that for a quadratic a constant positive-definite Hessian makes the local minimum global. Plan. Solve the linear gradient system; write the constant $H$ ; check $\Delta_1=f_{xx}$ and $\Delta_2=\det H$ ; get the eigenvalues from trace and determinant; evaluate $f$ at the point, and if time allows complete the square to prove the minimum is global.

Show the full solution

(a) Step 1 — the critical point.

\nabla f=(4x+2y-4,\ 2x+6y-2)=\mathbf 0

, i.e.

2x+y=2

and

x+3y=1

. From the first,

y=2-2x

; substitute:

x+3(2-2x)=1\ \Rightarrow\ -5x=-5\ \Rightarrow\ x=1,\ y=0.

The critical point is

(1,0)

.

(b) Step 2 — Sylvester. Because

f

is quadratic the Hessian is the same everywhere:

H=\begin{bmatrix}4&2\\2&6\end{bmatrix},\qquad \Delta_1=4>0,\quad \Delta_2=\det H=24-4=20>0.

Both leading principal minors are positive ⇒

H

is positive-definite.

Step 3 — eigenvalues. Trace

10

, determinant

20

:

\lambda=\frac{10\pm\sqrt{100-80}}{2}=5\pm\sqrt5=7.236,\ 2.764,

both positive ⇒ positive-definite again. Since

H\succ0

at every point,

f

is strictly convex: it has exactly one stationary point and that point is the global minimum.

(c) Step 4 — the value.

f(1,0)=2-4=-2

.

Step 5 — completing the square proves "global" by hand. Shift to the critical point with

x=1+u,\ y=v

:

\begin{aligned} f&=2(1+u)^2+2(1+u)v+3v^2-4(1+u)-2v\\ &=-2+\big(2u^2+2uv+3v^2\big)\\ &=-2+2\big(u+\tfrac v2\big)^2+\tfrac52v^2\ \ \ge\ -2, \end{aligned}

with equality only at

u=v=0

. One manoeuvre certifies definiteness and pins the value.

Step 6 — how elongated is the bowl? The condition number

\lambda_{\max}/\lambda_{\min}=7.236/2.764=2.618

(exactly

(3+\sqrt5)/2

). The contours are ellipses whose axes are in the ratio

\sqrt{2.618}\approx1.62

: steepest along the

\lambda\approx7.24

eigenvector, gentlest along the

\lambda\approx2.76

one. The larger this ratio, the more gradient descent zig-zags before it settles.

(a) $(1,0)$ . (b) $H=\begin{bmatrix}4&2\\2&6\end{bmatrix}$ : minors $4>0,\ 20>0$ ; eigenvalues $5\pm\sqrt5\approx7.236,\ 2.764$ ; $H\succ0$ everywhere ⇒ global minimum. (c) $f_{\min}=-2$ ; condition number $\approx2.618$ .

Remember

Sylvester uses the leading minors: the top-left $1\times1$ entry and then the full determinant. Checking $f_{yy}>0$ instead of $f_{xx}>0$ is not the criterion (it happens to agree here). For a quadratic, $H$ is constant, so positive-definite once means convex everywhere — and "local" upgrades to "global" for free.

Problem 8medium

Consider $g(x,y)=x^4+y^4$ and $h(x,y)=x^4-y^4$ . (a) Show that for both, the origin is a critical point and the Hessian there is the zero matrix, so the second-derivative test is inconclusive. (b) Determine the true nature of the origin for each by a direct argument. (c) In one line, say why the Hessian could not tell them apart.

What this tests. What $D=0$ actually means — "no verdict", not "flat" — and how to classify a point when the quadratic clone is silent. Plan. Compute gradient and Hessian at the origin for both; when the test is silent, argue directly from the sign of the function (a global inequality, or two directions with opposite signs); the ray slice $(t\cos\theta,t\sin\theta)$ is the systematic version of that.

Show the full solution

(a) Step 1 — gradients and Hessians.

\begin{aligned} \nabla g&=(4x^3,\ 4y^3), & H_g&=\begin{bmatrix}12x^2&0\\0&12y^2\end{bmatrix},\\ \nabla h&=(4x^3,\ -4y^3), & H_h&=\begin{bmatrix}12x^2&0\\0&-12y^2\end{bmatrix}. \end{aligned}

At the origin both gradients vanish (critical point) and both Hessians are the zero matrix, so

D=\det H=0

for each: the test says nothing.

(b) Step 2 — $g$ by a global inequality.

g(x,y)=x^4+y^4\ge0

for every

(x,y)

, with equality only at the origin ⇒ a strict local (indeed global) minimum.

Step 3 — $h$ by two directions. Along the

x

-axis,

h(t,0)=t^4>0

; along the

y

-axis,

h(0,t)=-t^4<0

. A point that is a minimum in one direction and a maximum in another is a saddle.

Step 4 — the ray slice does both at once. Put

(x,y)=t(\cos\theta,\sin\theta)

:

\begin{aligned} g&=t^4\big(\cos^4\theta+\sin^4\theta\big)=t^4\big(1-\tfrac12\sin^2 2\theta\big)\ \ge\ \tfrac12t^4>0,\\ h&=t^4\big(\cos^4\theta-\sin^4\theta\big)=t^4\cos2\theta, \end{aligned}

using

\cos^4+\sin^4=(\cos^2+\sin^2)^2-2\sin^2\cos^2

and

\cos^4-\sin^4=(\cos^2-\sin^2)(\cos^2+\sin^2)

. Every ray of

g

climbs; the rays of

h

climb for

|\theta|<45^\circ

and fall for

45^\circ<\theta<135^\circ

. A sign that changes with direction is the signature of a saddle.

(c) Step 5 — why the Hessian was blind. The Hessian records second-order behaviour only; here the first non-zero behaviour is fourth order, and the quadratic clone of both functions is the same flat plane

z=0

.

(a) Both have $\nabla=\mathbf 0$ and $H=0$ at the origin, so $D=0$ . (b) $g\ge0$ ⇒ minimum; $h=t^4$ along $x$ and $-t^4$ along $y$ ⇒ saddle (ray slice: $g=t^4(1-\tfrac12\sin^22\theta)$ , $h=t^4\cos2\theta$ ). (c) The difference is quartic; a second-order test cannot see it.

Remember

$D=0$ is a refusal to judge, not a verdict of "flat" or "neither". Writing "inconclusive" and stopping loses the marks: the question asks for the true nature, which needs a direct argument — an inequality, two opposite directions, or a ray slice.

Problem 9easy

Find the local extreme values of $f(x,y)=xy-x^2-y^2-2x-2y+4$ .

What this tests. The three-step routine on a function whose Hessian is constant: critical point, $D$ , verdict, value. Plan. Solve the two linear gradient equations by substitution; write the three second partials (they are numbers here); compute $D$ and then look at $f_{xx}$ ; substitute back to get the value, and cross-check with the eigenvalues.

Show the full solution

Step 1 — critical points.

f_x=y-2x-2=0

and

f_y=x-2y-2=0

. From the first,

y=2x+2

; into the second:

\begin{aligned} x-2(2x+2)-2=0\ &\Rightarrow\ -3x-6=0\ \Rightarrow\ x=-2,\\ y&=2(-2)+2=-2. \end{aligned}

The only critical point is

(-2,-2)

.

Step 2 — second partials and $D$ .

f_{xx}=-2,\ f_{yy}=-2,\ f_{xy}=1

, constants:

D=f_{xx}f_{yy}-f_{xy}^2=(-2)(-2)-1^2=4-1=3>0.

Step 3 — verdict and value.

D>0

and

f_{xx}=-2<0

⇒ local maximum. Its value:

f(-2,-2)=4-4-4+4+4+4=8.

Step 4 — cross-check by eigenvalues.

H=\begin{bmatrix}-2&1\\1&-2\end{bmatrix}

has trace

-4

and determinant

3

, so

\lambda^2+4\lambda+3=0

gives

\lambda=-1,\ -3

: both negative ⇒ negative-definite ⇒ a dome. Because

H

is constant,

f

is concave everywhere and

8

is in fact the global maximum.

One critical point $(-2,-2)$ ; $D=3>0$ , $f_{xx}=-2<0$ : a local (indeed global) maximum with $f(-2,-2)=8$ .

Remember

A maximum needs both $D>0$ and $f_{xx}<0$ ; a negative $f_{xx}$ with $D<0$ is a saddle, not a maximum. The marks that go missing here are almost always a sign slip in the two-line linear system — substitute your point back into both gradient equations before moving on.

Problem 10medium

Find the local extreme values of $f(x,y)=x^3+y^3-3xy+1$ .

What this tests. A nonlinear gradient system with a real-root argument, and a Hessian that must be re-evaluated at each point. Plan. Turn each gradient equation into a substitution, reduce to one polynomial equation, keep only the real roots; evaluate $f_{xx},f_{yy},f_{xy}$ separately at each point; give both the verdict and the value, and say whether the minimum is global.

Show the full solution

Step 1 — critical points.

f_x=3x^2-3y=0\ \Rightarrow\ y=x^2,\qquad f_y=3y^2-3x=0\ \Rightarrow\ x=y^2.

Substituting

y=x^2

into the second:

x=x^4

, so

x(x^3-1)=0

. The real roots are

x=0

and

x=1

(the other two roots of

x^3=1

are complex). Hence the critical points are

(0,0)

and

(1,1)

.

Step 2 — the Hessian entries.

f_{xx}=6x,\ f_{yy}=6y,\ f_{xy}=-3

— the diagonal changes from point to point, the off-diagonal does not.

Step 3 — at $(0,0)$ .

f_{xx}=0,\ f_{yy}=0,\ f_{xy}=-3

:

D=0\cdot0-(-3)^2=-9<0\ \Rightarrow\ \text{saddle},\qquad f(0,0)=1.

Step 4 — at $(1,1)$ .

f_{xx}=6,\ f_{yy}=6,\ f_{xy}=-3

:

\begin{aligned} D&=6\cdot6-(-3)^2=36-9=27>0,\quad f_{xx}=6>0\\ &\Rightarrow\ \text{local minimum},\qquad f(1,1)=1+1-3+1=0. \end{aligned}

Step 5 — eigenvalue check, and local versus global. At

(1,1)

,

H=\begin{bmatrix}6&-3\\-3&6\end{bmatrix}

has eigenvalues

6\pm3=9,\ 3

(both positive: bowl); at

(0,0)

,

H=\begin{bmatrix}0&-3\\-3&0\end{bmatrix}

has eigenvalues

\pm3

(saddle). The minimum value

0

is local only:

f(-2,0)=-8+1=-7

, and a cubic runs to

-\infty

— the test describes a neighbourhood, nothing more.

$(0,0)$ : $D=-9$ , saddle, $f=1$ . $(1,1)$ : $D=27,\ f_{xx}=6$ , local minimum, $f=0$ (not global: $f(-2,0)=-7$ ).

Remember

$f_{xx}=f_{yy}=0$ at the origin does not make $D=0$ : the cross term $f_{xy}=-3$ gives $D=-9$ and a clear saddle. And for cubics, local is not global — say so when you report a minimum.

Problem 11hard

Find the absolute maximum and minimum of $f(x,y)=x^2+y^2-x-y$ on the region $D=\{(x,y):x^2\le y\le1\}$ .

What this tests. Absolute extrema on a closed, bounded region: interior critical points and every piece of the boundary and the corners, then a comparison. Plan. Draw the region first; collect the interior critical points that lie inside it; reduce each boundary curve to a one-variable function and find its critical points; add the corners; make one list of values and pick the largest and smallest — the second-derivative test is not needed for any of this.

Show the full solution

Step 1 — draw the region in words.

D

is the set of points above the parabola

y=x^2

and below the line

y=1

: a parabolic bowl with a flat lid, meeting at the corners

(-1,1)

and

(1,1)

. It is closed and bounded, so an absolute maximum and minimum exist, and each lives either at an interior critical point or somewhere on the boundary. The second-derivative test can only classify flat points in the open interior; on the boundary the gradient need not vanish at all, so the boundary must be searched separately.

Step 2 — interior critical points.

f_x=2x-1=0

and

f_y=2y-1=0

give

\big(\tfrac12,\tfrac12\big)

. Is it inside

D

?

\big(\tfrac12\big)^2=\tfrac14\le\tfrac12\le1

, yes. Its value:

f\big(\tfrac12,\tfrac12\big)=\tfrac14+\tfrac14-\tfrac12-\tfrac12=-\tfrac12.

(Here

H=2I

, a bowl, so this is a local minimum — but that says nothing about the boundary.)

Step 3 — the parabola arc $y=x^2$ , $-1\le x\le1$ . Substitute:

\begin{aligned} g(x)&=f(x,x^2)=x^2+x^4-x-x^2=x^4-x,\\ g'(x)&=4x^3-1=0\ \Rightarrow\ x=\big(\tfrac14\big)^{1/3}=0.62996. \end{aligned}

Then

y=x^2=0.39685

, and since

4x^3=1

we have

x^4=x/4

, so

g=\tfrac x4-x=-\tfrac34x=-0.47247

. The arc's endpoints are the corners:

f(-1,1)=1+1+1-1=2,\qquad f(1,1)=1+1-1-1=0.

Step 4 — the lid $y=1$ , $-1\le x\le1$ .

h(x)=f(x,1)=x^2+1-x-1=x^2-x

, with

h'(x)=2x-1=0

at

x=\tfrac12

:

f\big(\tfrac12,1\big)=\tfrac14+1-\tfrac12-1=-\tfrac14.

Its endpoints are the same two corners, already counted.

Step 5 — compare. The candidate values are

\{-\tfrac12,\ -0.47247,\ 2,\ 0,\ -\tfrac14\}

. The absolute maximum is

2

at the corner

(-1,1)

; the absolute minimum is

-\tfrac12

at the interior point

\big(\tfrac12,\tfrac12\big)

.

Step 6 — a geometric cross-check. Completing squares,

f=\big(x-\tfrac12\big)^2+\big(y-\tfrac12\big)^2-\tfrac12

:

f

is the squared distance from

\big(\tfrac12,\tfrac12\big)

, minus

\tfrac12

. The minimum is at that centre, which lies in

D

; the maximum is at the point of

D

farthest from it — the corner

(-1,1)

, at squared distance

1.5^2+0.5^2=2.5

, giving

2.5-0.5=2

. ✓ The arc point

(0.63,0.40)

is merely the foot of the perpendicular from the centre to the parabola — the closest point of the arc, never a contender for either extreme. Note

\nabla f(-1,1)=(-3,1)\neq\mathbf 0

: the maximum sits where the gradient does not vanish, which is exactly why the boundary must be checked by hand.

Interior: $f(\tfrac12,\tfrac12)=-\tfrac12$ . Parabola: $f(0.62996,0.39685)=-0.47247$ , corners $f(-1,1)=2$ , $f(1,1)=0$ . Lid: $f(\tfrac12,1)=-\tfrac14$ . Absolute maximum $2$ at $(-1,1)$ ; absolute minimum $-\tfrac12$ at $(\tfrac12,\tfrac12)$ .

Remember

On a closed, bounded region keep three lists — interior critical points, each boundary piece reduced to one variable, and the corners — and only then compare. The marks vanish when the corners are skipped (that is where the maximum lives here) or when the arc's $-0.47$ is reported as the minimum without checking it against $-\tfrac12$ .

Problem 12hardadded — covers the judge's identity

The judge's identity, proved and used. Let $Q(h,k)=h^2f_{xx}+2hkf_{xy}+k^2f_{yy}$ with the partials taken at a critical point, and $D=f_{xx}f_{yy}-f_{xy}^2$ . (a) Prove that $f_{xx}\,Q=(hf_{xx}+kf_{xy})^2+D\,k^2$ by expanding the right-hand side. (b) Deduce the three verdicts: $D>0,\ f_{xx}>0$ ⇒ $Q>0$ for all $(h,k)\neq\mathbf 0$ ; $D>0,\ f_{xx}<0$ ⇒ $Q<0$ ; $D<0$ ⇒ $Q$ takes both signs. (c) For a symmetric $H=\begin{bmatrix}a&b\\b&c\end{bmatrix}$ , show $\lambda_1\lambda_2=ac-b^2=D$ and $\lambda_1+\lambda_2=a+c$ ; hence $D<0$ exactly when the eigenvalues have opposite signs, and $D>0$ with $a>0$ forces $c>0$ and both eigenvalues positive. (d) Apply (c) to $H=\begin{bmatrix}3&-3\\-3&6\end{bmatrix}$ .

What this tests. The one identity behind the whole second-derivative test, and why the discriminant $D$ and the eigenvalue signs are the same criterion in two languages. Plan. Expand, cancel, factor; for each verdict start from the sign of the right-hand side and divide by $f_{xx}$ with the correct flip; for the saddle, exhibit two explicit directions; for (c) write the characteristic polynomial and read off sum and product of roots; finish with the numbers.

Show the full solution

(a) Step 1 — expand the right-hand side.

\begin{aligned} (hf_{xx}+kf_{xy})^2&=h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xy}^2,\\ D\,k^2&=f_{xx}f_{yy}\,k^2-f_{xy}^2\,k^2. \end{aligned}

Adding, the two

k^2f_{xy}^2

terms cancel and what is left has a common factor

f_{xx}

:

h^2f_{xx}^2+2hk\,f_{xx}f_{xy}+k^2f_{xx}f_{yy}=f_{xx}\big(h^2f_{xx}+2hkf_{xy}+k^2f_{yy}\big)=f_{xx}\,Q.\ \ \blacksquare

The identity is nothing more than completing the square in

h

, with everything multiplied by

f_{xx}

so that no fractions appear.

(b) Step 2 — verdict 1: $D>0$ and $f_{xx}>0$ . The right-hand side is a square plus

D\,k^2

, so it is

\ge0

. It can be

0

only if

k=0

and

hf_{xx}+kf_{xy}=0

, i.e.

hf_{xx}=0

, i.e.

h=0

. So for

(h,k)\neq\mathbf 0

we have

f_{xx}Q>0

, and dividing by

f_{xx}>0

gives

Q>0

: uphill in every direction — a bowl, a local minimum.

Step 3 — verdict 2: $D>0$ and $f_{xx}<0$ . The same right-hand side is still strictly positive for

(h,k)\neq\mathbf 0

— its sign never depended on the sign of

f_{xx}

. Dividing by

f_{xx}<0

flips the inequality:

Q<0

in every direction — a dome, a local maximum. (Note that

D>0

forces

f_{xx}\neq0

, since

f_{xx}f_{yy}>f_{xy}^2\ge0

, so the division is always legal in verdicts 1 and 2.)

Step 4 — verdict 3: $D<0$ . Test two explicit directions. Along

(h,k)=(1,0)

:

Q(1,0)=f_{xx}.

Along

(h,k)=(-f_{xy},\,f_{xx})

:

\begin{aligned} Q(-f_{xy},f_{xx})&=f_{xy}^2f_{xx}-2f_{xy}f_{xx}f_{xy}+f_{xx}^2f_{yy}\\ &=f_{xx}\big(f_{xx}f_{yy}-f_{xy}^2\big)=f_{xx}\,D, \end{aligned}

so

f_{xx}\cdot Q(-f_{xy},f_{xx})=f_{xx}^2D<0

whenever

f_{xx}\neq0

: the two values

Q(1,0)=f_{xx}

and

Q(-f_{xy},f_{xx})=f_{xx}D

have opposite signs — up one way, down another, a saddle. If

f_{xx}=0

, then

D=-f_{xy}^2<0

means

f_{xy}\neq0

, and along

(h,k)=(1,t)

:

Q(1,t)=2t\,f_{xy}+t^2f_{yy}=t\,(2f_{xy}+t\,f_{yy}),

which for small

t

has the sign of

t\,f_{xy}

— opposite signs at

t=\pm\varepsilon

. (When

f_{yy}\neq0

one can instead swap the roles of

x

and

y

and run the identity with

f_{yy}

.) Either way

Q

changes sign: saddle.

(c) Step 5 — eigenvalues of a symmetric $2\times2$ .

\det(H-\lambda I)=(a-\lambda)(c-\lambda)-b^2=\lambda^2-(a+c)\lambda+(ac-b^2).

For a monic quadratic with roots

\lambda_1,\lambda_2

, the sum of the roots is the negated middle coefficient and the product is the constant term:

\lambda_1+\lambda_2=a+c=\operatorname{tr}H,\qquad \lambda_1\lambda_2=ac-b^2=\det H=D.

(The roots are real because the discriminant

(a+c)^2-4(ac-b^2)=(a-c)^2+4b^2\ge0

.) Hence

D<0\iff\lambda_1\lambda_2<0\iff

the eigenvalues have opposite signs. And if

D>0

with

a>0

:

ac>b^2\ge0

with

a>0

forces

c>0

, so the sum

a+c>0

and the product

D>0

— two real numbers with positive sum and positive product are both positive. (By the same token

a<0

forces

c<0

and both eigenvalues negative.) This is why "

D

and

f_{xx}

" and "eigenvalue signs" are one criterion in two languages.

(d) Step 6 — the numbers.

H=\begin{bmatrix}3&-3\\-3&6\end{bmatrix}

: trace

9

, determinant

18-9=9=D

. So

\lambda^2-9\lambda+9=0

:

\lambda=\frac{9\pm\sqrt{81-36}}{2}=\frac{9\pm\sqrt{45}}{2}=7.854,\ 1.146.

Check: sum

9

, product

9

. ✓ Both positive ⇒ positive-definite ⇒ a bowl — this is the Hessian of Problem 6 at its minimum

\big(\tfrac12,\tfrac14\big)

. Sylvester says the same in one line (

3>0,\ 9>0

), and the identity of part (a) shows it explicitly:

3Q=(3h-3k)^2+9k^2>0

for

(h,k)\neq\mathbf 0

.

(a) Expanding gives $f_{xx}(h^2f_{xx}+2hkf_{xy}+k^2f_{yy})=f_{xx}Q$ . (b) $D>0$ : the right side is positive for $(h,k)\neq\mathbf 0$ , so $Q$ has the sign of $f_{xx}$ ; $D<0$ : $Q(1,0)=f_{xx}$ and $Q(-f_{xy},f_{xx})=f_{xx}D$ have opposite signs (or $Q(1,\pm\varepsilon)$ when $f_{xx}=0$ ). (c) $\lambda_1\lambda_2=D$ , $\lambda_1+\lambda_2=a+c$ . (d) $\lambda=(9\pm\sqrt{45})/2\approx7.854,\ 1.146$ , both positive: bowl.

Remember

The whole test is one line of completed square: $f_{xx}Q=(hf_{xx}+kf_{xy})^2+Dk^2$ . Read it as "a square plus $D$ times a square" and the three verdicts follow, and $D=\det H=\lambda_1\lambda_2$ is why the discriminant's sign is the eigenvalue story in disguise. Marks are lost by forgetting the $f_{xx}=0$ case in the saddle argument, or by claiming $D>0$ alone gives a minimum.

Next up

Unit 9 · Gradient Descent — coming soon

You now know what a landscape looks like near a flat point and how to tell a bowl from a saddle. Unit 9 starts walking: step sizes, why the mean value theorem guarantees a descent step, how the Hessian's eigenvalues set the speed limit, and what happens in the fog of a million dimensions.

← All units