Unit 2 · Vector Spaces — The Math Behind the Machine

The universe upgrade

Imagine this

At a kirana shop, the owner writes your order as a list: 2 kg rice, 1 kg dal, 3 packets of biscuits. Your friend's order is another list.

Add the two lists and you get one order for both of you. Double a list and you get the order for a party. The owner never stops to ask whether adding or doubling "makes sense". It always does, and the answer is always another order list.

That quiet guarantee is what this unit is about.

In Unit 1 we solved equations. Now we step back and look at the world the answers live in.

Almost everything in machine learning is a long list of numbers. A cat photo is 150,528 numbers. A word inside a language model is 768 numbers. Your taste in films is a list. The weights of a neural network are a list. Training a model means moving through the world of such lists, one small step at a time.

Here is the kirana idea with small numbers. Two orders, as lists:

(2,\,1,\,3)+(1,\,0,\,2)=(3,\,1,\,5)

2\times(2,\,1,\,3)=(4,\,2,\,6)

Adding gives a list of the same length. Scaling gives a list of the same length. Nothing breaks. A world where these two moves always work is called a vector space, and the lists in it are called vectors. The world of all 3-number lists is written $\mathbb{R}^3$ (read it: "R three").

Decoder ring: how to read ℝ with a little number on top (worth 60 seconds)

Notation	Shape	Mental picture
$\mathbb{R}$	a single number	a dot on the number line
$\mathbb{R}^n$	a column of $n$ numbers	an arrow
$\mathbb{R}^{m\times n}$	an $m\times n$ grid	a spreadsheet
$\mathbb{R}^{H\times W\times 3}$	a stack of 3 grids	a colour photo
$\mathbb{R}^{B\times H\times W\times 3}$	$B$ such stacks	a batch of photos

Read $\mathbb{R}^{\text{(shape)}}$ as "all the real-number data with this shape". $\mathbb{R}^{768}$ is nothing scarier than "a column with 768 numbers in it".

Six ideas, one after the other. Each one builds on the last:

①Groups — combining things with a trustworthy undo button.
②Vector spaces — add scaling, and you get the board machine learning plays on.
③Subspaces — smaller worlds inside, like a sheet of glass through the origin.
④Span — everywhere a few arrows can take you.
⑤Independence — which arrows are extra baggage.
⑥Basis and dimension — the smallest toolkit, and the true size of the world.

In one sentence: every object in machine learning is a list of numbers, and this unit maps the world those lists live in and the rules for moving around it.

Groups — the bare skeleton of "combining things"

Imagine this

You type a message on your phone, then press undo. The words disappear, and you are back where you started.

For undo to be trustworthy, four things must be true. Every move keeps you inside the app. It does not matter how you bracket a chain of moves. There is a "do nothing" move. And every move can be taken back.

A world with these four promises is called a group.

Take whole numbers with ordinary adding. Watch all four promises hold:

1 $5+3=8$ is a whole number. You stay inside. (closure)
2 $(2+3)+4=2+(3+4)=9$ . Brackets don't matter. (associativity)
3 $5+0=5$ . Zero is the do-nothing move. (identity)
4 $5+(-5)=0$ . The number $-5$ undoes $5$ . (inverse)

In symbols: a group is a set $G$ with a way of combining, written $\otimes$ , such that $x\otimes y$ stays in $G$ ; $(x\otimes y)\otimes z=x\otimes(y\otimes z)$ ; some $e$ has $x\otimes e=e\otimes x=x$ ; and every $x$ has a partner $y$ with $x\otimes y=y\otimes x=e$ . If also $x\otimes y=y\otimes x$ for every pair, the group is called Abelian (order never matters).

Fine print: many lecture notes write the identity and inverse promises from one side only ( $x\otimes e=x$ , $x\otimes y=e$ ). For groups, the one-sided version already forces the two-sided one, so both definitions describe the same objects.

Closure and brackets almost never break. The promise that fails most often is the undo: the partner you need is simply not in the set. Try it:

Try: pick a set, press do, then press undo. The sets that fail are the ones whose undo partner is not a member.

—

hand at 0

Undo lab

—

Press do, then undo.

The realization

A group is a set with a trustworthy undo button. You can combine any two members and stay inside, brackets never matter, there is a do-nothing member $e$ , and every $x$ has a partner that brings you back to $e$ .

When a set fails, it is nearly always the last promise. The counting numbers have no $-5$ . The whole numbers under multiplication have no $\tfrac12$ .

Pause & predict

The counting numbers with zero, $\{0,1,2,3,\dots\}$ under $+$ , are not a group. Which promise breaks?

Pause & predict

On a 12-hour clock, moving the hand forward 5 hours can be undone by moving it forward some number of hours. Which number is the undo partner of 5?

In one sentence: a group is any world where combining keeps you inside and every move can be undone.

Vector spaces = group + scaling

Imagine this

An auto-rickshaw driver gives directions: "2 km along the main road, then 1 km up the side lane." You can chain directions one after the other. That is adding. You can say "go twice as far". That is scaling.

However you mix these two moves, you still end up somewhere in the city. You never fall off the map.

Small numbers first. Take $\mathbf{u}=(1,2)$ and $\mathbf{v}=(3,1)$ . Then $\mathbf{u}+\mathbf{v}=(4,3)$ , and doubling gives $2(\mathbf{u}+\mathbf{v})=(8,6)$ .

Now do it the other way round. Double each one first: $2\mathbf{u}=(2,4)$ and $2\mathbf{v}=(6,2)$ . Add them: $(2,4)+(6,2)=(8,6)$ . Same place. Two different routes, one destination.

A vector space is any world where adding and scaling behave like this. In symbols, $V=(\mathcal{V},+,\cdot)$ must keep four promises:

V1 $(\mathcal{V},+)$ is an Abelian group.Adding never leaves the set, $\mathbf{0}$ does nothing, $-\mathbf{v}$ undoes $\mathbf{v}$ , and order doesn't matter.
V2 $\lambda(\mathbf{x}+\mathbf{y})=\lambda\mathbf{x}+\lambda\mathbf{y}$ and $(\lambda+\psi)\mathbf{x}=\lambda\mathbf{x}+\psi\mathbf{x}$ .Scale-then-add equals add-then-scale. That is the "two routes" picture.
V3 $\lambda(\psi\mathbf{x})=(\lambda\psi)\mathbf{x}$ .Stretch by 2, then by 3, is one stretch by 6.
V4 $1\cdot\mathbf{x}=\mathbf{x}$ .Scaling by 1 changes nothing.

Race the two routes yourself, in 3D:

Try: press ▶ race and watch the two travellers arrive together. Then drag λ below 0: both routes flip backwards, and they still meet.

drag the picture to orbit

stretch λ1.5

—

Here $\mathbf{u}=(2,1,0)$ and $\mathbf{v}=(-1,1,1)$ . Blue: add first, then stretch. Orange: stretch each, then add. Promise V2 says they always meet.

The realization

$\lambda(\mathbf{u}+\mathbf{v})=\lambda\mathbf{u}+\lambda\mathbf{v}$

Adding and scaling can be done in either order, and you never leave the world. Anything you can add and scale by these rules counts as a vector. Every "linear" idea in this course, from gradients to averages, is this route-swapping trick.

Watch out

In this course, "vector" does not mean "a column of numbers". It means anything that lives in a vector space. A $4\times4$ matrix is a vector. A polynomial is a vector. If you can add two of them and scale one, and stay in the same kind of thing, it is a vector. Example: a weekly-sales spreadsheet plus a corrections spreadsheet is a spreadsheet of the same shape.

Pause & predict

Is a $3\times3$ weight matrix a "vector"?

In one sentence: a vector space is any world where you can add and scale freely and never fall off the board.

Subspaces — spaces inside spaces

Imagine this

Hold a big sheet of glass so that it passes through one corner of your room. Ants live on the glass.

Any walk the ants take along the glass stays on the glass. Chain two walks: still on the glass. Walk twice as far, or walk backwards: still on the glass. The ants have a complete little world of their own, inside your bigger room.

That little world is a subspace.

A tiny example on flat paper. Take the line $y=2x$ . The points $(1,2)$ and $(3,6)$ are on it. Add them: $(4,8)$ . Is $8=2\times4$ ? Yes, still on the line. Scale $(1,2)$ by $-1$ : $(-1,-2)$ . Still on the line. And the origin $(0,0)$ is on it too. It passes.

Now take the line $y=2x+1$ . Is $(0,0)$ on it? $0\neq 2\cdot0+1$ . No. It fails straight away.

A subset $U$ of a vector space is a subspace when it is a vector space by itself, using the same adding and scaling. The rules about brackets and order come free from the big space. Only three things can go wrong, and all three are about staying inside:

S1The origin is in it: $\mathbf{0}\in U$ . If not, stop. It fails.
S2Adding stays inside: $\mathbf{u},\mathbf{v}\in U\Rightarrow\mathbf{u}+\mathbf{v}\in U$ .
S3Scaling stays inside: $\mathbf{u}\in U\Rightarrow\lambda\mathbf{u}\in U$ for every number $\lambda$ , including $0$ and negatives.

Now be the stress-tester. Each button applies a real move to a member and checks where the answer lands:

Try: pick the unit square and press u × 2: the answer flies out of the fence. Then pick the parabola and press ▶ run all four tests. Only the flat sets through the origin survive.

drag the picture to orbit

slide u along—

S1 · has 0 S2 · + stays in S3 · × stays in

Pick a set, then try to break it.

The realization

In $\mathbb{R}^n$ , the subspaces are exactly the flat things through the origin: the origin alone, lines through $\mathbf{0}$ , planes through $\mathbf{0}$ , and so on up to the whole space. A set that is shifted, curved or fenced in always lets something escape.

This is a theorem, not just a pattern. Sections 10 and 11 show why: every subspace is the span of a few independent arrows, and such a span is always a flat through $\mathbf{0}$ .

Watch out — the bias term

A model with a bias, $\mathbf{w}^\top\mathbf{x}+b$ , draws a flat that is shifted off the origin. That is not a subspace (it is called affine). The usual trick is to add a constant 1 to every input $\mathbf{x}$ . One dimension up, the shifted flat becomes a genuine flat through the origin again.

Pause & predict

Which of these is a subspace of $\mathbb{R}^3$ ?

In one sentence: a subspace is a flat world through the origin where adding and scaling never take you outside.

The promised passport: the null space is a subspace

Imagine this

A shop sells soap at ₹10, tea at ₹10 and biscuits at ₹20. The billing machine multiplies each count by its price and adds.

You change your basket: one soap fewer, one tea more. The bill does not move: $-10+10=0$ . The machine cannot see this change at all.

Every change the machine cannot see is part of its blind spot. The surprise of this section: for every machine, the blind spot is a subspace.

Write a basket change as $\mathbf{x}=(\text{soap},\text{tea},\text{biscuits})$ . Counting the bill in tens of rupees, the machine is $A=\begin{pmatrix}1&1&2\end{pmatrix}$ , and the change in the bill is $A\mathbf{x}$ .

Two invisible changes: $\mathbf{x}=(-1,1,0)$ gives $-1+1+0=0$ . And $\mathbf{y}=(2,0,-1)$ , two soaps more and one biscuit packet fewer, gives $2+0-2=0$ .

Do both changes: $\mathbf{x}+\mathbf{y}=(1,1,-1)$ , and $1+1-2=0$ . Still invisible. Do the first change three times: $3\mathbf{x}=(-3,3,0)$ , and $-3+3+0=0$ . Still invisible.

Unit 1 promised the blind spot its papers. Here they are. The null space of a matrix is

N(A)=\{\mathbf{x}: A\mathbf{x}=\mathbf{0}\}

and it always passes the three-point subspace test. Stamp the passports yourself:

Try: press ▶ stamp the passports. Every invisible change, every sum and every stretch stays on the glass plane with the meter at ₹0. Then switch to bill goes up ₹40 and run it again: the sum overshoots to ₹80 and is refused.

drag the picture to orbit

stretch λ1.5

—

The machine is $A=\begin{pmatrix}1&1&2\end{pmatrix}$ (prices in tens of rupees). The glass plane is every change with $A\mathbf{x}$ equal to the chosen bill change. Only the ₹0 plane passes through the origin.

The realization

$\begin{aligned}A(\mathbf{x}+\mathbf{y})&=A\mathbf{x}+A\mathbf{y}=\mathbf{0}+\mathbf{0}=\mathbf{0}\\ A(\lambda\mathbf{x})&=\lambda A\mathbf{x}=\mathbf{0}\end{aligned}$

Zero in, zero out. Invisible plus invisible is invisible, and a stretched invisible change is still invisible. So the null space $N(A)$ is always a subspace. The same two rules make the column space $C(A)=\{A\mathbf{x}\}$ , all the outputs the machine can produce, a subspace too.

But the solutions of $A\mathbf{x}=\mathbf{b}$ with $\mathbf{b}\neq\mathbf{0}$ are never a subspace: two of them add up to $2\mathbf{b}$ , not $\mathbf{b}$ .

If you want the algebra · null space and column space are subspaces

Prove it · null space and column space are subspaces — derived

Claim. For any $m\times n$ matrix $A$ : $N(A)$ is a subspace of $\mathbb{R}^n$ and $C(A)$ is a subspace of $\mathbb{R}^m$ . The only tool is linearity: $A(\mathbf{x}+\mathbf{y})=A\mathbf{x}+A\mathbf{y}$ and $A(\lambda\mathbf{x})=\lambda A\mathbf{x}$ .

(S1 for

N(A)

)

A\mathbf{0}=\mathbf{0}

, so

\mathbf{0}\in N(A)

. Every entry of

A\mathbf{0}

is a sum of terms

a_{ij}\cdot 0

. Nothing in, nothing out.

(S2) Take

\mathbf{x},\mathbf{y}\in N(A)

, so

A\mathbf{x}=\mathbf{0}

and

A\mathbf{y}=\mathbf{0}

. Then

\begin{aligned}A(\mathbf{x}+\mathbf{y})&=A\mathbf{x}+A\mathbf{y}\\ &=\mathbf{0}+\mathbf{0}=\mathbf{0},\end{aligned}

\mathbf{x}+\mathbf{y}\in N(A)

. Share

A

out over the sum, then use what we know about each piece.

(S3) Take

\mathbf{x}\in N(A)

and any number

\lambda

. Then

A(\lambda\mathbf{x})=\lambda\,A\mathbf{x}=\lambda\mathbf{0}=\mathbf{0},

\lambda\mathbf{x}\in N(A)

. Three for three:

N(A)

is a subspace. Pull the number through

A

. Nothing about

\lambda

was assumed, so

0

and negatives are covered.

(S1 for

C(A)

)

\mathbf{0}=A\mathbf{0}

is an output, so

\mathbf{0}\in C(A)

. The same line as step 1, read the other way:

\mathbf{0}

is produced by some input.

(S2, S3 for

C(A)

) Take two outputs

A\mathbf{x}

and

A\mathbf{y}

, and a number

\lambda

. Then

\begin{aligned}A\mathbf{x}+A\mathbf{y}&=A(\mathbf{x}+\mathbf{y}),\\ \lambda\,A\mathbf{x}&=A(\lambda\mathbf{x}),\end{aligned}

and both right-hand sides are outputs of

A

. So

C(A)

is a subspace too. ∎ The same two linearity rules, read right to left: a sum of outputs is the output of the summed input.

This explains the pictures from Unit 1. The solutions of $A\mathbf{x}=\mathbf{b}$ form a flat that does not pass through the origin. It is the null space, picked up and carried by one particular solution $\mathbf{x}_p$ : every solution is $\mathbf{x}_p+\mathbf{n}$ with $\mathbf{n}$ in $N(A)$ . Here it is for the smallest possible machine, $A=\begin{pmatrix}1&2\end{pmatrix}$ and $b=4$ :

drag the picture to orbit

Subspace + shift = solution set. The null space (blue) passes through $\mathbf{0}$ . The solution set (orange) is the same line carried by $\mathbf{x}_p$ . Every orange point is $\mathbf{x}_p$ plus a null vector, so the orange line has the shape of a subspace, but it misses $\mathbf{0}$ , so it is not one.

Watch out — the null space in a trained model

Let $A$ be a data matrix and $\mathbf{w}$ the weights. For any $\mathbf{n}$ in $N(A)$ , $A(\mathbf{w}+\mathbf{n})=A\mathbf{w}$ : the two weight settings make exactly the same predictions on the training data. The data cannot tell them apart. Only an extra rule, such as "prefer small weights", picks one.

Pause & predict

Why does adding two members of $N(A)$ always give another member?

In one sentence: whatever a matrix cannot see forms a subspace, because zero plus zero is zero and a stretched zero is still zero.

Span — everywhere you can reach

Imagine this

Your home station has two Metro lines. One runs north-east, the other north-west. You can ride any distance on each, forwards or backwards, and change lines as often as you like.

Every place you can reach by mixing the two rides is your reachable city.

Now you buy a pass for a third line, but it runs exactly along the first one. Your reachable city does not grow by a single street.

Small numbers. With $\mathbf{v}_1=(1,0)$ and $\mathbf{v}_2=(0,1)$ , the mix $3\mathbf{v}_1+2\mathbf{v}_2=(3,2)$ . In fact any point $(a,b)$ is $a\mathbf{v}_1+b\mathbf{v}_2$ . These two arrows reach the whole plane.

Now try $\mathbf{v}_1=(1,2)$ and $\mathbf{v}_2=(2,4)$ . Every mix, like $3\mathbf{v}_1+\mathbf{v}_2=(5,10)$ , is a multiple of $(1,2)$ . Two arrows, but they only reach one line.

A mix $\lambda_1\mathbf{v}_1+\cdots+\lambda_k\mathbf{v}_k$ is called a linear combination. The set of every mix you can make is the span:

\begin{aligned}\operatorname{span}(\mathbf{v}_1,\dots,\mathbf{v}_k)=\{&\lambda_1\mathbf{v}_1+\cdots+\lambda_k\mathbf{v}_k :\\ &\text{any numbers } \lambda_i\}\end{aligned}

Watch a span get painted, one mix at a time:

Try: press ▶ sweep every mix: the trip point visits hundreds of mixes and paints the span. Then press 3 coplanar and sweep again: the paint stays flat. Now 3 independent: the third arrow lifts the paint off the plane and fills the room.

drag the picture to orbit

vectors in play: 2 dim span = 2

—

The golden dot is one mix, $\lambda_1\mathbf{v}_1+\lambda_2\mathbf{v}_2+\lambda_3\mathbf{v}_3$ , drawn leg by leg. However you slide the λ's, it cannot leave the span. "dim span" counts the different directions you can reach, not the vectors you own.

The realization

The span is every mix $\lambda_1\mathbf{v}_1+\cdots+\lambda_k\mathbf{v}_k$ . It is always a flat through the origin, so it is always a subspace: a point, a line, a plane, or more.

Its size depends on how many different directions the arrows point in. A new arrow that lies inside the current span adds nothing. An arrow that points out of it adds a whole new direction.

If you want the algebra · why a span is always a subspace

Prove it · a span is a subspace — derived

Claim. $U=\operatorname{span}(\mathbf{x}_1,\dots,\mathbf{x}_k)$ passes S1, S2 and S3.

(S1)

\mathbf{0}=0\,\mathbf{x}_1+\cdots+0\,\mathbf{x}_k\in U

. The all-zero mix goes nowhere, and "nowhere" is a legal place.

(S2) Take two members

\mathbf{u}=\sum_i\lambda_i\mathbf{x}_i

and

\mathbf{v}=\sum_i\mu_i\mathbf{x}_i

. Then

\begin{aligned}\mathbf{u}+\mathbf{v}&=\sum_i\lambda_i\mathbf{x}_i+\sum_i\mu_i\mathbf{x}_i\\ &=\sum_i(\lambda_i+\mu_i)\,\mathbf{x}_i,\end{aligned}

one mix with weights

\lambda_i+\mu_i

. So

\mathbf{u}+\mathbf{v}\in U

. Regroup term by term, then use

(\lambda+\mu)\mathbf{x}=\lambda\mathbf{x}+\mu\mathbf{x}

backwards.

(S3) For any number

c

\begin{aligned}c\,\mathbf{u}&=c\sum_i\lambda_i\mathbf{x}_i\\ &=\sum_i(c\lambda_i)\,\mathbf{x}_i,\end{aligned}

a mix with weights

c\lambda_i

. So

c\mathbf{u}\in U

. Subspace. ∎ Share

c

out over the sum (V2), then combine the scalings (V3). Rescaled weights are still weights.

Watch out — span is everywhere in ML

Linear regression's predictions $\hat{\mathbf{y}}=X\mathbf{w}$ always lie in the span of the columns of $X$ . PCA squeezes data onto the span of a few top directions. "What can this model possibly produce?" is usually a question about a span.

Pause & predict

Two vectors in $\mathbb{R}^3$ point along the same line (one is $-\tfrac34$ times the other). Their span is…

In one sentence: the span is everywhere your arrows can take you, and it only grows when a new arrow points somewhere new.

Linear independence — the art of non-redundancy

Imagine this

Three friends give you directions to a chai stall. Ravi says "1 block east". Meena says "1 block north". Arjun says "1 block east and 2 blocks north".

Arjun told you nothing new. His route is Ravi's plus two of Meena's. So you could walk Ravi's route, then Meena's twice, then Arjun's backwards, and you would be home again.

When a set of directions has a round trip like this, it is called dependent.

The chai-stall trip in numbers: $\mathbf{v}_1=(1,0)$ , $\mathbf{v}_2=(0,1)$ , $\mathbf{v}_3=(1,2)$ , and

1\,\mathbf{v}_1+2\,\mathbf{v}_2-1\,\mathbf{v}_3=(1,0)+(0,2)-(1,2)=(0,0).

So the test for redundancy is one question: can the vectors bring you back to $\mathbf{0}$ with weights that are not all zero?

\lambda_1\mathbf{x}_1+\lambda_2\mathbf{x}_2+\cdots+\lambda_k\mathbf{x}_k=\mathbf{0},\qquad\text{some }\lambda_i\neq 0\ ?

Yes → dependent. Rearrange the equation and one vector is a mix of the others. No → independent. The only way home is to never leave (all weights 0), and every vector brings something the others cannot copy.

Two quick shortcuts. ① If $\mathbf{0}$ is one of your vectors, the set is dependent: put weight 1 on $\mathbf{0}$ and 0 on everything else. ② For non-zero vectors, "dependent" means exactly "at least one is a mix of the others".

Feel it on two tiny examples

Dependent: $\binom{1}{2}$ and $\binom{2}{4}$ . The second is the first, doubled, so $2\binom{1}{2}-1\binom{2}{4}=\binom{0}{0}$ . A round trip with weights $(2,-1)$ . Caught.

Independent: $\binom{1}{0}$ and $\binom{1}{1}$ . Try $\lambda_1\binom{1}{0}+\lambda_2\binom{1}{1}=\binom{0}{0}$ . The bottom row says $\lambda_2=0$ . Then the top row says $\lambda_1=0$ . Only the "never leave" trip works.

Dependence is a round trip. Left: out along one arrow and back along the other, with weights $(2,-1)$ , lands you home. Right: no such trip exists, because the two arrows point in truly different directions.

Now take the round trip into 3D. Set the weights and see whether the walk ends at home:

Try: on a dependent trio, press ▶ find the way home: the walk bends back and lands on the origin. Then load staircase trio and press it again: no non-zero weights ever bring you home.

drag the picture to orbit

weight c₁1

weight c₂1

weight c₃1

—

distance from home = —

—

The realization

Vectors are independent when the only way to make $\lambda_1\mathbf{x}_1+\cdots+\lambda_k\mathbf{x}_k=\mathbf{0}$ is with every $\lambda_i=0$ . You can only get home by never leaving.

If some other weights bring you home, the set is dependent, and at least one vector is a mix of the rest: extra baggage.

Pause & predict

A set of five vectors happens to include $\mathbf{0}$ . What can you say straight away?

In one sentence: vectors are independent when the only walk that brings you home is the one where you never leave.

The detector: elimination hunts redundancy

Imagine this

You are packing for a trip. You pick up each item and ask one question: "Does this add something I don't already have?" A second charger for the same phone adds nothing, so it stays home.

Gaussian elimination asks the same question of each vector, left to right: does it bring a new direction, or is it a mix of the ones before it?

Small example from the slides. Put three vectors side by side as columns and eliminate:

\begin{pmatrix}1&2&3\\2&4&4\end{pmatrix}\xrightarrow{R_2-2R_1}\begin{pmatrix}1&2&3\\0&0&-2\end{pmatrix}

Column 1 has a pivot. Column 2 lost its chance: below the first pivot it is all zeros. So column 2 brings nothing new. Indeed $(2,4)=2\times(1,2)$ . Column 3 gets a pivot in row 2, so it brings a new direction.

The rule: stack the vectors as columns, eliminate, read the pivots. A pivot column is a genuine new direction. A column with no pivot is a mix of the pivot columns to its left, and in the fully reduced form its numbers are the exact recipe. Step through it:

Try: load The recipes example and press ▶ play. Watch the column cards: three light up as new directions, and column 4 is caught as a mix. Then edit one entry of v₄ and see whether it earns a pivot back.

Your vectors (edit any entry) — columns v₁, v₂, …

Elimination, step by step — pivots glow

step 0 / 0

rank = — dim span = —

—

The highlighted columns on the left are the original pivot columns. Keep those and you keep the whole span. The reduced matrix on the right is an X-ray: it tells you which columns matter and how the others are mixed, but the vectors you keep are always the originals.

The realization

Stack the vectors as columns and eliminate. A pivot column brings a new direction. A column without a pivot is a mix of earlier columns, and in the reduced form its numbers are the exact recipe.

The number of pivots, the rank, is the number of truly different directions. If every column has a pivot, the vectors are independent.

But row operations scramble every column. Why can we trust a recipe read off the scrambled matrix? Because they scramble every column in exactly the same way, so any mixing rule between columns survives. The drawer has the details.

If you want the algebra · why the reduced column is the recipe for the original column

Prove it · a non-pivot RREF column is a recipe for the original column — derived

Claim. If $R=\operatorname{rref}(A)$ and column $j$ of $R$ has no pivot, then the numbers in that column (rows 1 to rank) are the weights that build the original column $\mathbf{a}_j$ from the original pivot columns.

Every row operation is multiplying on the left by an invertible matrix (swap two rows, scale a row, add a multiple of one row to another). After all of them,

R=E_s\cdots E_2E_1\,A=:E\,A,

and

E

is invertible. Each row operation can be undone by another row operation, and a product of invertible matrices is invertible.

Multiplying on the left acts on each column separately: column

j

R

\mathbf{r}_j=E\,\mathbf{a}_j

. So for any weights

c_i

\begin{aligned}&\textstyle\sum_i c_i\mathbf{a}_i=\mathbf{0}\\ \iff\ &\textstyle\sum_i c_i\mathbf{r}_i=\mathbf{0}.\end{aligned}

"⇒": apply

E

to both sides. "⇐": apply

E^{-1}

. So

A

and

R

obey exactly the same mixing rules among their columns.

R

, the

i

-th pivot column is the unit vector

\mathbf{e}_i

, and a non-pivot column

\mathbf{r}_j

has numbers

\rho_1,\dots,\rho_r

in the pivot rows and zeros below. So, just by reading,

\mathbf{r}_j=\rho_1\mathbf{r}_{p_1}+\cdots+\rho_r\mathbf{r}_{p_r}.

In reduced form each pivot column is a single 1, so a column is literally the list of its own weights.

That is a mixing rule among the columns of

R

. By step 2 the same rule holds among the columns of

A

\mathbf{a}_j=\rho_1\mathbf{a}_{p_1}+\cdots+\rho_r\mathbf{a}_{p_r}.

The X-ray reads the original recipe. ∎ The same argument shows the original pivot columns are independent: their reduced versions are different unit vectors, and step 2 says "which weights give 0" is the same question for

A

and

R

Pause & predict

After row-reducing $A$ , the third column has no pivot and reads $(3,-1,0)^\top$ . Which statement is right?

In one sentence: eliminate the columns, and every column without a pivot is exposed as a mix of the pivot columns before it.

The recipes theorem — independence survives translation

Imagine this

A cook has two basic masalas that are truly different: neither can be made from the other. Every dish she makes comes with a recipe card: "1 spoon of the first, half a spoon of the second".

Question: are two of her dishes secretly the same mix under different names? She does not need to taste the food. Comparing the recipe cards is enough.

Small example. Let $\mathbf{b}_1,\mathbf{b}_2$ be independent. Dish $\mathbf{x}_1=\mathbf{b}_1+\mathbf{b}_2$ has card $(1,1)$ . Dish $\mathbf{x}_2=2\mathbf{b}_1+2\mathbf{b}_2$ has card $(2,2)$ . The cards are dependent: card 2 is twice card 1. And sure enough, the dishes are too: $\mathbf{x}_2=2\mathbf{x}_1$ .

In symbols: put the ingredients side by side as $B=[\mathbf{b}_1\ \cdots\ \mathbf{b}_k]$ and each recipe card as a column $\boldsymbol{\lambda}_j$ . Cooking is a matrix product, $\mathbf{x}_j=B\boldsymbol{\lambda}_j$ . The theorem:

Aha

$\{\mathbf{x}_1,\dots,\mathbf{x}_m\}$ independent $\iff$ $\{\boldsymbol{\lambda}_1,\dots,\boldsymbol{\lambda}_m\}$ independent.

Reading the recipe cards is enough. Whatever is true of the dishes is true of the cards, and the other way round. That mirror is the whole reason coordinates are useful.

Try: press ▶ slide into line: x₂'s card slides to (2, 1), twice x₁'s card, and both floors flag dependence at the same instant. The faint slanted grid on the left is the recipe grid seen through the mirror.

drag the picture to orbit

x₂ recipe: λ₁₂−1

x₂ recipe: λ₂₂1.5

—

x₁'s card is (1, 0.5). Line x₂'s card up with it and the dishes line up at the same moment. The "+ 3rd dish" button shows the pigeonhole law: cards here have only 2 numbers, and three 2-number cards can never be independent, so three dishes cannot be either.

The pigeonhole law. Cards over $k$ ingredients are vectors in $\mathbb{R}^k$ , and more than $k$ of them can never be independent (a $k\times m$ matrix has at most $k$ pivots). By the mirror, $m>k$ dishes are dependent too, without tasting a single one.

The realization

If the ingredients $\mathbf{b}_i$ are independent, then the dishes $\mathbf{x}_j=B\boldsymbol{\lambda}_j$ are independent exactly when their recipe cards $\boldsymbol{\lambda}_j$ are. So you can always do the check on the small, simple cards instead of the big dishes.

And more cards than ingredients ( $m>k$ ) always means dependent.

The slides run this at full size: four dishes from four ingredients, with recipe cards $(1,-2,1,-1)$ , $(-4,-2,0,4)$ , $(2,3,-1,-3)$ and $(17,-10,11,1)$ . Elimination gives pivots in columns 1–3, and column 4 reads $(-7,-15,-18)$ . So

\boldsymbol{\lambda}_4=-7\boldsymbol{\lambda}_1-15\boldsymbol{\lambda}_2-18\boldsymbol{\lambda}_3 \quad\Longrightarrow\quad 7\mathbf{x}_1+15\mathbf{x}_2+18\mathbf{x}_3+\mathbf{x}_4=\mathbf{0}

by the mirror. The four dishes are dependent, and we know the exact relation. (Load preset 3 in the detector above and read it off yourself.)

If you want the algebra · the recipes theorem, both directions

Prove it · the recipes theorem, both directions — derived

Setting. $B=[\mathbf{b}_1\cdots\mathbf{b}_k]$ has independent columns, and $\mathbf{x}_j=B\boldsymbol{\lambda}_j$ for $j=1,\dots,m$ . Pick any weights $\psi_1,\dots,\psi_m$ and write $\mathbf{v}=\psi_1\boldsymbol{\lambda}_1+\cdots+\psi_m\boldsymbol{\lambda}_m$ .

A mix of dishes is

B

applied to the same mix of cards:

\begin{aligned}&\psi_1\mathbf{x}_1+\cdots+\psi_m\mathbf{x}_m\\ &\quad=\psi_1B\boldsymbol{\lambda}_1+\cdots+\psi_mB\boldsymbol{\lambda}_m\\ &\quad=B\,\mathbf{v}.\end{aligned}

Put in

\mathbf{x}_j=B\boldsymbol{\lambda}_j

, then pull

B

out of the sum.

Key fact:

B\mathbf{v}=\mathbf{0}\iff\mathbf{v}=\mathbf{0}

. "⇐" is clear. "⇒":

B\mathbf{v}=v_1\mathbf{b}_1+\cdots+v_k\mathbf{b}_k

is a mix of independent columns. If it is

\mathbf{0}

, every

v_i

must be 0.

(Cards independent ⇒ dishes independent.) Suppose

\sum_j\psi_j\mathbf{x}_j=\mathbf{0}

. By step 1,

B\mathbf{v}=\mathbf{0}

. By step 2,

\mathbf{v}=\sum_j\psi_j\boldsymbol{\lambda}_j=\mathbf{0}

. The cards are independent, so every

\psi_j=0

. A round trip among the dishes is forced to be a round trip among the cards, and that is forced to be the "never leave" trip.

(Dishes independent ⇒ cards independent.) Suppose

\sum_j\psi_j\boldsymbol{\lambda}_j=\mathbf{0}

, so

\mathbf{v}=\mathbf{0}

. Then

\sum_j\psi_j\mathbf{x}_j=B\mathbf{v}=B\mathbf{0}=\mathbf{0}

. The dishes are independent, so every

\psi_j=0

. Both directions hold. ∎ This direction never needed step 2: a rule among the cards is carried by

B

into the same rule among the dishes.

Pause & predict

Someone hands you 50 word-embedding vectors that live in a 32-dimensional space. Independent or not?

In one sentence: with independent ingredients, dishes are independent exactly when their recipe cards are, so you can always check the cards instead.

Basis — the just-right toolkit

Imagine this

You ask two people the way to the same house. The first says "3 blocks east, 2 blocks north". The second thinks in terms of the diagonal main road and says "1 block east, then 2 blocks along the main road".

Same house, two addresses. They are using different sets of directions. Each set of directions is a basis, and the numbers in an address are the coordinates.

Check the numbers. East is $(1,0)$ , north is $(0,1)$ , and one block along the main road is $(1,1)$ . The first address: $3(1,0)+2(0,1)=(3,2)$ . The second: $1(1,0)+2(1,1)=(1,0)+(2,2)=(3,2)$ . Same point.

A good set of directions must do two jobs. It must reach every place (it spans the space), and it must carry no extra baggage (it is independent). "North, east and north-east" reaches everything but wastes a direction. "North" alone wastes nothing but misses most places. "North and east" is just right.

Four ways to say the same thing. A basis of $V$ is:

①an independent set that spans $V$ .Reaches everything, no extra baggage.
②a smallest spanning set.Remove any one vector and some places become unreachable.
③a largest independent set.Add any vector of $V$ and redundancy appears.
④a perfect address book.Every $\mathbf{x}\in V$ has exactly one recipe $\mathbf{x}=\sum_i\lambda_i\mathbf{b}_i$ . The numbers $\lambda_i$ are its coordinates.

Walk the same house through different address books:

Try: keep v where it is and switch between Canonical, Staircase and Rotated: the graph paper re-rules itself, the house never moves, only its address changes. Press ▶ walk the address to follow the recipe. Then press Broken.

drag the picture to orbit

v (east)3

v (north)2

—

The realization

A basis reaches everything with nothing extra. Because of that, every vector has exactly one address:

$\mathbf{x}=\lambda_1\mathbf{b}_1+\cdots+\lambda_k\mathbf{b}_k$

The $\lambda_i$ are the coordinates. Change the basis and the address changes, but the point itself stays put. A dependent pair cannot be a basis: the graph paper collapses to a line.

If you want the algebra · why the address is unique

Prove it · a basis gives unique coordinates — derived

Claim. If $\mathbf{b}_1,\dots,\mathbf{b}_k$ is a basis of $V$ , every $\mathbf{x}\in V$ has one and only one list of coordinates.

At least one recipe exists:

\mathbf{x}=\lambda_1\mathbf{b}_1+\cdots+\lambda_k\mathbf{b}_k

for some

\lambda_i

. That is what "spans" means: every vector of

V

can be reached.

Suppose a second recipe also works:

\mathbf{x}=\psi_1\mathbf{b}_1+\cdots+\psi_k\mathbf{b}_k

. Subtract the two:

\begin{aligned}\mathbf{0}&=\mathbf{x}-\mathbf{x}\\ &=(\lambda_1-\psi_1)\mathbf{b}_1+\cdots\\ &\quad+(\lambda_k-\psi_k)\mathbf{b}_k.\end{aligned}

Same vector, two recipes. Subtracting gives a mix of the basis vectors that lands on

\mathbf{0}

: a round trip.

The

\mathbf{b}_i

are independent, so the only round trip is the all-zero one:

\lambda_i-\psi_i=0

, so

\lambda_i=\psi_i

for every

i

. The two recipes were the same. ∎ "No extra baggage" and "only one address" are the same promise, read in two directions.

Watch out — why "change of basis" runs half of ML

PCA is "find a new basis for the data, ordered by importance". The Fourier transform is "write the signal in a basis of waves". Autoencoders learn a basis. The data never changes; only the address book does. In the right book, the pattern you are hunting becomes easy to see.

Pause & predict

Three independent vectors in $\mathbb{R}^4$ . Do they form a basis of $\mathbb{R}^4$ ?

In one sentence: a basis is a set of directions that reaches everything with nothing to spare, so every vector gets exactly one address.

Dimension — how big is the board, really?

Imagine this

To say where a train is, you need one number: the kilometre mark on the line. To say where a shop is on a city map, you need two. To say where a drone is, you need three: east, north and height.

Choose your directions however you like. The number of numbers you need never changes. That number is the dimension.

Small example. Take the $y$ -axis in the flat plane. Its points look like $(0,\,1.5)$ or $(0,\,-1)$ : two numbers each. But one number, "how far up", already tells you everything. So the $y$ -axis has 2 components but dimension 1.

The key fact: pick any basis of a space and count its vectors. Pick a completely different basis and count again. You always get the same number. That count is the dimension, $\dim(V)$ .

So $\dim(\mathbb{R}^n)=n$ , and $\dim(\mathbb{R}^{m\times n})=mn$ (one basis matrix for each cell: a 1 in that cell, 0 everywhere else). A subspace is never bigger than its home: $\dim(U)\le\dim(V)$ , with equality only when $U=V$ .

Count the knobs yourself. Pick vectors from inside a space, one at a time, and watch the counter:

Try: choose a plane and press ▶ pick six vectors. The counter climbs to 2 and then refuses to move. Press ↺ start over for a completely different set of picks: it stops at 2 again. Then try all of ℝ³.

drag the picture to orbit

0independent directions found
0 vectors picked

components per vector: 3dimension: 2

—

The realization

Every basis of a space has the same number of vectors. That number is the dimension: the count of independent directions, the number of knobs you really have.

Components count the room you are written in. Dimension counts the room you actually use. And more vectors than the dimension always means redundancy.

If you want the algebra · why all bases have the same size

Prove it · all bases have the same size — derived

Claim. If $\mathcal{B}=\{\mathbf{b}_1,\dots,\mathbf{b}_m\}$ and $\mathcal{C}=\{\mathbf{c}_1,\dots,\mathbf{c}_k\}$ are both bases of $V$ , then $m=k$ .

Write each

\mathbf{b}_j

as a recipe over

\mathcal{C}

\mathbf{b}_j=C\boldsymbol{\lambda}_j

with

C=[\mathbf{c}_1\cdots\mathbf{c}_k]

and

\boldsymbol{\lambda}_j\in\mathbb{R}^k

\mathcal{C}

spans

V

, so every

\mathbf{b}_j

has a recipe card, and the card has

k

numbers.

Suppose

m>k

. Then the

m

cards

\boldsymbol{\lambda}_1,\dots,\boldsymbol{\lambda}_m\in\mathbb{R}^k

are dependent. Pigeonhole: side by side they form a

k\times m

matrix with at most

k

pivots, so at least one column has no pivot.

By the recipes theorem (

\mathcal{C}

is independent, so the mirror works), dependent cards mean the dishes

\mathbf{b}_1,\dots,\mathbf{b}_m

are dependent. But

\mathcal{B}

is a basis, so that is impossible. Hence

m\le k

. A basis is independent by definition. The assumption

m>k

broke that, so it was false.

Swap the roles of

\mathcal{B}

and

\mathcal{C}

, and the same three steps give

k\le m

. Therefore

m=k

. ∎ Nothing in steps 1–3 cared which basis was which, so the argument runs both ways.

Watch out — components ≠ dimension

A photo is written with 150,528 numbers, but real photos do not fill that whole space. Faces, speech and text seem to sit on a thin, curled surface of much smaller dimension inside it. This bet is called the manifold hypothesis, and much of representation learning is the hunt for that small true dimension.

Components ≠ dimension. Left: points on the $y$ -axis are written with two numbers, but one number locates them. Right: an image is written with 150,528 numbers, yet natural images cluster on a thin surface of far smaller dimension.

So what for ML: an embedding table with one row per word and $d$ columns has rank at most $d$ . "How many independent directions does this representation really use?" is a dimension question, and the answer is the rank.

Pause & predict

$V=\operatorname{span}\!\big(\binom{0}{1}\big)$ , a set inside $\mathbb{R}^2$ . What is $\dim(V)$ ?

In one sentence: dimension is the number of independent directions a space really has, and every basis agrees on it.

Finding a basis — the algorithm

Imagine this

You have a drawer full of old phone chargers, and many are duplicates. You want the smallest set that still charges every device you own.

So you pick them up one at a time. You keep a charger only if it does something the ones you kept cannot. That is exactly how to find a basis.

You are handed a pile of vectors $\mathbf{x}_1,\dots,\mathbf{x}_m$ and you want a basis of everything they span, $U=\operatorname{span}(\mathbf{x}_1,\dots,\mathbf{x}_m)$ . Three steps:

①Stack the vectors as the columns of a matrix $A$ .
②Row-reduce $A$ and find the pivot columns.
③Keep the original vectors that sit over the pivot columns. Throw the rest away.

The kept vectors are independent (each one has a pivot), and they still span everything (each discarded vector is a mix of kept ones). Count them, and you have measured $\dim(U)$ too. Watch the charger test happen in 3D:

Try: press ▶ run the sieve on a plane's worth. x₁ and x₂ are kept and open up a glass plane; x₃ and x₄ land inside it and drop out, each showing its recipe. Then try a room's worth: x₄ points out of the plane and fills the room.

drag the picture to orbit

—

basis: { }dim = 0

—

Watch out — the classic slip

The basis is the original columns of $A$ at the pivot positions, not the tidy columns of the reduced matrix. Row operations change how each column looks, while keeping every mixing rule between columns (Section 8 showed why). Read the pivot positions from the reduced matrix; take the vectors from $A$ .

X-ray vs skeleton. The pivot positions (1, 2, 4) and the recipe $(-1,2)$ come from $\operatorname{rref}(A)$ . The basis vectors $\mathbf{x}_1,\mathbf{x}_2,\mathbf{x}_4$ come from $A$ .

The slide example, worked to the end. Four vectors in $\mathbb{R}^5$ :

\mathbf{x}_1=\begin{pmatrix}1\\2\\-1\\-1\\-1\end{pmatrix},\ \mathbf{x}_2=\begin{pmatrix}2\\-1\\1\\2\\-2\end{pmatrix},\ \mathbf{x}_3=\begin{pmatrix}3\\-4\\3\\5\\-3\end{pmatrix},\ \mathbf{x}_4=\begin{pmatrix}-1\\8\\-5\\-6\\1\end{pmatrix}

Stack them and reduce (or load preset 4 in the detector of Section 8 and watch):

\operatorname{rref}(A)=\begin{pmatrix}1&0&-1&0\\0&1&2&0\\0&0&0&1\\0&0&0&0\\0&0&0&0\end{pmatrix}

Pivots sit in columns 1, 2 and 4. Column 3 is the only redundant one, and its numbers $(-1,2)$ give the recipe $\mathbf{x}_3=-\mathbf{x}_1+2\mathbf{x}_2$ . (Check it on all five entries. It works.)

The realization

To find a basis of a span: stack, reduce, and keep the original vectors in the pivot columns. The number you keep is the dimension.

For the slide example: a basis of $U$ is $\{\mathbf{x}_1,\mathbf{x}_2,\mathbf{x}_4\}$ , and $\dim(U)=3$ . Four vectors walked in, one was a mix, and three genuine directions remain.

Note for companion-PDF readers — a slip in this example

The companion's version of this example reports the basis $\{\mathbf{x}_1,\mathbf{x}_2\}$ and $\dim=2$ , using $\mathbf{x}_4=3\mathbf{x}_1-2\mathbf{x}_2$ . Check the fourth entry: $3(-1)-2(2)=-7$ , but the fourth entry of $\mathbf{x}_4$ is $-6$ . That mix fails, $\mathbf{x}_4$ earns its own pivot, and the numbers above (three pivots, $\dim U=3$ , pivots in columns 1, 2, 4) are machine-checked. The detector's preset 4 reproduces them.

One more from the slides, the other way round: $(1,2,3,4)$ , $(2,-1,0,2)$ and $(1,1,0,4)$ in $\mathbb{R}^4$ reduce with a pivot in every column, so they are independent. Yet they are not a basis of $\mathbb{R}^4$ : three arrows cannot cover a four-dimensional world. Independent is not enough until the count matches the dimension.

In one sentence: to shrink a pile of vectors to a basis, keep each vector only if it points outside the span of the ones you already kept.

The whole story, in six lines

①A group is combining with an undo button. Add scaling and you get a vector space: the board every ML object lives on.
②A subspace is a flat world through the origin inside a bigger one. The null space and the column space are the star examples.
③The span is everywhere your vectors can reach. It never grows when you add a vector that points nowhere new.
④Independence means no round trip home except "never leave". Elimination's pivots detect it.
⑤A basis spans and is independent: the smallest toolkit, giving every vector exactly one address.
⑥Dimension, the size of any basis, is the board's true size. More vectors than the dimension always means redundancy.

The one thing to remember

Four questions to ask of any vector or matrix you meet in ML: What space does it live in? Is this set independent, or is some of it redundant? What is its span, column space or null space? Is there a smarter basis? That loop is most of the geometry behind classical ML, and a surprising amount of deep learning.

Take these home — the companion's five practice problems (with answer keys)

P1. Which are subspaces of $\mathbb{R}^3$ ? (a) $x+y+z=0$ (b) $x+y+z=1$ (c) $x\ge0$ (d) $x=2y$ . Key: (a) and (d), flat and through the origin. (b) misses 0; (c) fails under λ = −1.

P2. Express $\mathbf{b}=(4,3,1,2)^T$ in terms of $\mathbf{v}_1=(1,-1,0,1)^T,\ \mathbf{v}_2=(0,1,-2,2)^T,\ \mathbf{v}_3=(3,-1,0,1)^T,\ \mathbf{v}_4=(1,1,1,1)^T$ . Key: row-reduce $[\,\mathbf{v}_1\,\mathbf{v}_2\,\mathbf{v}_3\,\mathbf{v}_4\,|\,\mathbf{b}\,]$ . The unique answer is $\mathbf{b}=-\tfrac{15}{7}\mathbf{v}_1+\tfrac{3}{7}\mathbf{v}_2+\tfrac{10}{7}\mathbf{v}_3+\tfrac{13}{7}\mathbf{v}_4$ .

P3. $\mathbf{b}_1,\mathbf{b}_2,\mathbf{b}_3$ independent; $\mathbf{x}_1=\mathbf{b}_1+\mathbf{b}_2,\ \mathbf{x}_2=\mathbf{b}_2+\mathbf{b}_3,\ \mathbf{x}_3=\mathbf{b}_1+\mathbf{b}_3$ . Independent? Key: check the cards $(1,1,0),(0,1,1),(1,0,1)$ : three pivots, independent. The mirror does the rest.

P4. Is $S=\{(2,3,5),(5,7,9),(1,11,1)\}$ a basis of $\mathbb{R}^3$ ? Key: row-reduce, get 3 pivots, so independent; 3 independent vectors in a 3-D space automatically span it. Yes.

P5. Basis and dimension of $\operatorname{span}\big((1,2,1,3),(2,4,2,6),(1,0,1,1),(3,2,3,5)\big)\subseteq\mathbb{R}^4$ . Key: pivots in columns 1 and 3, with $\mathbf{v}_2=2\mathbf{v}_1$ and $\mathbf{v}_4=\mathbf{v}_1+2\mathbf{v}_3$ . Basis $\{\mathbf{v}_1,\mathbf{v}_3\}$ , dimension 2. (Preset 5 in the detector.)

In one sentence: know your space, spot the redundancy, find the span, and pick the smartest basis.

Practice arena — the unit's problem set, solved in full

This is Prof. Saurabh's practice set for this unit (Vector Spaces): Problems 1–6 drill the subspace test, Problems 7–12 cover linear combinations and independence, and Problems 13–18 build bases and count dimensions. Attempt each problem on paper first — the fold-out solution shows every step, nothing skipped.

Problem 1Easy

Show that $W = \{(x, 0) : x \in \mathbb{R}\}$ is a subspace of $\mathbb{R}^2$ .

Show the worked solution — every step

Strategy. Apply the three subspace tests: does

W

contain

\mathbf{0}

, and is it closed under addition and scalar multiplication?

Step 1 · Zero vector. Take

x = 0

. Then

(0, 0)

is in

W

. ✓

Step 2 · Closed under addition. Take

u = (a, 0)

and

v = (b, 0)

W

. Then

u + v = (a + b,\ 0)

. The second coordinate is still

0

, so

u + v \in W

. ✓

Step 3 · Closed under scalar multiplication. For

c \in \mathbb{R}

and

u = (a, 0) \in W

c \cdot u = (ca,\ 0) \in W

. ✓

Answer. All three tests pass, so

W

is a subspace of

\mathbb{R}^2

— and therefore a vector space in its own right.

Key takeaway

In simple words: $W$ is just the $x$ -axis. A line through the origin in $\mathbb{R}^2$ is always a subspace.

Problem 2Easy

Determine whether $W = \{(x, y) \in \mathbb{R}^2 : x + y = 0\}$ is a subspace of $\mathbb{R}^2$ .

Show the worked solution — every step

Strategy. The rule

x + y = 0

is linear and homogeneous — promising.

Step 1 · Zero vector.

0 + 0 = 0

, so

(0, 0) \in W

. ✓

Step 2 · Closed under addition. If

(a, b)

and

(c, d)

are in

W

, then

a + b = 0

and

c + d = 0

. Their sum is

(a + c,\ b + d)

, and

(a + c) + (b + d) = (a + b) + (c + d) = 0 + 0 = 0.

So the sum is in

W

. ✓

Step 3 · Closed under scalar multiplication. For

k \in \mathbb{R}

k(a, b) = (ka,\ kb)

with

ka + kb = k(a + b) = k \cdot 0 = 0

. ✓

Answer.

W

is a subspace of

\mathbb{R}^2

. Geometric picture:

W

is the line

y = -x

through the origin.

Key takeaway

Linear + homogeneous (right-hand side $0$ ) is the signature of a subspace: the defining rule survives adding and scaling automatically.

Problem 3Medium

Show that $W = \{(x, y, z) \in \mathbb{R}^3 : x + y + z = 0\}$ is a subspace of $\mathbb{R}^3$ .

Show the worked solution — every step

Strategy. A linear homogeneous equation in 3 variables — likely a plane through the origin.

Step 1 · Zero vector.

0 + 0 + 0 = 0

, so

(0, 0, 0) \in W

. ✓

Step 2 · Closed under addition. If

x_1 + y_1 + z_1 = 0

and

x_2 + y_2 + z_2 = 0

, then

\begin{aligned}(x_1 + x_2) + (y_1 + y_2) + (z_1 + z_2) &= (x_1 + y_1 + z_1) + (x_2 + y_2 + z_2)\\ &= 0 + 0 = 0. \end{aligned}

✓

Step 3 · Closed under scaling.

c \cdot (x + y + z) = c \cdot 0 = 0

, so

cx + cy + cz = 0

and

c(x, y, z) \in W

. ✓

Answer.

W

is a subspace of

\mathbb{R}^3

. Geometrically, it is a plane through the origin.

Key takeaway

Every linear homogeneous equation in $\mathbb{R}^3$ carves out a flat sheet through the origin. Keep this plane in mind — Problem 16 comes back to it and measures it: basis of two vectors, dimension 2.

Problem 4Medium

Show that $W = \{(x, y) \in \mathbb{R}^2 : x \ge 0\}$ is NOT a subspace of $\mathbb{R}^2$ .

Show the worked solution — every step

Strategy. Only one failure is needed for disproof. Try a negative scalar.

Step 1 · Pick a member and a scalar. Take

u = (2, 3) \in W

(since

2 \ge 0

) and

c = -1

Step 2 · Scale it.

c \cdot u = (-2, -3)

, and

-2

is not

\ge 0

. So

c \cdot u \notin W

Step 3 · Conclude.

W

fails closure under scalar multiplication, hence

W

is not a subspace.

Answer. Not a subspace — the single counterexample

u = (2,3)

c = -1

breaks closure under scalar multiplication.

Watch out

Sets defined with strict inequalities (like $x > 0$ ), nonzero constants (like $x + y = 5$ ), or products of variables (like $xy = 0$ ) typically fail the test. Subspaces correspond to linear, homogeneous conditions like $x + 2y - 3z = 0$ .

Key takeaway

In simple words: $W$ is the right half-plane. Negating a vector flips it to the other side, violating closure. One concrete counterexample is a complete disproof.

Problem 5Hard

Let $V$ be the set of all $2 \times 2$ real matrices. Show that the set $W$ of all $2 \times 2$ symmetric matrices (matrices $A$ with $A^{\mathsf{T}} = A$ ) is a subspace of $V$ .

Show the worked solution — every step

Strategy. Use the transpose properties

(A + B)^{\mathsf{T}} = A^{\mathsf{T}} + B^{\mathsf{T}}

and

(cA)^{\mathsf{T}} = c \cdot A^{\mathsf{T}}

Step 1 · Zero matrix. The

2 \times 2

zero matrix equals its own transpose. So

\mathbf{0} \in W

. ✓

Step 2 · Closed under addition. If

A^{\mathsf{T}} = A

and

B^{\mathsf{T}} = B

, then

(A + B)^{\mathsf{T}} = A^{\mathsf{T}} + B^{\mathsf{T}} = A + B.

A + B \in W

. ✓

Step 3 · Closed under scaling.

(cA)^{\mathsf{T}} = c \cdot A^{\mathsf{T}} = c \cdot A

. So

cA \in W

. ✓

Answer. All three tests hold, so

W

is a subspace of

V

Wait — really?

The "vectors" in this problem are matrices. The subspace test never asked what the objects were — anything you can add and scale (tuples, matrices, polynomials, even functions) plays the vector game by the same rules.

Key takeaway

In simple words: symmetric matrices mirror across the diagonal. Adding or scaling preserves that mirror symmetry.

Problem 6Hard

Show that the set $P_n$ of all polynomials of degree $\le n$ (with the zero polynomial included) is a vector space under the usual polynomial addition and scalar multiplication.

Show the worked solution — every step

Strategy.

P_n

sits inside the vector space of all polynomials, so the 8 axioms (commutativity, associativity, distributivity, identity, inverses) are inherited from real addition. We just verify closure and the special elements.

Step 1 · Closure under addition. If

\deg(p) \le n

and

\deg(q) \le n

, then

\deg(p + q) \le n

. ✓

Step 2 · Closure under scaling. If

\deg(p) \le n

and

c \in \mathbb{R}

, then

\deg(c \cdot p) \le n

. ✓

Step 3 · Zero element. The zero polynomial

0(x)

is in

P_n

and

p + 0 = p

. ✓

Step 4 · Additive inverse. For

p(x) = a_0 + a_1 x + \dots + a_n x^n

, the polynomial

-p(x) = -a_0 - a_1 x - \dots - a_n x^n

is in

P_n

and

p + (-p) = 0

. ✓

Step 5 · Remaining axioms. Commutativity, associativity, distributivity (

(c + d)p = cp + dp

c(p + q) = cp + cq

), associativity of scalar product

(cd)p = c(dp)

, and identity

1 \cdot p = p

— all follow coefficient-wise from properties of real numbers. ✓

Answer. All 8 axioms hold, so

P_n

is a vector space.

Key takeaway

In simple words: polynomials behave like vectors — coefficients act like coordinates. (That is why $\dim(P_n) = n + 1$ : one coordinate per coefficient $a_0, a_1, \dots, a_n$ .)

Problem 7Easy

Express $v = (3, 7)$ as a linear combination of $u_1 = (1, 1)$ and $u_2 = (1, 2)$ .

Show the worked solution — every step

Strategy. Set

v = a \cdot u_1 + b \cdot u_2

and solve for

a, b

coordinate-wise.

Step 1 · Write the system. Matching first and second coordinates:

\begin{aligned} a + b &= 3\\ a + 2b &= 7 \end{aligned}

Step 2 · Eliminate. Subtract row 1 from row 2:

(a + 2b) - (a + b) = 7 - 3

, so

b = 4

Step 3 · Back-substitute.

a + 4 = 3 \Rightarrow a = -1

Step 4 · Check.

-1(1,1) + 4(1,2) = (-1 + 4,\ -1 + 8) = (3, 7)

. ✓

Answer.

v = -1 \cdot u_1 + 4 \cdot u_2

Key takeaway

A "find the recipe" question is always a linear system in disguise: one equation per coordinate, one unknown per ingredient vector.

Problem 8Easy

Are the vectors $(1, 2)$ and $(2, 4)$ linearly independent in $\mathbb{R}^2$ ?

Show the worked solution — every step

Strategy. Two vectors — check whether one is a scalar multiple of the other.

Step 1 · Spot the multiple.

(2, 4) = 2 \cdot (1, 2)

Step 2 · Exhibit a non-trivial combination.

2 \cdot (1, 2) + (-1) \cdot (2, 4) = (0, 0)

with non-zero scalars.

Answer. A non-trivial combination yields zero, so the vectors are linearly dependent.

Key takeaway

Picture: both arrows point along the same line — they encode the same direction. For exactly two vectors, dependent $\Leftrightarrow$ one is a scalar multiple of the other.

Problem 9Medium

Express $v = (1, -2, 5)$ as a linear combination of $u_1 = (1, 1, 1)$ , $u_2 = (1, 2, 3)$ , $u_3 = (2, -1, 1)$ .

Show the worked solution — every step

Strategy. Set up a

3 \times 3

system from

a \cdot u_1 + b \cdot u_2 + c \cdot u_3 = v

and solve.

Step 1 · Write the system. One equation per coordinate:

\begin{aligned} a + b + 2c &= 1 &&\text{(i)}\\ a + 2b - c &= -2 &&\text{(ii)}\\ a + 3b + c &= 5 &&\text{(iii)} \end{aligned}

Step 2 · (ii) − (i). Entry by entry:

a - a = 0

2b - b = b

-c - 2c = -3c

-2 - 1 = -3

. So

b - 3c = -3

…(iv)

Step 3 · (iii) − (i). Entry by entry:

a - a = 0

3b - b = 2b

c - 2c = -c

5 - 1 = 4

. So

2b - c = 4

…(v)

Step 4 · Solve the 2-variable system. From (iv):

b = 3c - 3

. Substitute into (v):

2(3c - 3) - c = 6c - 6 - c = 5c - 6 = 4 \Rightarrow 5c = 10 \Rightarrow c = 2

Step 5 · Back-substitute.

b = 3(2) - 3 = 3

. From (i):

a = 1 - b - 2c = 1 - 3 - 4 = -6

Step 6 · Check.

\begin{aligned}-6(1,1,1) + 3(1,2,3) + 2(2,-1,1) &= (-6,-6,-6) + (3,6,9) + (4,-2,2)\\ &= (1,-2,5). \end{aligned}

✓

Alternate route · Augmented matrix → REF. Place

u_1, u_2, u_3

as columns and

v

on the right:

[\,u_1 \mid u_2 \mid u_3 \mid v\,] = \left[\begin{array}{ccc|c} 1 & 1 & 2 & 1\\ 1 & 2 & -1 & -2\\ 1 & 3 & 1 & 5 \end{array}\right]

R_2 \to R_2 - R_1

(entries:

1-1=0

2-1=1

-1-2=-3

-2-1=-3

) and

R_3 \to R_3 - R_1

(entries:

1-1=0

3-1=2

1-2=-1

5-1=4

\left[\begin{array}{ccc|c} 1 & 1 & 2 & 1\\ 0 & 1 & -3 & -3\\ 0 & 2 & -1 & 4 \end{array}\right]

R_3 \to R_3 - 2R_2

(entries:

0

2-2=0

-1-2(-3)=5

4-2(-3)=10

\left[\begin{array}{ccc|c} 1 & 1 & 2 & 1\\ 0 & 1 & -3 & -3\\ 0 & 0 & 5 & 10 \end{array}\right]

Back-substitute:

5c = 10 \Rightarrow c = 2

;

b - 3c = -3 \Rightarrow b = 3

;

a + b + 2c = 1 \Rightarrow a = -6

. Same answer. ✓

Answer.

v = -6 \cdot u_1 + 3 \cdot u_2 + 2 \cdot u_3

Key takeaway

The REF recipe for linear-combination questions: augment with the target vector, reduce, back-substitute. It is the same computation as elimination by equations, just written compactly.

Problem 10Medium

Are the vectors $(1, 0, 1)$ , $(0, 1, 1)$ , $(1, 1, 0)$ linearly independent in $\mathbb{R}^3$ ?

Show the worked solution — every step

Strategy. Set

a \cdot v_1 + b \cdot v_2 + c \cdot v_3 = \mathbf{0}

and solve the homogeneous system.

Step 1 · Write the system. Coordinate by coordinate:

\begin{aligned} a + c &= 0\\ b + c &= 0\\ a + b &= 0 \end{aligned}

Step 2 · Express in terms of $c$ . From row 1,

a = -c

. From row 2,

b = -c

Step 3 · Plug into row 3.

-c + (-c) = 0 \Rightarrow -2c = 0 \Rightarrow c = 0

. Then

a = b = 0

Alternate route · Row reduce to REF. Form the matrix with the vectors as rows:

\begin{bmatrix} 1 & 0 & 1\\ 0 & 1 & 1\\ 1 & 1 & 0 \end{bmatrix}

R_3 \to R_3 - R_1

(entries:

1-1=0

1-0=1

0-1=-1

\begin{bmatrix} 1 & 0 & 1\\ 0 & 1 & 1\\ 0 & 1 & -1 \end{bmatrix}

R_3 \to R_3 - R_2

(entries:

0

1-1=0

-1-1=-2

\begin{bmatrix} 1 & 0 & 1\\ 0 & 1 & 1\\ 0 & 0 & -2 \end{bmatrix}

Three non-zero rows

\Rightarrow

rank

= 3

. Since rank equals the number of vectors, they are independent. ✓

Determinant shortcut. The determinant of the matrix with these rows is

-2 \ne 0

, confirming independence.

Answer. Only the trivial solution works, so the vectors are linearly independent.

Key takeaway

Rank $=$ number of vectors $\Leftrightarrow$ independent. And for $n$ vectors in $\mathbb{R}^n$ , the fastest test is $\det \ne 0 \Leftrightarrow$ independent.

Problem 11Hard

Are the polynomials $p_1(x) = 1 + x$ , $p_2(x) = x + x^2$ , $p_3(x) = 1 + x^2$ linearly independent in $P_2$ ?

Show the worked solution — every step

Strategy. Equate a linear combination to the zero polynomial; the coefficient of every power of

x

must vanish.

Step 1 · Expand the combination.

a(1 + x) + b(x + x^2) + c(1 + x^2) = (a + c) + (a + b)x + (b + c)x^2 = 0.

Step 2 · Coefficient equations.

\begin{aligned} a + c &= 0 &&\text{(i)}\\ a + b &= 0 &&\text{(ii)}\\ b + c &= 0 &&\text{(iii)} \end{aligned}

Step 3 · Solve. From (i),

c = -a

. From (ii),

b = -a

. Substitute into (iii):

-a + (-a) = -2a = 0 \Rightarrow a = 0

. Then

b = c = 0

Answer. Only the trivial solution exists, so

p_1, p_2, p_3

are linearly independent.

Wait — really?

Write down the coefficient vectors: $p_1 \to (1,1,0)$ , $p_2 \to (0,1,1)$ , $p_3 \to (1,0,1)$ . Those are exactly the three vectors of Problem 10 — the polynomial question is secretly the same $\mathbb{R}^3$ computation you already did.

Key takeaway

In simple words: none of these polynomials can be built from the other two by adding or scaling alone. Matching coefficients turns any $P_2$ question into an $\mathbb{R}^3$ question.

Problem 12Hard

Determine whether $\{(1, 2, 3),\ (4, 5, 6),\ (7, 8, 9)\}$ is linearly independent in $\mathbb{R}^3$ .

Show the worked solution — every step

Strategy. Reduce the system; if a row vanishes, you get a free parameter and hence dependence.

Step 1 · Write the homogeneous system. From

a(1,2,3) + b(4,5,6) + c(7,8,9) = \mathbf{0}

\begin{aligned} a + 4b + 7c &= 0\\ 2a + 5b + 8c &= 0\\ 3a + 6b + 9c &= 0 \end{aligned}

Step 2 · $R_2 \to R_2 - 2R_1$ . Entry by entry:

2 - 2 = 0

5 - 8 = -3

8 - 14 = -6

. So

-3b - 6c = 0 \Rightarrow b + 2c = 0

Step 3 · $R_3 \to R_3 - 3R_1$ . Entry by entry:

3 - 3 = 0

6 - 12 = -6

9 - 21 = -12

. So

-6b - 12c = 0 \Rightarrow b + 2c = 0

— the same equation again.

Step 4 · Free parameter. Only two independent equations remain. Let

c = t

(free); then

b = -2t

and

a = -4b - 7c = -4(-2t) - 7t = 8t - 7t = t

. Pick

t = 1

(a, b, c) = (1, -2, 1)

Step 5 · Verify the relation.

\begin{aligned}1 \cdot (1,2,3) - 2 \cdot (4,5,6) + 1 \cdot (7,8,9) &= (1,2,3) - (8,10,12) + (7,8,9)\\ &= (0,0,0). \end{aligned}

Entry by entry:

1 - 8 + 7 = 0

2 - 10 + 8 = 0

3 - 12 + 9 = 0

. ✓

Alternate route · Row reduce to REF. Form the matrix with the vectors as rows:

\begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6\\ 7 & 8 & 9 \end{bmatrix}

R_2 \to R_2 - 4R_1

(entries:

4-4=0

5-8=-3

6-12=-6

) and

R_3 \to R_3 - 7R_1

(entries:

7-7=0

8-14=-6

9-21=-12

\begin{bmatrix} 1 & 2 & 3\\ 0 & -3 & -6\\ 0 & -6 & -12 \end{bmatrix}

R_3 \to R_3 - 2R_2

(entries:

0

-6-(-6)=0

-12-(-12)=0

\begin{bmatrix} 1 & 2 & 3\\ 0 & -3 & -6\\ 0 & 0 & 0 \end{bmatrix}

This is REF with only 2 non-zero rows, so rank

= 2 < 3

(the number of vectors). Dependent.

Step 6 · Bonus — extract the relation from REF. Track the operations on

R_3

: we did

R_3 \to R_3 - 7R_1

, then

R_3 \to R_3 - 2R_2'

where

R_2' = R_2 - 4R_1

. Combining: final

R_3 = (v_3 - 7v_1) - 2(v_2 - 4v_1) = v_3 + v_1 - 2v_2

. Setting this

= \mathbf{0}

gives

v_1 - 2v_2 + v_3 = \mathbf{0}

— the same dependency relation found by direct solving.

Step 7 · Quick determinant check.

\det = 0

, instantly confirming dependence.

Answer. Linearly dependent, with dependency relation

v_1 - 2v_2 + v_3 = \mathbf{0}

Watch out

Two vectors are dependent $\Leftrightarrow$ one is a scalar multiple of the other. But for three or more vectors, dependence does NOT mean two of them are parallel — here no vector is a multiple of another, yet the set is dependent. Always set up the equation and solve.

Wait — really?

Rearrange the relation: $v_2 = \tfrac{1}{2}(v_1 + v_3)$ . The middle vector is exactly the average of the outer two — consecutive-integer patterns hide a built-in redundancy.

Problem 13Easy

Find the dimension of $\mathbb{R}^3$ and give a basis.

Show the worked solution — every step

Strategy. Use the standard basis

e_1, e_2, e_3

Step 1 · Candidates. Let

e_1 = (1, 0, 0)

e_2 = (0, 1, 0)

e_3 = (0, 0, 1)

Step 2 · Spanning. Any

(x, y, z) = x \cdot e_1 + y \cdot e_2 + z \cdot e_3

Step 3 · Independence.

a \cdot e_1 + b \cdot e_2 + c \cdot e_3 = (a, b, c) = \mathbf{0}

forces

a = b = c = 0

Answer.

\{e_1, e_2, e_3\}

is a basis and

\dim(\mathbb{R}^3) = 3

Key takeaway

Dimension $=$ degrees of freedom — how many independent knobs you can tune. $\mathbb{R}^3$ has three: $x$ , $y$ , and $z$ .

Problem 14Easy

Show that $B = \{(1, 0),\ (0, 1)\}$ is a basis of $\mathbb{R}^2$ .

Show the worked solution — every step

Step 1 · Spanning.

(x, y) = x(1, 0) + y(0, 1)

Step 2 · Independence.

a(1, 0) + b(0, 1) = (a, b) = \mathbf{0} \Rightarrow a = b = 0

Answer. Both conditions hold, so

B

is a basis and

\dim(\mathbb{R}^2) = 2

Key takeaway

Basis $=$ independent AND spans. Once both hold, every vector has exactly one expression in terms of the basis — its coordinates.

Problem 15Medium

Show that $B = \{(1, 1, 0),\ (1, 0, 1),\ (0, 1, 1)\}$ is a basis of $\mathbb{R}^3$ .

Show the worked solution — every step

Strategy.

\dim(\mathbb{R}^3) = 3

and

B

has 3 vectors — it suffices to prove independence.

Step 1 · Set up the homogeneous system.

a(1,1,0) + b(1,0,1) + c(0,1,1) = \mathbf{0}

gives:

\begin{aligned} a + b &= 0\\ a + c &= 0\\ b + c &= 0 \end{aligned}

Step 2 · Solve.

b = -a

c = -a

. Substitute into

b + c = 0

-2a = 0 \Rightarrow a = 0

. Then

b = c = 0

Step 3 · Conclude. Independent + size 3 in

\mathbb{R}^3

\Rightarrow

B

is a basis of

\mathbb{R}^3

Alternate route · Row reduce to REF. Form the matrix with the three vectors as rows:

\begin{bmatrix} 1 & 1 & 0\\ 1 & 0 & 1\\ 0 & 1 & 1 \end{bmatrix}

R_2 \to R_2 - R_1

(entries:

1-1=0

0-1=-1

1-0=1

\begin{bmatrix} 1 & 1 & 0\\ 0 & -1 & 1\\ 0 & 1 & 1 \end{bmatrix}

R_3 \to R_3 + R_2

(entries:

0

1+(-1)=0

1+1=2

\begin{bmatrix} 1 & 1 & 0\\ 0 & -1 & 1\\ 0 & 0 & 2 \end{bmatrix}

Three non-zero rows

\Rightarrow

rank

= 3

. Combined with 3 vectors in a 3-dimensional space,

B

is a basis. ✓

Equivalent shortcut. The determinant of the matrix with these rows is

-2 \ne 0

Answer.

B

is a basis of

\mathbb{R}^3

Key takeaway

Counting trick: in an $n$ -dimensional space, any $n$ linearly independent vectors form a basis automatically — no need to re-prove spanning. Likewise, any spanning set of size $n$ is automatically independent.

Problem 16Medium

Find a basis and the dimension of $W = \{(x, y, z) \in \mathbb{R}^3 : x + y + z = 0\}$ .

Show the worked solution — every step

Strategy. Express

x

in terms of the free variables

y

and

z

Step 1 · Solve the constraint. From

x + y + z = 0

x = -y - z

. So a generic element is

(x, y, z) = (-y - z,\ y,\ z) = y \cdot (-1, 1, 0) + z \cdot (-1, 0, 1).

Step 2 · Spanning. Let

v_1 = (-1, 1, 0)

v_2 = (-1, 0, 1)

. Every element of

W

is a combination of them, so they span

W

Step 3 · Independence.

a \cdot v_1 + b \cdot v_2 = (-a - b,\ a,\ b) = (0, 0, 0) \Rightarrow a = 0,\ b = 0

Answer.

\{v_1, v_2\} = \{(-1, 1, 0),\ (-1, 0, 1)\}

is a basis and

\dim(W) = 2

. (

W

is a plane through the origin — the same plane whose subspace-hood was proved in Problem 3.)

Key takeaway

Number of free variables $=$ dimension. A subspace of $\mathbb{R}^n$ cut out by $k$ independent linear equations has dimension $n - k$ — here $3 - 1 = 2$ .

Problem 17Hard

Find a basis and the dimension of $W = \{(x, y, z, w) \in \mathbb{R}^4 : x + y = 0 \ \text{and}\ z - w = 0\}$ .

Show the worked solution — every step

Strategy. Two independent constraints in

\mathbb{R}^4

should leave

4 - 2 = 2

free directions.

Step 1 · Solve the constraints. From

x + y = 0

x = -y

. From

z - w = 0

z = w

. Pick

y

and

w

as parameters.

Step 2 · Generic element.

(x, y, z, w) = (-y,\ y,\ w,\ w) = y \cdot (-1, 1, 0, 0) + w \cdot (0, 0, 1, 1).

Let

v_1 = (-1, 1, 0, 0)

v_2 = (0, 0, 1, 1)

— they span

W

Step 3 · Independence.

a \cdot v_1 + b \cdot v_2 = (-a,\ a,\ b,\ b) = \mathbf{0}

forces

a = b = 0

Answer.

\{(-1, 1, 0, 0),\ (0, 0, 1, 1)\}

is a basis and

\dim(W) = 2

Key takeaway

Predict before you compute: $n$ minus the number of independent constraints gives the dimension ( $4 - 2 = 2$ here). If the algebra disagrees with the prediction, recheck both.

Problem 18Hard

Find the dimension and a basis of the vector space $S$ of all $2 \times 2$ symmetric matrices.

Show the worked solution — every step

Strategy. Count free entries. A symmetric

2 \times 2

matrix has the two diagonal entries and one off-diagonal — three free numbers.

Step 1 · General form.

A = \begin{bmatrix} a & b\\ b & c \end{bmatrix}, \qquad a, b, c \in \mathbb{R}.

Step 2 · Decompose.

A = a \cdot E_1 + b \cdot E_2 + c \cdot E_3

, where

E_1 = \begin{bmatrix} 1 & 0\\ 0 & 0 \end{bmatrix}, \quad E_2 = \begin{bmatrix} 0 & 1\\ 1 & 0 \end{bmatrix}, \quad E_3 = \begin{bmatrix} 0 & 0\\ 0 & 1 \end{bmatrix}.

Step 3 · Spanning. Every symmetric

A

is built from

\{E_1, E_2, E_3\}

via the decomposition above.

Step 4 · Independence.

\alpha E_1 + \beta E_2 + \gamma E_3 = \begin{bmatrix} \alpha & \beta\\ \beta & \gamma \end{bmatrix} = \mathbf{0}

forces

\alpha = \beta = \gamma = 0

Answer.

\{E_1, E_2, E_3\}

is a basis and

\dim(S) = 3

Wait — really?

All $2 \times 2$ matrices form a 4-dimensional space, yet symmetric ones form only a 3-dimensional one: the single mirror constraint $a_{12} = a_{21}$ kills exactly one degree of freedom. Problem 5 proved $S$ is a subspace; this problem measures how big it is.

Key takeaway

To find the dimension of a matrix space, count the entries you can choose freely — each free entry contributes one basis matrix.

Next up

Unit 3 · Analytic Geometry →

Vector spaces gave you a board to play on, but nothing on it has a size or an angle yet. Unit 3 installs the ruler and the protractor — norms, inner products, orthogonality — and shows that choosing how to measure similarity is half of machine learning.

← All units