Vector Spaces

In Unit 1 you played inside the game: rows, pivots, solutions. Now we ask a question one level up: what kind of world makes those moves legal in the first place? The answer is a vector space — and once you know its rules, "add a gradient step," "average two models," "project onto principal components" all become obviously-allowed moves on the same board.

Nearly every object in ML is a vector in some space: a cat photo is a vector in $\mathbb{R}^{224\times224\times3}$ (150,528 numbers), a word inside a language model is a vector in $\mathbb{R}^{768}$, your Netflix taste is a vector in some $\mathbb{R}^d$, a layer's weights are a vector in $\mathbb{R}^{m\times n}$. Training a model is moving through these spaces in clever ways — and you can't reason about a move without knowing the rules of the board.

Before we can say what a vector space is, we need the minimum structure under which "combine two things, get a sensible third thing" works. That's a group: a set $G$ with an operation $\otimes$ such that four promises hold. Closure: $x\otimes y$ stays in $G$ — you don't fall out of the set. Associativity: $(x\otimes y)\otimes z = x\otimes(y\otimes z)$ — grouping doesn't matter. Identity: some do-nothing element $e$ exists with $x\otimes e = x$. Inverse: everything can be undone — each $x$ has a partner $y$ with $x\otimes y = e$. If order also never matters ($x\otimes y = y\otimes x$), the group is Abelian.

Closure and associativity rarely break. The killers are identity and, above all, inverse — sets fail the moment they're missing the negatives, the fractions, or whatever "undo partner" their elements need. Audit the classics yourself:

Take an Abelian group of vectors under addition, and bolt on one more power: scaling by real numbers. A real vector space $V=(\mathcal{V},+,\cdot)$ demands: (V1) $(\mathcal{V},+)$ is an Abelian group — the zero vector $\mathbf{0}$ is its do-nothing element, $-\mathbf{v}$ its undo. (V2) Distributivity, both ways: $\lambda(\mathbf{x}+\mathbf{y})=\lambda\mathbf{x}+\lambda\mathbf{y}$ and $(\lambda+\psi)\mathbf{x}=\lambda\mathbf{x}+\psi\mathbf{x}$. (V3) Scaling stacks: $\lambda(\psi\mathbf{x})=(\lambda\psi)\mathbf{x}$. (V4) $1\cdot\mathbf{x}=\mathbf{x}$. Translation: a vector space is a place where the two basic moves of linear algebra — add and scale — always make sense and never throw you off the board.

Two examples to hold on to. Lists: $\mathbb{R}^n$ with entrywise addition and scaling — three exam scores $(85,92,78)$ plus a bonus $(5,-2,3)$ is $(90,90,81)$, still three scores. Spreadsheets: $\mathbb{R}^{m\times n}$ — whole matrices treated as single vectors, added cell by cell, scaled cell by cell. A weekly-sales sheet plus a corrections sheet is a sheet of the same shape. Same axioms, bigger object.

Formally: $U\subseteq\mathcal{V}$, $U\neq\emptyset$, is a subspace of $V$ if $U$ is itself a vector space under the same operations. The good news: associativity, distributivity and friends are inherited free from $V$ — they hold for all of $\mathcal{V}$, so certainly on $U$. Only three things can go wrong, and they're all about staying inside:

Now stress-test four candidate sets with your own hands — the buttons apply real operations to a sample point and check whether the result stays inside:

Unit 1 ended with a promise: the blind spot would get its full papers here. Time to deliver. Recall $N(A)=\{\mathbf{x}\in\mathbb{R}^n : A\mathbf{x}=\mathbf{0}\}$ — the directions the machine can't see. Run the three-point test on it:

(S1) $A\mathbf{0}=\mathbf{0}$, so $\mathbf{0}\in N(A)$ — the machine maps silence to silence. (S2) if $A\mathbf{x}=\mathbf{0}$ and $A\mathbf{y}=\mathbf{0}$, then $A(\mathbf{x}+\mathbf{y})=\mathbf{0}+\mathbf{0}=\mathbf{0}$. (S3) $A(\lambda\mathbf{x})=\lambda A\mathbf{x} = \lambda\mathbf{0}=\mathbf{0}$. Three for three — for every matrix $A$, no exceptions. The blind spot isn't just a set of unlucky inputs; it is a full flat-through-the-origin world of its own. (Its sibling, the column space — all outputs $A\mathbf{x}$ — is a subspace too, by an equally quick argument. Both officially citizens now.)

And this is why the geometry of Unit 1 looked the way it did: the solution set of $A\mathbf{x}=\mathbf{b}$ was "a flat thing not through the origin" — precisely because it's a subspace (the null space) shifted by $\mathbf{x}_p$. Subspace + shift = affine flat. The vocabulary was waiting for us one unit ahead.

Formally, $\operatorname{span}(\mathbf{x}_1,\dots,\mathbf{x}_k)=\{\lambda_1\mathbf{x}_1+\cdots+\lambda_k\mathbf{x}_k : \lambda_i\in\mathbb{R}\}$ — and it is always a subspace (it contains $\mathbf{0}$ via all-zero weights, and sums/scalings of itineraries are itineraries). Fly through what spans look like in 3D:

The span machine kept whispering one idea: some vectors add nothing. Here's the formal detector. Ask: can the vectors cancel each other out — combine to $\mathbf{0}$ — without every weight being zero?

Yes → linearly dependent. Rearrange that equation and one vector is exposed as a blend of the others — a redundant tram, the "agreeing friend" from Unit 1's group chat. No → linearly independent: the only way to $\mathbf{0}$ is the boring all-zeros way, and every vector contributes something the rest collectively can't replicate. There is no third option.

Two instant shortcuts. ① If $\mathbf{0}$ is one of your vectors, the set is dependent on the spot — put weight 1 on $\mathbf{0}$, zero elsewhere: a non-trivial combination summing to $\mathbf{0}$. ② For non-zero vectors, "dependent" is exactly "at least one is a combination of the others" — redundancy made literal.

You never have to guess. The foolproof procedure is one you already own: stack the vectors as columns, run Gaussian elimination, read the pivots. Every column with a pivot is a genuine contributor; every pivot-less column is a blend of the pivot columns to its left — and (this is the beautiful part) in reduced row-echelon form, the entries of a non-pivot column literally spell out its recipe.

Try the presets against the slides: the $2\times3$ example exposes column 2 as $2\times$ column 1; the three vectors in $\mathbb{R}^4$ come out all-pivots — independent, no redundancy anywhere. Keep the detector in reach; the rest of the unit leans on it twice more.

Pack the ingredients as columns of $B=[\mathbf{b}_1\ \cdots\ \mathbf{b}_k]$ and each recipe as a column $\boldsymbol{\lambda}_j$; then cooking is a matrix product, $\mathbf{x}_j = B\boldsymbol{\lambda}_j$. The theorem:

Watch the mirror operate — the two panels are different spaces, yet they always agree:

The slides run this at full scale: four dishes from four ingredients with recipe matrix columns $(1,-2,1,-1)$, $(-4,-2,0,4)$, $(2,3,-1,-3)$, $(17,-10,11,1)$. Elimination gives RREF pivots in columns 1–3 and a pivot-less column 4 reading $(-7,-15,-18)$: so $\boldsymbol{\lambda}_4=-7\boldsymbol{\lambda}_1-15\boldsymbol{\lambda}_2-18\boldsymbol{\lambda}_3$, and by the mirror, $7\mathbf{x}_1+15\mathbf{x}_2+18\mathbf{x}_3+\mathbf{x}_4=\mathbf{0}$ — the four dishes are dependent, redundancy named exactly. (Feed preset 3 of the detector above and read it off yourself.)

Four portraits of the same object, all provably equivalent: a basis is ① a generating set that's independent (the textbook line); ② a minimal generating set — remove any vector and some part of the space becomes unreachable; ③ a maximal independent set — add any vector and redundancy appears; ④ a unique address book — every $\mathbf{x}\in V$ has exactly one recipe $\mathbf{x}=\sum_i\lambda_i\mathbf{b}_i$. The fourth is the money view. Why unique? Two recipes for the same $\mathbf{x}$ would subtract to a non-trivial combination $\sum_i(\lambda_i-\psi_i)\mathbf{b}_i=\mathbf{0}$ — forbidden by independence. No redundancy in the toolkit ⟺ no ambiguity in the addresses.

And a basis is thoroughly non-unique — $\mathbb{R}^3$ is happy with the canonical $\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3$, or the staircase $(1,0,0),(1,1,0),(1,1,1)$, or any three independent vectors whatsoever. Same space, different address books:

Here is a small miracle you can lean on forever: pick any basis of a space and count its vectors; pick a completely different basis and count again — you always get the same number. (If a 3-vector basis and a 4-vector basis coexisted, the recipes theorem's pigeonhole would convict the larger one of redundancy — contradiction.) That invariant count is the dimension, $\dim(V)$: the number of independent directions the space truly has. So $\dim(\mathbb{R}^n)=n$, and $\dim(\mathbb{R}^{m\times n})=mn$ — a basis for matrix-space is the family of one-hot matrices $E_{ij}$, one per cell. For subspaces: $\dim(U)\le\dim(V)$, with equality only when $U$ is $V$.

Handed a pile of vectors that span some subspace $U=\operatorname{span}(\mathbf{x}_1,\dots,\mathbf{x}_m)\subseteq\mathbb{R}^n$, how do you shrink the pile to a basis? Three steps you can now guess: ① stack them as columns of $A$; ② row-reduce to echelon form; ③ keep the original vectors sitting over the pivot columns. Those survivors are independent (pivots certify it), and they span everything the pile spanned (each discarded vector is a recipe of survivors — the RREF column says exactly which recipe). Basis found; count it and you've measured $\dim(U)$ too.

The slide example, finished honestly. Four vectors spanning $U\subseteq\mathbb{R}^5$:

Stack, reduce (or press preset 4 in the detector of Section 8 and watch it happen):

Pivots in columns 1, 2 and 4. Column 3 is the only freeloader, and its RREF entries $(-1,2)$ name the recipe: $\mathbf{x}_3=-\mathbf{x}_1+2\mathbf{x}_2$ (check it — it works on all five components). So:

One more from the slides, in the other direction: the three vectors $(1,2,3,4),(2,-1,0,2),(1,1,0,4)$ in $\mathbb{R}^4$ reduce to all-pivots — independent — yet they are not a basis of $\mathbb{R}^4$: three trams can't cover a four-dimensional city. Independent ≠ basis until the count matches the dimension.

① A group is combining-with-guarantees; add scaling and you get a vector space — the board every ML object lives on. ② A subspace is a flat world through the origin inside a bigger one — null space and column space are the star citizens. ③ The span is everywhere your vectors can reach; it never grows when you add a redundant vector. ④ Independence is non-redundancy, and elimination's pivots detect it mechanically. ⑤ A basis is span + independence: the smallest toolkit reaching everything, giving every vector exactly one address. ⑥ Dimension — the size of any basis — is the board's true size, and it polices everything: more vectors than dimensions ⇒ redundancy, guaranteed.