Ligang Yan颜力刚

· NSCC

SVD by Hand: Splitting a 2×3 Matrix into UΣVᵀ

Singular value decomposition breaks any matrix into a rotation, a stretch and another rotation. Working one 2×3 matrix all the way through: eigenvalues of AAᵀ, square roots for the singular values, U and V column by column, and a check that multiplies it all back. Along the way, why the square root and why signs can go wrong.

linear-algebrasvdmachine-learningnumpy

中文版：手算 SVD：把一个 2×3 矩阵拆成 UΣVᵀ

Singular value decomposition (SVD) says that any matrix, square or not, can be written as a product of three matrices.

A  =  U   ·   Σ   ·   Vᵀ
m×n   m×m     m×n     n×n
Part Name Property
U left singular vectors orthogonal: every column has length 1, and the columns are mutually perpendicular
Σ singular values diagonal, with σ₁ ≥ σ₂ ≥ … ≥ 0 on the diagonal and zeros everywhere else
Vᵀ right singular vectors V is orthogonal too; Vᵀ is its transpose, which is also its inverse

Geometrically, applying A to a vector happens in three moves: Vᵀ rotates, Σ stretches along the axes, U rotates again. Every linear transformation fits this rotate–stretch–rotate pattern.

Below, one concrete matrix worked all the way through by hand.

The problem

A = [3   2   2]
    [2   3  −2]

A is 2×3, so the pieces have these shapes:

  A    =   U    ·    Σ    ·   Vᵀ
 2×3      2×2       2×3      3×3

There are min(2, 3) = 2 singular values. Σ is 2×3, so after the two singular values there is a whole column of zeros left over. V, though, needs 3 columns, and the third one has to be found some other way.

The plan: start with the small one

The standard hand method rests on these facts (derived in step 3):

  • the columns of V are eigenvectors of AᵀA
  • the columns of U are eigenvectors of AAᵀ
  • both have eigenvalues σ²

Here AAᵀ is 2×2 and AᵀA is 3×3. Eigenvalues of a 2×2 matrix only need one quadratic equation, so start with AAᵀ:

  1. Find the eigenvalues of AAᵀ, take square roots to get Σ; its eigenvectors form U
  2. Use vᵢ = Aᵀuᵢ / σᵢ to get the first two columns of V
  3. Fill in the third column of V with a cross product

Step 1: compute AAᵀ

Each entry of AAᵀ is a dot product between two rows of A. Call them r₁ = (3, 2, 2) and r₂ = (2, 3, −2):

r₁ · r₁ = 9 + 4 + 4  = 17
r₁ · r₂ = 6 + 6 − 4  = 8
r₂ · r₂ = 4 + 9 + 4  = 17

AAᵀ = [17   8]
      [ 8  17]

Step 2: eigenvalues

det(AAᵀ − λI) = (17 − λ)² − 8² = 0
17 − λ = ±8

λ₁ = 25,  λ₂ = 9          # largest first

Step 3: take square roots, write Σ

σ₁ = √25 = 5,   σ₂ = √9 = 3

Σ = [5  0  0]              # 2×3, same shape as A
    [0  3  0]

Why the square root? Substitute A = UΣVᵀ into AAᵀ:

AAᵀ = (UΣVᵀ)(UΣVᵀ)ᵀ
    = UΣ(VᵀV)ΣᵀUᵀ        # V is orthogonal, so VᵀV = I
    = U · ΣΣᵀ · Uᵀ

ΣΣᵀ squares every number on the diagonal. A appears twice in AAᵀ, so the stretch is applied twice, and the eigenvalues are σ² rather than σ. The square root gives back how much A stretches in a single pass.

You can check it directly: the unit vector v₁ = (1, 1, 0)/√2 comes out of A with length 5, and out of Aᵀ after that scaled by 25.

Why not take eigenvalues of A directly? A isn’t square, so it has none. Even a square matrix can have negative or complex eigenvalues. AAᵀ is always symmetric with eigenvalues ≥ 0, so the method detours through it and takes square roots at the end.

Step 4: eigenvectors of AAᵀ → U

λ = 25: (17 − 25)x₁ + 8x₂ = 0  →  x₁ = x₂   →  (1, 1)
λ = 9:  (17 − 9)x₁  + 8x₂ = 0  →  x₁ = −x₂  →  (1, −1)

Both vectors have length √2. Normalize them and place them as columns in the same order as σ, largest first:

U = (1/√2) [1   1]
           [1  −1]
            ↑    ↑
          σ₁=5  σ₂=3

Check: the dot product of the two columns is 1 − 1 = 0, so they are perpendicular.

Step 5: the first two columns of V from vᵢ = Aᵀuᵢ / σᵢ

Aᵀ = [3   2]
     [2   3]
     [2  −2]

Aᵀ · (1, 1)  = (5, 5, 0)
v₁ = (5, 5, 0) ÷ (√2 · 5)  = (1, 1, 0) / √2

Aᵀ · (1, −1) = (1, −1, 4)
v₂ = (1, −1, 4) ÷ (√2 · 3) = (1, −1, 4) / √18

Check: |v₂|² = (1 + 1 + 16) / 18 = 1, and v₁ · v₂ = 1 − 1 + 0 = 0.

Why not just take eigenvectors of AᵀA? An eigenvector’s sign is arbitrary: (1, 1) is an eigenvector and so is (−1, −1). If U and V are found independently, their signs may not match, and the product won’t come back to A. Deriving vᵢ from uᵢ guarantees they pair up, and it skips a 3×3 eigenvalue problem.

Step 6: v₃ from a cross product

v₃ has to be perpendicular to v₁ and v₂ and satisfy Av₃ = 0, which means it is perpendicular to both rows of A. In three dimensions, the direction perpendicular to two vectors is their cross product:

(3, 2, 2) × (2, 3, −2)
  = (2·(−2) − 2·3,  2·2 − 3·(−2),  3·3 − 2·2)
  = (−10, 10, 5)
  → simplify to (−2, 2, 1)

v₃ = (−2, 2, 1) / 3

Check: Av₃ = (−6 + 4 + 2, −4 + 6 − 2) = (0, 0); v₃ · v₁ = 0; v₃ · v₂ = 0.

Step 7: transpose V into Vᵀ

The columns of V become the rows of Vᵀ:

      [ 1/√2     1/√2      0    ]   ← v₁
Vᵀ =  [ 1/√18   −1/√18   4/√18  ]   ← v₂
      [−2/3      2/3      1/3   ]   ← v₃

The result

[3  2   2]  =  (1/√2) [1   1]  ·  [5  0  0]  ·  Vᵀ
[2  3  −2]            [1  −1]     [0  3  0]

Checking it: add up the layers

UΣVᵀ can also be written as a sum of layers:

A = σ₁·u₁v₁ᵀ + σ₂·u₂v₂ᵀ

Each layer is “one column times one row”, so each is 2×3.

Layer 1: σ₁ · (1/√2) · (1/√2) = 5 · ½

  5 · ½ · [1  1  0]  =  [2.5  2.5  0]
          [1  1  0]     [2.5  2.5  0]

Layer 2: σ₂ · (1/√2) · (1/√18) = 3 · ⅙ = ½

  ½ · [ 1  −1   4]  =  [ 0.5  −0.5   2]
      [−1   1  −4]     [−0.5   0.5  −2]

Sum:                    [3  2   2]   = A
                        [2  3  −2]

Where did v₃ go? It lines up with the all-zero third column of Σ, and anything times zero disappears. That’s why in practice people usually use the “thin SVD”, which never computes it.

Each layer’s weight is σ²: 25 for layer 1, 9 for layer 2. Keeping only layer 1 keeps 25/34 ≈ 74% of the information.

Where it goes wrong

  • Column order follows Σ. The first column of both U and V must go with the largest σ. Get the order wrong and the product isn’t A.
  • Σ is 2×3, not 2×2. It has the same shape as A, with zeros in every extra position.
  • The signs of uᵢ and vᵢ must match. You can flip both together, never just one.
  • v₃ is only needed for the full SVD. The thin SVD leaves it out.

Checking with NumPy

import numpy as np

A = np.array([[3, 2, 2], [2, 3, -2]])
U, s, Vt = np.linalg.svd(A)

s    # [5. 3.]
U    # [[-0.7071 -0.7071]
     #  [-0.7071  0.7071]]
Vt   # [[-0.7071 -0.7071  0.    ]
     #  [-0.2357  0.2357 -0.9428]
     #  [-0.6667  0.6667  0.3333]]

Same singular values, 5 and 3. NumPy flipped the sign of both (u₁, v₁) and (u₂, v₂); the two minus signs cancel, so the product is still A. Matching up the numbers: 0.7071 = 1/√2, 0.2357 = 1/√18, 0.9428 = 4/√18.

Two small traps:

  • s is a 1-D array, not a matrix. Rebuilding A takes U @ np.diag(s) @ Vt, and for a non-square matrix that only lines up with full_matrices=False (or after padding np.diag(s) to the right shape).
  • The third return value is already Vᵀ. Don’t transpose it again.

What it’s for

A = UΣVᵀ isn’t the goal in itself. SVD is useful because it sorts the matrix by importance: σ runs from largest to smallest, the first few layers carry most of the information, and the rest can be dropped.

Try it on a 600×512 grayscale photo, which is just a matrix of 307,200 numbers. Keeping the first k layers takes k × (600 + 512 + 1) numbers:

Layers kept, k Numbers stored Share of original Information kept
1 1,113 0.4% 75%
20 22,260 7.2% 96.6%
50 55,650 18% 98.9%
100 111,300 36% 99.6%

At k = 50 the photo already looks essentially sharp. PCA, recommender systems and denoising all run on the same idea: keep the large σ, drop the small ones.