Linear Algebra

Linear Algebra Recall

Vectors, Norms, Matrix Views, and Special Matrices. A deep dive into the building blocks.

1 Vector Operations

Basic Arithmetic

Operations are element-by-element.

Addition:
$c = a + b \iff c_i = a_i + b_i$
Scaling:
$b = \sigma a \iff b_i = \sigma a_i$
Transpose:
$u = [1, 2, 3] \implies u^T = \begin{bmatrix}1\\2\\3\end{bmatrix}$
Linear Combination:
$z = \alpha u + \beta v$

Inner Product ($u \cdot v$)

Scalar result. Geometric interpretation:

$u^T v = \sum u_i v_i = \|u\|_2 \|v\|_2 \cos\theta$

Commutative: $u^T v = v^T u$.

Outer Product ($u v^T$)

Matrix result (Rank-1). Element $A_{ij} = u_i v_j$.

$\begin{bmatrix}u_1\\u_2\end{bmatrix} \begin{bmatrix}v_1 & v_2\end{bmatrix} = \begin{bmatrix}u_1v_1 & u_1v_2 \\ u_2v_1 & u_2v_2\end{bmatrix}$

Matlab Implementation

>> u = [0; 1; 2; 3]; v = [3; 2; 1; 0];
>> dot_prod = u' * v % Inner Product (Scalar)
ans = 4
>> out_prod = u * v' % Outer Product (Matrix)
ans =
0 0 0 0
3 2 1 0
6 4 2 0
9 6 3 0

2 Vector Norms

Norms allow us to compare the "size" or "magnitude" of vectors, similar to absolute value for scalars.

$L_1$ Norm

Manhattan Distance

$\|x\|_1 = \sum |x_i|$

Sum of absolute values.

$L_2$ Norm

Euclidean Distance

$\|x\|_2 = \sqrt{\sum x_i^2}$

Standard geometric length. $\sqrt{x^T x}$.

$L_\infty$ Norm

Max Norm

$\|x\|_\infty = \max |x_i|$

Magnitude of the largest element.

Comparing Vectors

To check if $y \approx z$, we use the relative error with norms:

$\frac{\|y - z\|}{\|z\|} < \delta$

3 Matrix Views

Understanding matrix-vector multiplication $Ax=b$ in different ways is crucial for intuition.

Column View

"Linear Combination of Columns"

$b = x_1 \mathbf{a_1} + x_2 \mathbf{a_2} + \dots + x_n \mathbf{a_n}$

The vector $b$ is constructed by scaling and adding the columns of $A$. This reveals the Range.

Row View

"Dot Product with Rows"

$b_i = \mathbf{row_i}(A) \cdot x$

Each element $b_i$ is the projection of $x$ onto the $i$-th row of $A$. Good for hand calculations.

Also: The Vector View sees $A$ as a function that stretches and rotates the vector $x$.

4 Independence & Rank

Linear Independence

A set of vectors $\{v_1, \dots, v_n\}$ is linearly independent if the only solution to:

$\alpha_1 v_1 + \dots + \alpha_n v_n = 0$

is the trivial solution $\alpha_1 = \dots = \alpha_n = 0$.

Matrix Rank

The rank of matrix $A$ is the number of linearly independent columns (or rows). It measures the "dimension" of the output space.

$\rank{A} \le \min(m, n)$.
Full Rank: Maximum possible rank.
Rank Deficient: Columns are dependent.

Numerical Rank is Tricky!

>> A = [1 0 0; 0 1 0; 0 0 1e-15];
>> rank(A) % Might be 3
>> A(3,3) = 0;
>> rank(A) % Is 2

Tiny computer errors can make a singular matrix appear non-singular.

5 Special Matrices

Identity ($I$)

[1 0 0; 0 1 0; 0 0 1]

Like multiplying by 1. $AI = A$. Created with `eye(n)`.

Diagonal

[d1 0 0; 0 d2 0; 0 0 d3]

Non-zeros only on main diagonal. Created with `diag(v)`.

Symmetric

$A = A^T$

Mirror image across diagonal. $A^TA$ is always symmetric.

Orthogonal ($Q$)

$Q^T Q = I$

Columns are orthonormal. Preserves norms: $\|Qx\|_2 = \|x\|_2$.

Tridiagonal

Non-zeros on main, super, sub diagonals.

Common in 1D Finite Difference problems. Sparse storage.

🧠 Knowledge Check +20 XP

Which of the following statements about Matrix-Matrix multiplication $C=AB$ is FALSE?