Linear Algebra · College

Linear Algebra: Least Squares and Projections

Orthogonal projections and the least squares method for approximating solutions from a college linear algebra course.

30 cards · basic cards · AI-written, checked twice. Edit anything.

Define the orthogonal projection of vector b onto vector a.

proj_a(b) = (a . b / a . a) * a

What is an orthogonal projection matrix P?

A matrix such that P*v gives the projection of v onto a subspace, with P = P^2 (idempotent) and P^T = P (symmetric).

Define the orthogonal complement of a subspace W.

The set of all vectors in R^n orthogonal to every vector in W, denoted W^perp.

State the Orthogonal Decomposition Theorem.

Any vector y can be uniquely written as y = y_hat + z, where y_hat is in W and z is in W^perp.

What is the least squares problem?

Find a vector x that minimizes ||b - Ax||^2, where Ax is typically in the column space of A.

When does the least squares method apply?

When the system Ax = b is overdetermined (more equations than unknowns) or inconsistent, so no exact solution exists.

State the Normal Equations for least squares.

A^T * A * x = A^T * b

What defines an orthonormal set of vectors?

A set of vectors where each vector has unit length and every pair is orthogonal to each other.

Why is an orthonormal basis computationally advantageous?

The projection onto an orthonormal basis is easy to compute: proj_v(b) = (b . v) * v for each basis vector v, without dividing by squared norms.

What is the Gram-Schmidt process?

An algorithm that transforms a set of linearly independent vectors into an orthogonal (or orthonormal) set spanning the same subspace.

What is QR factorization?

A factorization A = Q * R where Q has orthonormal columns and R is upper triangular, often computed via Gram-Schmidt.

How does QR factorization solve least squares?

From A = Q * R and normal equations A^T * A * x = A^T * b, we get R * x = Q^T * b, which is easy to solve since R is triangular.

What is the residual in a least squares problem?

The vector r = b - Ax, which is the error or leftover when Ax is the approximation to b.

State the Best Approximation Theorem for least squares.

The least squares solution x* gives A*x* as the closest point in the column space of A to b, minimizing ||b - A*x*||.

How do you find the projection of b onto the column space of A?

The projection is A * x*, where x* is the least squares solution to Ax = b.

15 more cards in the app