Definition 6.3.1.
Let \(U\subset \R^n\text{.}\) Then \(U^\perp:=\{x\in \R^n: x\cdot u=0, \forall u\in U\}\) is called the orthogonal complement of \(U\text{.}\)
Section 6.3 Orthogonal Complements
In this section, we shall explore the notion of orthogonal complement of a subspace in a \(\mathbb{R}^n\text{.}\)
Subsection 6.3.1 Orthogonal Complements
Checkpoint 6.3.2.
-
\(\displaystyle \{0\}^\perp = \R^n\)
-
\({\R^n}^{\perp}=\{0\}\text{.}\)
-
Let \(U\subset \R^n\text{.}\) Then \(U^\perp\) is a subspace of \(\R^n\text{.}\) Note that \(U\) need not be a subspace of \(\R^n\text{.}\)
Example 6.3.3.
Let \(U={\rm span}(\{(2,-1,1),(1,2,-1)\})\text{.}\) Then
\begin{align*}
U^\perp =\amp \{(x_1,x_2,x_3): 2x_1-x_2+x_3=0,x_1+2x_2-x_3=0\}\\
=\amp {\rm null}\left(\begin{array}{rrr} 2 \amp -1 \amp 1 \\ 1 \amp 2 \amp -1 \end{array} \right)= \{t(1,-3,-5):t\in \R\}\text{.}
\end{align*}
Example 6.3.4.
Find the orthogonal complement of \(span(\{\left(2,\,0,\,1,\,-1\right), \left(1,\,2,\,0,\,-1\right)\})\text{.}\)
\begin{align*}
U^\perp =\amp \{(x_1,x_2,x_3,x_3): 2x_1+x_3-x_4=0,x_1+2x_2-x_4=0\}\\
=\amp {\rm null}\left(\begin{array}{rrrr} 2 \amp 0 \amp 1 \amp -1 \\ 1 \amp 2 \amp 0 \amp -1 \end{array} \right)\\
=\amp {\rm span}(\{(1, 0, -1, 1),(0, 1, 2, 2)\})
\end{align*}
Definition 6.3.5.
Let \(U\) be a subspace of \(\R^n\) with an orthogonal basis \(\{u_1,\ldots,u_k\}\text{.}\) If \(x\in \R^n\text{,}\) then
\begin{equation}
\proj_U(x)=\frac{x\cdot u_1}{\norm{u_1}^2}u_1+\frac{x\cdot u_2}{\norm{u_2}^2}u_2+\cdots + \frac{x\cdot u_k}{\norm{u_k}^2}u_k\text{.}\tag{6.3.1}
\end{equation}
\(\proj_U(x)\) called the orthogononal projection of \(x\) onto \(U\text{.}\)
Checkpoint 6.3.6.
(ii) \(p\) is a vector in \(U\text{,}\) which is closet to \(x\text{.}\) That is for all \(y\in U\text{,}\)
\begin{equation*}
\norm{x-p}\leq \norm{x-y}\text{.}
\end{equation*}
Theorem 6.3.7.
Let \(W\) be a subspace of \(\R^n\text{.}\) Then for \(x\in \R^n\text{,}\) there exist unique \(w\in W\) and \(w'\in W^{\perp}\) such that \(x=w+w'\text{.}\)
In particualar \(\R^n=W+W^{\perp}\text{.}\) Since \(W\cap W^{\perp}=\{0\}\text{,}\) we have \(\R^n=W\oplus W^{\perp}\text{,}\) called the direct sum of \(W\) and \(W^{\perp}\text{.}\)
Proof.
Choose an orthonormal basis \(\{w_1,\ldots,w_k\}\text{.}\) Define \(w={\rm proj}_W(x)\text{.}\) Define \(w'=x-p\text{.}\) We claim that \(w'\in W^{\perp}\text{.}\) Hence \(x=w+w'\text{.}\)
Suppose \(x=w+w'\) and also \(x=w_1+w_1'\) such that \(w,w_1\in W\) and \(w',w_1'\in W^{\perp}\text{.}\) Then \(w-w_1=w'-w_1'\text{.}\) This implies
\begin{equation*}
w-w_1,w'-w_1'\in W\cap W^{\perp}=\{0\}.
\end{equation*}
Hence we have \(w=w_1\) and \(w'=w_1'\text{.}\)
Checkpoint 6.3.8.
Hint.
Let \(x\in W\text{,}\) then for all \(y\in W^\perp\text{,}\) we have \(y^Tx=0\text{.}\) That is \(x\in {(W^\perp)}^{\perp}\text{.}\) Hence \(W\subset {(W^\perp)}^{\perp}\text{.}\) To show \(W={(W^\perp)}^{\perp}\text{,}\) it is enough to show \(\dim{(W)}=\dim{({(W^\perp)}^{\perp})}\text{.}\)
\begin{equation*}
\dim{({(W^\perp)}^{\perp})}=n-\dim{(W^\perp)}=n-(n-\dim{(W)})=\dim{(W)}.
\end{equation*}
Example 6.3.9.
Consider the plane \(\pi=\{(x,y,z)\in \R^3:3x+y-5z=0\}\text{.}\) It is easy to see that \(v_1=(2,-1,1),
v_2=(1,2,1)\) lie on the plane \(\pi\text{.}\) Using the Gram-Schmidt process we can find an orthogonal basis \(u_1 = (2,-1,1),
u_2=(2/3, 13/6, 5/6)\) on \(\pi\text{.}\) Let us find the orthogonal projection of \(v=(1,2,3)\) onto \(\pi\text{.}\) The required vector
\begin{equation*}
\proj_\pi(v)=\frac{v\cdot u_1}{\norm{u_1}^2}u_1+\frac{v\cdot u_2}{\norm{u_2}^2}u_2=(13/7, 16/7, 11/7)\text{.}
\end{equation*}
Orthogonal Projection onto a subspace
How to find the orthogonal projection of a vector on to the subspace spanned by a set of vectors in \(\R^n\text{?}\) Let \(\beta = \{w_1,\ldots, w_k\}\) be a basis of \(W\text{.}\) We want to find the vector \(p\) which is the orthogonal projection of \(v\) onto \(W\text{.}\)
Note that \(p\in W\text{,}\) therefore, there exist scalars \(x_1,\ldots, x_k\) such that
\begin{equation*}
p=x_1 v_1+\cdots + x_k v_k=Ax
\end{equation*}
where \(A=[v_1~\cdots ~ v_k]\) and \(x=[x_1~\cdots~x_k]^T\text{.}\)
It is clear that \(v-p=v-Ax\in W^\perp\text{.}\) Hence \((v_i)^T(v-Ax)=0\) for \(1\leq i\leq k\text{.}\) This is same as
\begin{equation*}
A^T(v-Ax)=0 \implies x=(A^TA)^{-1}(A^Tv)\text{.}
\end{equation*}
Example 6.3.10.
Consider a set of vectors
\begin{equation*}
v_1 = \left(\begin{array}{r}
-1 \\
1 \\
0 \\
-3 \\
-3
\end{array}\right), v_2=\left(\begin{array}{r}
1 \\
-5 \\
3 \\
-1 \\
4
\end{array}\right), v_3 = \left(\begin{array}{r}
-5 \\
13 \\
-6 \\
-7 \\
-17
\end{array}\right), v_4 = \left(\begin{array}{r}
-2 \\
1 \\
3 \\
8 \\
1
\end{array}\right).
\end{equation*}
Let \(W\) be the subspace spanned by \(\{v_1,v_2,v_3,v_4\}\) in \(\mathbb{R}^5\text{.}\) Find the orthogonal projection of the vector \(u = \left(\begin{array}{r}
1 \\
3 \\
5 \\
7 \\
11
\end{array}\right)\) onto \(W\) in Sage.
Solution.
Since pivots indices are \(\{0,1,3\}\text{,}\) \(v_1,v_2,v_4\) are linearly independent. We define a matrix \(M\) of whose columns are \(v_1,v_2,v_4\text{.}\) Then the orthogonal projection of \(u\) onto \(W\) is \(M(M^TM)^{-1}M^Tu\text{.}\)
Checkpoint 6.3.11.
Let \(A\) be an \(n\times n\) real matrix. Recall \({\cal R}(A)\) and \({\cal N}(A)\text{,}\) range or columns space of \(A\) and null or kernel of \(A\text{.}\) Then show that
-
\({\rm Row}(A)^\perp = {\rm null}(A)\text{.}\)
-
\({\cal N}(A)^\perp={\cal R}(A^T)\text{.}\)
Hint.
Let \(x \in {\cal R}(A)^\perp\text{.}\) Then for \(y\in \R^n\text{,}\) \(Ay\in {\cal R}(A)\text{.}\) Hence \(0=(Ay)^Tx=y^TA^Tx\text{.}\) This implies \(A^Tx=0\text{.}\) Hence \(x\in {\cal N}(A^T)\text{.}\) That is, \({\cal R}(A)^\perp\subseteq {\cal N}(A^T)\text{.}\)
Next let \(x\in {\cal N}(A^T)\) then \((Ay)^Tx=y^TA^Tx=0\) for all \(y\text{.}\) Hence \(x\in {\cal R}(A)^\perp\text{.}\) That is, \({\cal N}(A^T) \subseteq {\cal R}(A)^\perp\text{.}\) Hence \({\cal R}(A)^\perp={\cal N}(A^T)\text{.}\)
Replacing \(A\) by \(A^T\text{,}\) in (1), we get
\begin{equation*}
{{\cal R}(A^T)}^\perp={\cal N}(A).
\end{equation*}
Hence
\begin{equation*}
{{\cal N}(A)}^\perp={{{\cal R}(A^T)}^\perp}^\perp={\cal R}(A^T).
\end{equation*}
We write rows of \(A\) and \(r_1,\ldots r_m\) and columns of \(A\) as \(a_1,\ldots a_n\text{.}\) Then for any \(x\ijn \R^n\text{,}\) we have
\begin{equation*}
Ax=x_1a_1+x_2a_2+\cdots+x_na_n=
\begin{bmatrix} r_1\cdot x\\r_2\cdot a\\\vdots \\r_m\cdot x\end{bmatrix}.
\end{equation*}
Hence if \(Ax=0\text{,}\) then \(r_i\cdot x=0\) for all \(i\text{.}\) This implies \(x\in {\rm Row}(A)^\perp\text{.}\) That is \({\rm null}(A)\subset {\rm Row}(A)^\perp \text{.}\)
If \(y\in {\rm Row}(A)^\perp\text{,}\) then \(r_i\cdot y=0\) for all \(i\text{.}\) Hence \(Ay=0\text{.}\) This implies \(y\in {\rm null}(A)\text{.}\) That is, \({\rm Row}(A)^\perp\subset {\rm null}(A)\text{.}\) Hence \({\rm Row}(A)^\perp = {\rm null}(A)\text{.}\)
Checkpoint 6.3.12.
Let \(A\) be an \(n\times n\) real symmetric matrix and \(W\text{,}\) a \(A\)-invariant subspace of \(\R^n\text{.}\) Then show that \(W^\perp\) is \(A\)-invariant.
The above result is not true if \(A\) is not a symmetric matrix. Consider \(A=\begin{pmatrix}1 \amp 1\amp -1\\-1 \amp 1 \amp 0\\0 \amp 0 \amp 1\end{pmatrix}\text{.}\) Then it is easy to check that \(W\text{,}\) \(xy\)-plane is \(A\)-invariant and \(W^\perp =\R e_3\text{.}\) But \(W^\perp\) is not \(A\)-invariant.
Subsection 6.3.2 Projection and Reflection
In this subsection, we deal with two concepts, orthogonla projection of a vector onto a hyperplane in \(\mathbb{R}^n\) and reflection of a vector about a hyperplane.
Definition 6.3.13.
A hyperplane \(H\) in \(\mathbb{R}^n\) is an \(n-1\)-dimensional affine space defined by
\begin{equation}
H:=\{x\in \mathbb{R}^n: a^x=b\} \tag{6.3.3}
\end{equation}
where \(a\) is a nonzero vector in \(\mathbb{R}^n\) and \(b\in \mathbb{R}\text{.}\) If \(b=0\text{,}\) then \(H\) is an \(n-1\)-dimensiional subspace of of \(\mathbb{R}^n\text{.}\)
Note that if \(H\) is an \(n-1\)-dimensiional subspace of of \(\mathbb{R}^n\text{,}\) defined by \(a\text{,}\) let us denote it by \(H_a\text{,}\) then \(a\) is orthogonal to \(H\text{.}\)
How to find the orthogonal projection of a vector \(v\in \mathbb{R}^n\) onto \(H_a\text{?}\) Note that we have already done one way by finding a basis vector of \(H_a\text{.}\) We would like to get a formula in terms of \(a\text{.}\) If \(v_{H_a}\) is the orthogonal projection of \(v\) onto \(H_a\text{,}\) then it can be obianed by taking out the orthogonal prjection of \(v\) onto the vector \(a\) from \(v\text{.}\) Thus we have
\begin{equation}
v_{H_a} = v-\frac{v\cdot a}{\norm{a}^2}a=v-\frac{aa^T}{\norm{a}^2}v=(I-\frac{aa^T}{\norm{a}^2})v.\tag{6.3.4}
\end{equation}
The matrix \(P_H=(I-\frac{aa^T}{\norm{a}^2})\) is nothing but the projection matrix.
How do we obtain \(v_H\text{,}\) the orthogonal projection of \(v\) onto \(H\text{?}\) Note that \(v-v_H\) is parallel to \(a\text{.}\) Hence \(v-v_H=ta\) for some scalar \(t\text{.}\) Since \(v_H=v-ta\in H\text{,}\) we have \(a^T(v-ta)=b\text{.}\) Simplyfying we get
\begin{equation}
v_H = v-\frac{v\cdot a-b}{\norm{a}^2}a.\tag{6.3.5}
\end{equation}
Example 6.3.14.
Find the orthogonal projevction of a vector \((1,2,3,4)\) to to the hyperplane given by \(3x_1-2x_2+5x_3+4x_4=7\) in \(\mathbb{R}^4\)
Next we wish to find the reflection of a vector \(v\) about the hyperplane \(H\text{.}\) Suppose \(v_R\) is the projection of \(v\) about \(H\text{,}\) then \(v_R\) can be obtained by going the same distance on the opposite side along the vector \(v-v_H\text{.}\) Thus
\begin{equation}
v_R = v_H-(v-v_H)=2v_H-v\tag{6.3.6}
\end{equation}
Substituting the formula for \(v_H\text{,}\) and after simplification, we get
\begin{equation}
v_R = v - 2 \frac{a^T v - b}{\norm{a}^2} a.\tag{6.3.7}
\end{equation}
As as special if \(b = 0\text{,}\) the hyperplane passes through the origin,
\begin{equation*}
v_R = v - 2 \frac{a^T v}{\norm{a}^2} a=(I-2\frac{aa^T}{\norm{a}^2}v).
\end{equation*}
If \(a\) is a unit vector, \(\|a\| = 1\text{,}\) this simplifies to:
\begin{equation*}
v_R = v - 2(a^T v) a=(I-2(aa^T))v.
\end{equation*}
The matrix \(R_H =I-2\frac{aa^T}{\norm{a}^2}\) is called the reflection matrix.
Example 6.3.15.
Find the reflection of the vector \(x=(4,2,-1,3)\) about the hyperplane \(2x_1-x_2+3x_3+4x_4=5\text{.}\)
Activity 6.3.1.
Plottig orthogonal projection and reclection in \(\mathbb{R}^3\)Use sage to plot the hyperplane \(H\) defined by the equation \(3x-2y+z=2\) and also plot the vector \(v=(1,-1,3)\) and its orthogonal projection \(v_H\) and the reflection \(v_R\) about \(H\text{.}\)
Orthogonal complements provide a way to decompose vector spaces and are closely related to projections and reflections.
