Definition 6.3.1.
Let \(U\subset \R^n\text{.}\) Then \(U^\perp:=\{x\in \R^n: x\cdot u=0, \forall u\in U\}\) is called the orthogonal complement of \(U\text{.}\)
Section 6.3 Orthogonal Complements
Checkpoint 6.3.2.
(i) \(\{0\}^\perp = \R^n\)
(ii) \({\R^n}^{\perp}=\{0\}\text{.}\)
(iii) Let \(U\subset \R^n\text{.}\) Then \(U^\perp\) is a subspace of \(\R^n\text{.}\) Note that \(U\) need not be a subspace of \(\R^n\text{.}\)
Example 6.3.3.
Let \(U=span(\{(2,-1,1),(1,2,-1)\})\text{.}\) Then
\begin{align*}
U^\perp =\amp \{(x_1,x_2,x_3): 2x_1-x_2+x_3=0,x_1+2x_2-x_3=0\}\\
=\amp null\left(\begin{array}{rrr} 2 \amp -1 \amp 1 \\ 1 \amp 2 \amp -1 \end{array} \right)= \{t(1,-3,-5):t\in \R\}\text{.}
\end{align*}
Example 6.3.4.
Find the orthogonal complement of \(span(\{\left(2,\,0,\,1,\,-1\right), \left(1,\,2,\,0,\,-1\right)\})\text{.}\)
\begin{align*}
U^\perp =\amp \{(x_1,x_2,x_3,x_3): 2x_1+x_3-x_4=0,x_1+2x_2-x_4=0\}\\
=\amp null\left(\begin{array}{rrrr} 2 \amp 0 \amp 1 \amp -1 \\ 1 \amp 2 \amp 0 \amp -1 \end{array} \right)\\
=\amp span(\{(1, 0, -1, 1),(0, 1, 2, 2)\})
\end{align*}
Definition 6.3.5.
Let \(U\) be a subspace of \(\R^n\) with and orthogonal basis \(\{u_1,\ldots,u_k\}\text{.}\) If \(x\in \R^n\text{,}\) then
\begin{equation}
\proj_U(x)=\frac{x\cdot u_1}{\norm{u_1}^2}u_1+\frac{x\cdot u_2}{\norm{u_2}^2}u_2+\cdots + \frac{x\cdot u_k}{\norm{u_k}^2}u_k\text{.}\tag{6.3.1}
\end{equation}
Checkpoint 6.3.6.
Let \(U\) be a subspace of \(\R^n\) and \(p=\proj_U(x)\text{.}\) Then
(i) \(p\in U\) and \((x-p) \in U^\perp\text{.}\)
(ii) \(p\) is a vector in \(U\text{,}\) which is closet to \(x\text{.}\) That is for all \(y\in U\text{,}\) \(\norm{x-p}\leq \norm{x-y}\text{.}\) Note that \((x-p)\perp (y-p)\text{.}\) Hence by the Pythagoras theorem, \(\norm{x-p}^2+\norm{p-y}^2=\norm{x-y}^2\text{.}\)
\begin{equation*}
\norm{x-y}^2=\norm{x-p+p-y}^2=\norm{x-p}^2+\norm{y-p}^2\geq \norm{x-p}^2\text{.}
\end{equation*}
Example 6.3.7.
Consider the plane \(\pi=\{(x,y,z\in \R^3:3x+y-5z=0\}\text{.}\) It is easy to see that \(v_1=(2,-1,1),
v_2=(1,2,1)\) lie on the plane \(\pi\text{.}\) Using the Gram-Schmidt process we can find an orthogonal basis \(u_1 = (2,-1,1),
u_2=(2/3, 13/6, 5/6)\) on \(\pi\text{.}\) Let us find the orthogonal projection of \(v=(1,2,3)\) onto \(\pi\text{.}\) The required vector
\begin{equation*}
\proj_\pi(v)=\frac{v\cdot u_1}{\norm{u_1}^2}u_1+\frac{v\cdot u_2}{\norm{u_2}^2}u_2=(13/7, 16/7, 11/7)\text{.}
\end{equation*}
How to find the orthogonal projection of a vector on to the subspace spanned by a set of vectors in \(\R^n\text{?}\) Let \(\beta = \{w_1,\ldots, w_k\}\) be a basis of \(W\text{.}\) We want to find the vector \(p\) which the orthogonal projection of \(v\) onto \(W\text{.}\)
Note that \(p\in W\text{,}\) therefore, there exist scalars \(x_1,\ldots, x_k\) such that
\begin{equation*}
p=x_1 v_1+\cdots + x_k v_k=Ax
\end{equation*}
where \(A=[v_1~\cdots ~ v_k]\) and \(x=[x_1~\cdots~x_k]^T\text{.}\)
It is clear that \(v-p=v-Ax\in W^T\text{.}\) Hence \((v_i)^T(v-Ax)=0\) for \(1\leq i\leq k\text{.}\) This is same as
\begin{equation*}
A^T(v-Ax)=0 \implies x=(A^TA)^{-1}(A^Tv)\text{.}
\end{equation*}
Hence
\begin{equation}
p=A(A^TA)^{-1}(A^Tv)\tag{6.3.2}
\end{equation}
The matrix \(Q=A(A^TA)^{-1}A^T\) is called the projection matrix for the subspace \(W\text{.}\)
Checkpoint 6.3.8.
Let \(A\) be an \(n\times n\) real matrix. Recall \({\cal R}(A)\) and \({\cal N}(A)\text{,}\) rangeor columns space of \(A\) and null or kernel of \(A\text{.}\) Then show that (i) \({\cal R}(A)^\perp={\cal N}(A^T)\) and (ii) \({\cal N}(A)^\perp={\cal R}(A^T)\text{.}\)
Hint.
Let \(x \in {\cal R}(A)^\perp\text{.}\) Then \(y^TA^Tx=(Ay)^Tx=0\) for all \(y\text{.}\) Hence \(x\in {\cal N}(A^T)\text{.}\) This implies that, \({\cal R}(A)^\perp\subseteq {\cal N}(A^T)\text{.}\) Next let \(x\in {\cal N}(A^T)\) then \((Ay)^Tx=y^TA^Tx=0\) for all \(y\text{.}\) Hence \(x\in {\cal R}(A)^\perp\text{.}\) That is, \({\cal N}(A^T) \subseteq {\cal R}(A)^\perp\text{.}\)
