Skip to main content

Linear Algebra with SageMath A Gentle Introduction

Ajit Kumar

Section 7.1 Inner Product

In the last chapter, we dealt with notion of dot product and geometry in \(\R^n\text{.}\) The dot product and related notion can be generalized to an arbitrary vector space over \(\R\) or \(\mathbb{C}\text{.}\) All the notions, we have learned in the last section can be generalized over an inner product space. In this chapter we, shall introduce inner product on a vector space \(V\) over \(\mathbb{R}\) and define extend the results studied in the last chapter on \(V\text{.}\)
🔗
Note that the dot product of two vectors in \(\R^n\) is a scalar, in particular, dot product can be thought of as a function from \(`\cdot' \colon \R^n\times \R^n \to\R\) satisfying the following properties:
🔗
  1. \(x \cdot x\geq 0\) for all \(x\in \R^n\text{.}\)
    🔗
    🔗
  2. \(x \cdot x= 0\) if and only if \(x=0\text{.}\)
    🔗
    🔗
  3. \(x \cdot y=y\cdot x\) for all \(x,y \in \R^n\text{.}\)
    🔗
    🔗
  4. \(x\cdot (y+z) = x\cdot y +x\cdot z\) for all \(x,y,z\in \R^n\text{.}\)
    🔗
    🔗
  5. \(x\cdot (\alpha y)=\alpha x\cdot y = (\alpha x)\cdot y\text{.}\)
    🔗
    🔗
🔗
The notion of dot product on \(\R^n\) can ve generalized on vector space known as inner product. We have the following definition.
🔗

Definition 7.1.1. Inner Product.

Let \(V\) be a vector space over \(\R\text{.}\) An inner product on \(V\) is a function that assigns a real number \(\langle x, y\rangle\) to every pair \(x,y\) of vectors in \(V\) (that is, a function \(\langle \cdot, \cdot\rangle\colon V \times V \to \R\)) satisfying the following properties.
🔗
  1. \(\langle x, x\rangle \geq 0\) for all \(x \in V\) and \(\langle x, x\rangle = 0\) if and only if \(x=0\text{.}\)
    🔗
    🔗
  2. \(\langle x, y\rangle = \langle y, x\rangle\) for all \(x,y \in V\text{.}\) (Symmetry)
    🔗
    🔗
  3. \(\langle x, (y+z)\rangle=\langle x, y\rangle+\langle x, z\rangle\) for all \(x,y,z\in V\text{.}\)
    🔗
    🔗
  4. \(\langle x, (\alpha y)\rangle=\alpha\langle x, y\rangle\) for all \(x,y\in V\) and \(\alpha \in \R\text{.}\)
    🔗
    🔗
🔗
If \(V\) is real vector space with inner product \(\langle. , .\rangle\text{.}\) Then \((V, \langle . , .\rangle)\) called in inner product space over \(\R\text{.}\)
🔗
🔗
The last two properties make the inner product linear in the second variable. Using the symmetry property, it can also be shown that the inner product is linear in the first variable as well. That is,
\begin{equation*} \langle (x+y),z\rangle=\langle x, z\rangle+\langle y, z\rangle, \text{ and } \langle (\alpha x) y\rangle=\alpha\langle x, y\rangle \end{equation*}
🔗
Next we look at several examples of inner procuct on various vector spaces that we have defined in Chapter 4.
🔗

Example 7.1.2.

On \(\R^n\text{,}\) the standard dot product is an inner product. Thus define
\begin{equation*} \langle x, y\rangle:=x\cdot y \end{equation*}
🔗
This is also called the Euclidean inner product on \(\R^n\text{.}\)
🔗
🔗

Example 7.1.3.

Let \(V=M_n(\R)\text{,}\) the set of all \(n\times n\) matrices over \(\R\text{.}\) Define
\begin{equation*} \langle A, B\rangle:=tr(AB^T) \end{equation*}
🔗
It is easy to show that this is an inner product on \(M_n(\R)\text{.}\)
🔗
Note that this inner product can be thought of as the standard dot product on \(\R^{n^2}\text{.}\) The elements of the matrix \(A\) can be thought of as a vector in \(\R^{n^2}\text{.}\) Then
\begin{equation*} tr(AB^T) =\sum\sum a_{ij}b_{ij}\quad \quad \text{ why?}\text{.} \end{equation*}
Work with \(2\times 2\) matrices and then try to prove this for \(n\times n\) matrices.
🔗
🔗

Example 7.1.4.

Let \(A\) be an \(n\times n\) symmetric and positive definite matrix. On \(\R^n\text{,}\) define
\begin{equation*} \langle x, y \rangle := x\cdot Ay = x^TAy \end{equation*}
🔗
Then this is an inner product on \(\R^n\text{.}\) (Where do we need \(A\) to be positive definie?)
🔗
🔗

Remark 7.1.5.

Since \(A\) is symmetric and positive definite matrix, there exists a matrix \(B\) such that \(B^2=A\text{.}\) We call \(B\) as positive definite square root of \(A\) and is denoted by \(A^{1/2}\text{.}\)
🔗
\begin{equation*} \langle x,y\rangle = x^T A y = (A^{1/2}x)\cdot(A^{1/2}y), \end{equation*}
which shows this inner product is just the usual dot product after the linear change of variables \(x\mapsto A^{1/2}x\text{.}\)
🔗

Remark 7.1.6.

We have defined an inner product on \(\R^n\) using a symmetric positive definite matrix \(A\text{.}\) In fact any inner product on \(\R^n\) can be obtained in this way.
🔗
Let \(\innprod{x}{y}\) be an inner product on \(\R^n\text{.}\) Let \(e_1, e_2, \ldots, e_n\) be the standard basis of \(\R^n\text{.}\) For \(x,y\in \R^n\) with
\begin{equation*} x=\sum x_i e_i, \quad y= \sum y_j e_j. \end{equation*}
we have,
\begin{equation*} \innprod{x}{y}=\sum_{i}\sum_j x_i\innprod{e_i}{e_j}y_j. \end{equation*}
Let us define \(a_{ij}=\innprod{e_i}{e_j}y_j\text{.}\) Then it is easy to see that
\begin{equation*} \innprod{x}{y}=\begin{bmatrix} x_1 \amp x_2 \amp \cdots \amp x_n \end{bmatrix} A \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}=x^TAy, \end{equation*}
where \(A=[a_{ij}]\) is a symmetric positive definite matrix.(why?)
🔗
🔗

Example 7.1.7.

Let \(V={\cal C}([0,1])\) the set of all continuous function from \([0,1]\) to \(\R\text{.}\) Define
\begin{equation*} \langle f, g \rangle:=\int_0^1 f(t)g(t)dt\text{.} \end{equation*}
🔗
This is an inner product on \(V\text{.}\) (You may need real analysis to conclude that \(\int_0^1 f(t)f(t)dt=0\) then \(f=0\text{.}\))
🔗
🔗

Example 7.1.8.

Let \(p\) and \(q\) be two polynomials in \({\cal P}_n(\mathbb{R})\text{.}\) Then define
\begin{equation*} \langle p,q \rangle := p(0)q(0)+p(1)q(1)+\cdots +p(n)q(n)\text{.} \end{equation*}
🔗
It is easy to see that \(\langle p,q \rangle\) defined inner product on the vector space \({\cal P}_n(R)\text{.}\) This inner product is called the discrete inner product on \({\cal P}_n(\mathbb{R})\text{.}\)
🔗
Here \(0, 1, 2, \ldots, n\) are nothing special. Instead, we can use any \(n+1\) distinct real numbers, \(c_0,c_1,\ldots, c_n\text{.}\)
🔗
🔗

Definition 7.1.9.

Let \((V, \langle .\rangle)\) be a real inner product space. Then norm of any vector \(x\in V\) corresponding to the inner product \(\langle . \rangle\) is defined as
\begin{equation*} \norm{x}=\sqrt{\langle x, x\rangle}\text{.} \end{equation*}
The distance between twp vectors \(x\) and \(y\) is defined as
\begin{equation*} d(x,y):=\norm{x-y}\text{.} \end{equation*}
🔗
🔗

Checkpoint 7.1.10.

Let \(V\) be an inner product space. Then for any two vectors \(x, y\in V\text{,}\) show that
1. \(\norm{x+y}^2=\norm{x}^2+\norm{y}^2+2\inner{x}{y}\)
🔗
2. \(\norm{x-y}^2=\norm{x}^2+\norm{y}^2-2\inner{x}{y}\)
🔗
🔗
Hint.
The proof follows by looking at \(\langle x+y,x+y \rangle \) and \(\langle x-y,x-y \rangle \) and using the properties of inner product.
🔗
🔗
🔗

Checkpoint 7.1.11.

If \(x, y\) are two vectors in an inner product space \(V\) with inner product \(\langle .\rangle\text{.}\) Then show that
\begin{equation*} \norm{x+y}^2+\norm{x-y}^2=2(\norm{x}^2+\norm{y}^2)\text{.} \end{equation*}
🔗
This is called the parallelogram identity. Geometrically, in a parallelogram, the sum of square of the diagonals is twice the sum of the squares of the side lengths.
🔗
Hint.
Proof follows from Checkpoint 7.1.10.
🔗
🔗
🔗

Checkpoint 7.1.12.

Let \(a,b,c\) be three positive real numbers and ,\(x=(x_1,x_2,x_3), y=(y_1,y_2,y_3)\) be vectors in \(\R^3\text{.}\) Define
\begin{equation*} \langle x, y\rangle:=\frac{x_1y_1}{a^2}+\frac{x_2y_2}{b^2}+\frac{x_3y_3}{c^2}. \end{equation*}
Show that this is an inner product on \(\R^3\text{.}\) Also what are the unit vectors with respect to this inner product on \(\R^3\text{?}\)
🔗
🔗

Proof.

If \(x = 0\) or \(y = 0\text{,}\) then \(\innprod{x}{y} = 0\text{.}\) Also either \(\innprod{x}{x} = 0\) or \(\innprod{y}{y} = 0\text{.}\) Hence the result follows.
🔗
First we consider the case when \(\norm{x} = \norm{y} = 1\) . Then
\begin{align*} 0 \leq \amp\innprod{x - y}{x - y}\\ = \amp\innprod{x}{x} + \innprod{y}{y} - 2\innprod{x}{y}\\ = \amp 2 - 2\innprod{x}{y} \end{align*}
This gives \(\innprod{x}{y} \leq 1=\norm{x}\norm{y}.\)
🔗
Similarly,
\begin{align*} 0 \leq \amp\innprod{x + y}{x + y}\\ = \amp\innprod{x}{x} + \innprod{y}{y} + 2\innprod{x}{y}\\ = \amp 2 + 2\innprod{x}{y}. \end{align*}
Thus \(\innprod{x + y}{x + y} \geq 0\) implies \(-\innprod{x}{y} \leq 1\text{.}\) From these two together we have
🔗
\begin{equation*} |\innprod{x}{y}|\leq 1 = \norm{x} \norm{y}. \end{equation*}
We now prove the statement for equality. Let \(\mod{\innprod{x}{y}} = 1.\) This implies either \(\innprod{x}{y} = 1\) or \(\innprod{x}{y} = -1.\) If \(\innprod{x}{y} = 1\text{,}\) from the above chain of inequalities we deduce that \(\innprod{x - y}{x - y} = 0\) or \(x = y.\) If \(\innprod{x}{y} = -1\text{,}\) we see that \(x = -y.\) Thus the equality holds if and only if \(x = \pm y.\)
🔗
Next suppose \(x\) and \(y\) are nonzero (not necessarily of unit length). Then we can write \(u = \frac{x}{\norm{x}}\) and \(v = \frac{y}{\norm{y}}\) are of unit length. Hence by the previous case, \(\mod{\innprod{u}{v}} \leq 1.\) This implies,
\begin{equation*} \mod{\innprod{\frac{x}{\norm{x}}}{\frac{y}{\norm{y}}}} = \mod{\frac{1}{\norm{x}} \frac{1}{\norm{y}} \innprod{x}{y}} \leq 1. \end{equation*}
thus we get \(\mod{\innprod{x}{y}} \leq \norm{x} \norm{y}.\)
🔗
If \(x\) and \(y\) are nonzero, then the equality from the earlier case means \(\innprod{x}{y} = \norm{x} \norm{y}\) or \(-\innprod{x}{y} = \norm{x} \norm{y}\text{.}\) Let us look at one of the case.
\begin{align*} \amp \innprod{x}{y} = \norm{x} \norm{y}\\ \amp \Longleftrightarrow \innprod{\frac{x}{\norm{x}}}{\frac{y}{\norm{y}}} = 1 \\ \amp \Longleftrightarrow \frac{x}{\norm{x}} = \frac{y}{\norm{y}}\\ \amp \Longleftrightarrow x = \frac{\norm{x}}{\norm{y}} y. \end{align*}
The other case is similar.
🔗
2nd Proof.
🔗
Fix \(x\) and \(y\) in \(V\text{.}\) If \(y = 0\text{,}\) then the result is obviously true. Without loss of generality assume that \(y \neq 0\text{.}\) Consider the real-valued function \(\varphi\colon \R \to \R\) defined as
\begin{equation*} \varphi(t) := \innprod{x + t y}{x + t y}\text{.} \end{equation*}
We shall use calclus and investigate the minimum value of \(\varphi(t)\) to prove the inequality.
\begin{align*} \varphi(t) \amp= \amp\innprod{x + t y}{x + t y}\\ \amp= \amp\innprod{x}{x} + 2 t \innprod{x}{y} + t^{2} \innprod{y}{y}. \end{align*}
Note that this is a differentiabke function of \(t\text{.}\) Differentiating \(\varphi(t)\) with respect to \(t\text{,}\) we get
\begin{equation*} \varphi'(t) = 2 \innprod{x}{y} + 2 t \innprod{y}{y}. \end{equation*}
Setting \(\varphi'(t) = 0\text{,}\) we get the critical point at
\begin{equation*} t = -\frac{\innprod{x}{y}}{\innprod{y}{y}}. \end{equation*}
Since \(\varphi''(t) =\innprod{y}{y} \gt 0\text{,}\) this critical point is a minimum. Thus
\begin{equation*} \varphi\left(-\frac{\innprod{x}{y}}{\innprod{y}{y}}\right) \geq 0. \end{equation*}
Substituting for \(t\) in the expression for \(\varphi(t)\text{,}\) we get
\begin{align*} 0 \amp\leq \innprod{x}{x} - \frac{\innprod{x}{y}^{2}}{\innprod{y}{y}}\\ \Longleftrightarrow \innprod{x}{y}^{2} \amp\leq \innprod{x}{x} \innprod{y}{y}\\ \Longleftrightarrow |\innprod{x}{y}| \amp\leq \norm{x} \norm{y}. \end{align*}
This completes the 2nd proof of the Cauchy-Schwarz inequality.
🔗
🔗
Let us play with the Cauchy-Schwarz inequality (7.1.1). Suppose \(x\) and \(y\) are non zero vectors in \(V\text{,}\) Then
\begin{equation*} |\inner{x}{y}|\leq \norm{x}\norm{y}\Rightarrow \frac{|\inner{x}{y}|}{\norm{x}\norm{y}}\leq 1 \end{equation*}
🔗
Hence we have
\begin{equation*} -1\leq \frac{\inner{x}{y}}{\norm{x}\norm{y}}\leq 1\text{.} \end{equation*}
🔗
Thus for any two non zero vectors, \(x\) and \(y\text{,}\) \(\frac{\inner{x}{y}}{\norm{x}\norm{y}}\) always lies between \(-1\) and 1. This allows us to define the angle between two non zero vectors. We assign this number to \(\cos\theta\) with \(\theta\in[-\pi,\pi]\) called the angle between \(x\) and \(y\text{.}\) Thus, if \(\theta\) is the angle between \(x\) and \(y\text{,}\) then we have
\begin{equation*} \cos \theta = \frac{\inner{x}{y}}{\norm{x}\norm{y}} \end{equation*}
🔗
All the notions that we defined for dot product, namely, orthogonality, orthogonal projection, Gram-Schmidt orthogonalization process can we defined in a similar manner. All we need to do is, replace the dot product by the given inner product.
🔗

Definition 7.1.16.

Any vector space \(V\) over \(\R\) with a function \(\norm{.} : V \to \R\) which satisfies all the properties mentioned in Theorem 7.1.15 is called a normed linear space.. Thus any inner product space is also a normed linear space.
🔗
🔗

Note 7.1.17.

The concepts such as orthogonality, orthogonal projection, orthogonal complement of any subset, orthogonal and orthonormal sets and Gram-Schmidt orthogonalization process etc that we defined and dealt with in the previuos chapter with respect to the dot product on \(\R^n\) can be defined on an inner product space. All we need to do is to replace the dot product by the corresponding inner product. We encourage readers to define each one of them.
🔗
🔗

Checkpoint 7.1.18.

Let \((V, \inner{.}{.})\) be a real inner product space. Define
  1. orthogonality of two vectors \(x\) and \(y\) in \(V\text{,}\)
    🔗
    🔗
  2. orthogonal complement of a subset \(U\) of \(V\text{,}\)
    🔗
    🔗
  3. Orthogonal projection of a vector \(v\) onto a non-zero vector \(u\text{,}\)
    🔗
    🔗
  4. orthogonal set and orthonormal sets in \(V\) and
    🔗
    🔗
  5. Gram-Schmidt orthogonalization process.
    🔗
    🔗
🔗
🔗
As mentioned earlier all the notions related to orthogonality can be defined in a similar manner on an inner product space and the results that we proved in the previous chapter hold true in this more general setting. However, let us state some of the definitions and results here for completeness.
🔗

Definition 7.1.19.

Let \((V,\innprod{.}{.})\) be an inner product space. Two vectors \(x,y\in V\) are said to be orthogonal if \(\innprod{x}{y}=0.\)
🔗
🔗

Definition 7.1.20.

Let \((V,\innprod{.}{.})\) be an inner product space. A set of vectors \(\{v_1,\ldots,v_k\}\) is said to be orthogonal set if \(\innprod{v_i}{v_j}=0\) for \(i\neq j\text{.}\)
🔗
In addition, if each vector in the set is of unit norm, i.e., \(\norm{v_i}=1\) for all \(i=1,2,\ldots,k\text{,}\) then the set is called an orthonormal set.
🔗
🔗

Checkpoint 7.1.21.

Let \(x, y\) be two vectors in an inner product space \(V\text{.}\) Then show that
  1. \(x\) and \(y\) are orthogonal if and only if \(\norm{x+y}=\norm{x-y}\text{.}\) (what does it mean geometrically?)
    🔗
    🔗
  2. \(x+y\) and \(x-y\) are orthogonal if and only if \(\norm{x}=\norm{y}\text{.}\)
    🔗
    🔗
🔗
🔗

Checkpoint 7.1.22.

Let \(C([-\pi,\pi])\) be the vectors space of set of continuous functions from \([-\pi,\pi]\) to \(\R\text{.}\) Define the inner product on \(C([-\pi,\pi])\) as
\begin{equation*} \inner{f}{g}:=\int_{-\pi}^\pi f(t) g(t)`dt\text{.} \end{equation*}
🔗
Show that under this inner product \(\{1, \sin nx,\cos mx\}\) is an orthogonal set.
🔗
🔗

Checkpoint 7.1.23. Pythagoras Theorem.

Let \((V, \inner{.}{.})\) be a real inner product space. Let \(x_1, x_2, \ldots x_n\) be \(n\) orthogonal vectors in \(V\text{.}\) Then
\begin{equation*} \norm{x_1+x_2+\cdots+x_n}^2=\norm{x_1}^2+\norm{x_2}^2+\cdots+\norm{x_n}^2\text{.} \end{equation*}
🔗
This is called the Pythagoras theorem.
🔗
🔗
Is the converse of the Pythagoras theorem true? Consider \(u=v=(1,1)\) and \(w=(0,-1)\text{.}\) It is easy to check that that
\begin{equation*} \norm{u+v+w}^2=\norm{u}^2+\norm{v}^2+\norm{w}^2, \end{equation*}
\(u,v, w\) ar not orthogonal to each other.
🔗

Checkpoint 7.1.24.

Let \(\beta=\{u_1,\ldots, u_n\}\) be an orthogonal basis of an inner product space \(V\text{.}\) Let \(v\in V\) and \(\theta_1,\ldots, \theta_n\) be the angle between \(v\) and \(u_1,\ldots, u_n\text{,}\) respectively. Then
\begin{equation*} \cos\theta_1^2+\cdots+\cos\theta_n^2=1\text{.} \end{equation*}
🔗
Here \(\cos\theta_i\) are called the direction cosines of \(v\) corresponding to \(\beta\text{.}\)
🔗
Hint.
If \(v=0\) the statement is trivial (all direction cosines are zero). Assume \(v\neq 0\text{.}\) Since \(\beta\) is an orthogonal basis we can expand \(v\) uniquely as
\begin{equation*} v = \sum_{i=1}^{n} \frac{\innprod{v}{u_{i}}}{\innprod{u_{i}}{u_{i}}}\, u_{i}. \end{equation*}
Taking norms squared and using orthogonality of the \(u_{i}\) gives
\begin{equation*} \norm{v}^{2} = \sum_{i=1}^{n} \frac{\lvert\innprod{v}{u_{i}}\rvert^{2}}{\innprod{u_{i}}{u_{i}}}. \end{equation*}
By definition of the angle \(\theta_{i}\) between \(v\) and \(u_{i}\) we have
\begin{equation*} \cos\theta_{i} = \dfrac{\innprod{v}{u_{i}}}{\norm{v}\,\norm{u_{i}}}. \end{equation*}
Using the relation
\begin{equation*} \dfrac{\lvert\innprod{v}{u_{i}}\rvert^{2}}{\innprod{u_{i}}{u_{i}}} = \norm{v}^{2}\,\dfrac{\lvert\innprod{v}{u_{i}}\rvert^{2}}{\norm{v}^{2}\,\norm{u_{i}}^{2}} = \norm{v}^{2}\,\cos^{2}\theta_{i}, \end{equation*}
substitute into the norm identity to obtain
\begin{equation*} \norm{v}^{2} = \sum_{i=1}^{n} \norm{v}^{2}\,\cos^{2}\theta_{i} = \norm{v}^{2} \sum_{i=1}^{n} \cos^{2}\theta_{i}. \end{equation*}
Dividing by \(\norm{v}^{2}\) (which is positive because \(v\neq 0\)) yields
\begin{equation*} \sum_{i=1}^{n}\cos^{2}\theta_{i}=1, \end{equation*}
as required.
🔗
🔗
🔗

Definition 7.1.25.

A basis \(\{v_1, \ldots, v_n\}\) of an inner product space \(V\) is said to be orthonormal basis if we have \(\innprod{v_i}{v_j}=\delta_{ij}\) for \(1 \leq i,j \leq n\text{.}\)
🔗
🔗
Let us assume that \(\{v_i\}\) is an orthonormal basis of \(V\text{.}\) Write \(v = \sum_i \alpha_i v_i\text{.}\) Taking the inner product of both sides with the vector \(v_j\text{,}\) and using the orthonormal properties of the basis, we get
\begin{align*} \innprod{v}{v_j} \amp = \innprod{\sum_i \alpha_i v_i}{v_j}\\ \amp =\sum_i \innprod{\alpha_i v_i}{v_j}\\ \amp = \sum_i \alpha_i \innprod{v_i}{v_j} \\ \amp= \sum_i \alpha_i \delta_{ij} =\alpha_j \end{align*}
Thus \(\alpha_j = \innprod{v}{v_j}\text{.}\) In particular, we have
\begin{equation*} v = \sum_i \innprod{v}{v_i}v_i. \end{equation*}
🔗
Next we turn our attention to finding an orthogonal basis in an inner product space using the Gram-Schmidt process. The process is exactly same as before, we just need to replace the dot product by the given inner product. Let \((V, \inner{.}{.})\) be an inner product space. Let \(\{v_1, v_2, \ldots, v_n\}\) be a basis of \(V\text{.}\) We construct an orthogonal basis \(\{u_1, u_2, \ldots, u_n\}\) as follows:
  1. Set \(u_1 = v_1\text{.}\)
    🔗
    🔗
  2. For \(2 \leq k \leq n\text{,}\) set
    \begin{equation*} u_k = v_k - \sum_{j=1}^{k-1} \frac{\inner{v_k}{u_j}}{\norm{u_j}^2} u_j. \end{equation*}
    🔗
    🔗
🔗
Thus we have the following result.
🔗

Proof.

The proof proceeds by induction on \(k\text{.}\) For \(k = 1\text{,}\) set \(u_1 = v_1\text{,}\) which is nonzero since \(v_1\) is linearly independent. Suppose \(u_1, \ldots, u_{k-1}\) have been constructed such that they are mutually orthogonal and \(\operatorname{span}\{u_1, \ldots, u_{k-1}\} = \operatorname{span}\{v_1, \ldots, v_{k-1}\}\text{.}\)
🔗
Define \(u_k = v_k - \sum_{j=1}^{k-1} \frac{\innprod{v_k}{u_j}}{\innprod{u_j}{u_j}} u_j.\) Then for each \(i < k\text{,}\)
🔗
\begin{align*} \innprod{u_k}{u_i} \amp = \innprod{v_k}{u_i} - \sum_{j=1}^{k-1} \frac{\innprod{v_k}{u_j}}{\innprod{u_j}{u_j}} \innprod{u_j}{u_i}\\ \amp= \innprod{v_k}{u_i} - \frac{\innprod{v_k}{u_i}}{\innprod{u_i}{u_i}}\innprod{u_i}{u_i} = 0. \end{align*}
Hence, \(u_k\) is orthogonal to all previous \(u_i\text{.}\) This completes the inductive step, and the result follows.
🔗
🔗

Remark 7.1.27.

The Gram–Schmidt process provides an explicit algorithm for transforming any linearly independent set into an orthonormal one. It is a fundamental tool in numerical linear algebra, functional analysis, and the construction of orthogonal polynomials.
🔗
🔗

Example 7.1.28.

Let \(V=\mathcal{P}_2(\mathbb{R})\) and define for \(p,q \in V\)
\begin{equation*} \innprod{p}{q} := p(0)q(0) + p(1)q(1) + p(2)q(2). \end{equation*}
Apply Gram–Schmidt to the linearly independent set \(\{v_{1},v_{2},v_{3}\} = \{1,x,x^{2}\}\text{.}\)
🔗
\(u_{1}=v_{1}=1\text{.}\)
\begin{equation*} \innprod{u_{1}}{u_{1}} = 1^2 + 1^2 + 1^2 = 3. \end{equation*}
🔗
\begin{equation*} u_{2} = v_{2} - \dfrac{\innprod{v_{2}}{u_{1}}}{\innprod{u_{1}}{u_{1}}}\,u_{1} = x - \dfrac{x(0)+x(1)+x(2)}{3}\cdot 1= x - 1. \end{equation*}
\begin{equation*} \innprod{u_{2}}{u_{2}} = ( -1)^2 + 0^2 + 1^2 = 1+0+1 = 2. \end{equation*}
🔗
\begin{equation*} u_{3} = v_{3} - \dfrac{\innprod{v_{3}}{u_{1}}}{\innprod{u_{1}}{u_{1}}}\,u_{1} - \dfrac{\innprod{v_{3}}{u_{2}}}{\innprod{u_{2}}{u_{2}}}\,u_{2}. \end{equation*}
\begin{equation*} \innprod{v_{3}}{u_{1}} = x^{2}(0)+x^{2}(1)+x^{2}(2) = 0 + 1 + 4 = 5. \end{equation*}
\begin{align*} \innprod{v_{3}}{u_{2}} \amp = = \innprod{x^{2}}{x-1}\\ \amp = 0\cdot(-1)+1\cdot 0 + 4\cdot 1 = 4. \end{align*}
Hence
\begin{equation*} u_{3} = x^{2} - \dfrac{5}{3}\cdot 1 - \dfrac{4}{2}\,(x-1) = x^{2} - \dfrac{5}{3} - 2x + 2 = x^{2} - 2x + \dfrac{1}{3}. \end{equation*}
\begin{equation*} \end{equation*}
Thus we have produced an orthogonal set \(\{u_{1},u_{2},u_{3}\}\) where
\begin{equation*} u_{1} = 1,\qquad u_{2} = x - 1,\qquad u_{3} = x^{2} - 2x + \tfrac{1}{3}. \end{equation*}
It is easy to see that
\begin{equation*} \innprod{u_{3}}{u_{3}} = \dfrac{2}{3}. \end{equation*}
Since
\begin{equation*} \norm{u_{1}} = \sqrt{3},\quad \norm{u_{2}} = \sqrt{2},\quad \norm{u_{3}} = \sqrt{\tfrac{2}{3}}, \end{equation*}
we have
\begin{equation*} e_{1} = \dfrac{1}{\sqrt{3}},\quad e_{2} = \dfrac{x-1}{\sqrt{2}},\quad e_{3} = \dfrac{x^{2}-2x+\tfrac{1}{3}}{\sqrt{2/3}} = \sqrt{\tfrac{3}{2}}\bigl(x^{2}-2x+\tfrac{1}{3}\bigr). \end{equation*}
The set \(\{e_{1},e_{2},e_{3}\}\) is an orthonormal basis of \(\operatorname{span}\{1,x,x^{2}\}=\mathcal{P}_2(\mathbb{R})\) (with respect to the discrete inner product above).
🔗
Let us do this using SageMath as well by on \(\mathcal{P}_4(\mathbb{R})\)
🔗
🔗

Checkpoint 7.1.29.

Let \(\beta=\{u_1,\ldots, u_n\}\) be an orthogonal basis of an inner product space \(V\text{.}\) Let \(x\) and \(y\) be two vectors such that \(x=\sum x_i u_i\) and \(y=\sum y_i u_i\text{.}\) Then
\begin{equation*} \inner{x}{y}=x_1y_1+x_2y_2+\cdots+x_ny_n \end{equation*}
and
\begin{equation*} \norm{x}^2=x_1^2+x_2^2+\cdots +x_n^2\text{.} \end{equation*}
🔗
🔗

Example 7.1.30. Legendre Polynomials.

Consider \(V ={\cal P}_3(\R)\) with inner product \(\inner{p}{q}:=\int_{-1}^1 p(x)q(x)\,dx\text{.}\) Use the standard basis \(\beta =\{v_1,v_2,v_3,v_4\} = \{1,x,x^2,x^3\}\) to find an orthogonal basis of \({\cal P}_3(\R)\text{.}\)
🔗
First of all notice that \(\beta\) is not an orthogonal basis. For \(\inner{v_1}{v_3}=\inner{1}{x^2} = \int_{-1}^1 x^2 dx = \frac23\text{,}\) \(\inner{v_2}{v_4}=\int_{-1}^1 x^4 dx = \frac25\text{.}\) Also note that \(\inner{v_1}{v_2}=\int_{-1}^1 xdx = 0\text{.}\) \(\inner{v_2}{v_3}=\int_{-1}^1 x^3dx = 0\text{.}\) \(\inner{v_1}{v_4}=\int_{-1}^1 x^3 dx = 0\text{.}\) \(\inner{v_3}{v_4}=\int_{-1}^1 x^5dx = 0\text{.}\)
🔗
Since \(v_1\) and \(v_2\) are already orthogonal, we can choose \(u_1=v_1=1\) and \(u_2 = v_2=x\) in the Gram-Schmidt process. Next
\begin{equation*} u_3 = v_3-\inner{v_3}{u_1}/\norm{u_1}^2u_1-\inner{v_3}{u_2}/\norm{u_2}^2u_2x^2\text{.} \end{equation*}
🔗
We have
\begin{equation*} \inner{v_3}{u_1}= \int_{-1}^1 x^2 dx = \frac23, \inner{u_1}{u_1}=\int_{-1}^1 1dx = 2\text{.} \end{equation*}
🔗
Hence
\begin{equation*} u_3 = x^2 - \frac{\frac23}{2} \times 1 = x^2-\frac13\text{.} \end{equation*}
🔗
Next
\begin{align*} u_4 \amp = u_4-\inner{v_4}{u_1}/\norm{u_1}^1u_1-\inner{v_4}{u_2}/\norm{u_2}^1u_2-\inner{v_4}{u_3}/\norm{u_3}^1u_3\\ \amp =v_4-\inner{v_4}{u_2}/\norm{u_2}^1u_2-\inner{v_4}{u_3}/\norm{u_3}^1u_3\\ \amp = x^3-\frac35 x\text{.} \end{align*}
🔗
Hence an orthogonal basis is \(\beta'=\{1,x,x^2-\frac{1}{3}, x^3-\frac35 x \}\text{.}\) These are the first four Legendre polynomials.
🔗
After normalizing the vectors, we get an orthonormal basis
\begin{equation*} \beta''=\left\{\frac{\sqrt{2}}{2}, \frac{\sqrt{6}}{2}x, \frac{3\sqrt{10}}{4}\left(x^2-\frac13\right), \frac{5\sqrt{14}}{4}\left(x^3-\frac35 x\right)\right\}\text{.} \end{equation*}
🔗
Figure 7.1.31. Graph of Legendre polynomials
🔗
🔗

Checkpoint 7.1.32.

Consider the standard basis \(\beta=\{1,x,x^2,x^3\}\) of \({\cal P}_3(\R)\) with inner product \(\inner{f}{g}:=\int_0^1 f(x)g(x)\,dx\text{.}\) Find an orthonormal basis starting with \(\beta\) using the Gram-Schmidt orthogonalization process.
🔗
Hint.
🔗
🔗
🔗

Example 7.1.33.

Let \(A=\left(\begin{array}{rrr}2 \amp -1 \amp 0 \\-1 \amp 2 \amp -1 \\0 \amp -1 \amp 2 \end{array} \right)\text{.}\) It is easy to check that \(A\) is a symmetric and positive definite matrix. (why?) Define an inner product on \(\mathbb{R}^3\) as \(\inner{u}{v}:=v^TAu\text{.}\)
🔗
Use the the Gram-Schmidt orthogonalization process to find an orthonormal basis of from the standard basis vectors \(\beta=\{e_1, e_2, e_3\}\) with respect to the above inner product.
\begin{align*} u_1: =\amp (1,0,0)\\ u_2: =\amp e_2-\frac{\inner{u_2\cdot e_2}{u_1}}{\norm{u_1}^2}u_1=(0,1,0)-\frac{-1}{2}(1,0,0)=(1/2,1,0)\\ u_3: =\amp e_3 -\frac{\inner{e_3}{u_1}}{\norm{u_1}^2}u_1-\frac{\inner{e_3}{u_1}}{\norm{u_2}^2}u_2\\ = \amp (0,0,1)-\frac{0}{2}(1,0,0)-\frac{-1}{3/2}(1/2,1,0)=(1/3, 2/3, 1) \end{align*}
🔗
Hint.
🔗
🔗

Example 7.1.34. Lagrange Interpolating Polynomials.

Fix any \(n+1\) distinct real numbers, \(c_0,c_1,\ldots, c_n\) and define
\begin{equation*} \langle p,q \rangle := p(c_0)q(c_0)+p(c_1)q(c_1)+\cdots +p(c_n)q(c_n)= \sum_{j=0}^n p(c_j)q(c_j) \end{equation*}
an inner product on \({\cal P}_n(\R)\text{.}\)
🔗
Recall the Lagrange Polynomial defined (Eqn. (4.5.1)).
\begin{equation*} \ell_i(x)=\prod_{j=0,j\neq i}^{n}\frac{x-c_j}{c_i-c_j}\text{.} \end{equation*}
🔗
Then
\begin{equation*} \inner{\ell_i}{\ell_j}=\sum_{k=0}^n \ell_i(c_k)\ell_j(c_k)=\delta_{ij}. \end{equation*}
🔗
Hence \(\{\ell_i\}\) is an orthonormal basis of \({\cal P}_n(\R)\text{.}\)
🔗
Let \(p(x)\in {\cal P}_n(\R)\) be any polynomial, then
\begin{equation*} \inner{p}{\ell_k}=p(c_0)\ell_k(c_0)+p(c_1)\ell_k(c_1)+\cdots +p(c_n)\ell_k(c_n)=p(c_k)\text{.} \end{equation*}
Since \(\{\ell_i\}\) is an orthonormal basis of \({\cal P}_n(\R)\text{,}\) we have
\begin{gather*} p(x) = \inner{p(x)}{\ell_0(x)}\ell_0(x)+\inner{p(x)}{\ell_1(x)}\ell_1(x)+\cdots +\inner{p(x)}{\ell_n(x)}\ell_n(x)\\ = p(c_0)\ell_0(x)+p(c_1)\ell_1(x)+p(c_2)\ell_2(x)+\cdots +p(c_n)\ell_n(x) \end{gather*}
which is the Lagrange interpolation expansion of \(p(x)\text{.}\)
🔗
🔗
How will you define the othogonal projection of a vector onto another vector in an inner product space?
🔗

Activity 7.1.1.

Notion of orthogonal projection and reflection in an inner product space.
🔗
Note that the concepts of Gram–Schmidt orthogonalization, orthogonal projection, and reflection can be naturally extended to an inner product space \((V, \langle \cdot, \cdot \rangle)\text{.}\) Explore how these notions generalize in such spaces, and implement solutions to related problems using Sage.
🔗
🔗

Definition 7.1.35.

Let \((V,\innprod{.}{.})\) be an inner product space. Suppose \(u,v\in V\) with \(u\neq 0\text{.}\) Then the orthogonal projection of \(v\) onto \(u\) is defined as
\begin{equation*} \proj_u(v):= \frac{\inner{v}{u}}{\inner{u}{u}}u. \end{equation*}
🔗
🔗

Checkpoint 7.1.36.

Let \(V=\mathcal{P}_2(\R)\) with inner product \(\inner{p}{q}=\int_0^1 p(x)q(x)dx\text{.}\) Find the orthogonal projection of \(p(x)=x^2\) onto \(u(x)=x+1\text{.}\)
🔗
🔗

Definition 7.1.37. Projection onto a subspace.

Let \(V\) be an inner product space and \(W\leq V\text{,}\) a finite dimensional subspace of \(V\text{.}\) Let \(\{u_1,\ldots, u_k\}\) be an orthonormal basis of \(W\text{.}\) Suppose \(v\in V\text{.}\) Similar to definition 6.3.5, we can define the orthogonal projection of \(v\) onto \(W\) as
\begin{equation*} \proj_W(v):= \inner{v}{u_1}u_1+\inner{v}{u_2}u_2+\cdots+\inner{v}{u_k}u_k\text{.} \end{equation*}
🔗
🔗

Checkpoint 7.1.38.

Find the orthogonal projection of vector \(b=\begin{bmatrix}1\\2\\3\\4 \end{bmatrix}\) onto the subspace spanned by three vectors \(\left\{\begin{bmatrix}1\\-1\\0\\1 \end{bmatrix} , \begin{bmatrix}0\\1\\1\\-1 \end{bmatrix} , \begin{bmatrix}1\\1\\-1\\0 \end{bmatrix} \right\}\text{.}\)
🔗
🔗

Definition 7.1.39.

Let \((V,\innprod{.}{.})\) be an inner product space over \(\R\) and \(U\subset V\text{.}\) The the orthogonal complement of \(U\) in \(V\) is defined as
\begin{equation*} U^{\perp}=\{v\in V: \innprod{u}{v}=0 \text{ for all } u\in U\}. \end{equation*}
🔗
🔗

Checkpoint 7.1.40.

Let \((V,\innprod{.}{.})\) be an inner product space over \(\R\) and \(U\subset V\text{.}\) Show that \(U^{\perp}\) is a subspace of \(V\text{.}\)
🔗
🔗

Example 7.1.41.

Let \(A\) be a symmetric \(n\times n\) and positive definite matrix. Define an inner product on \(\R^n\) as \(\innprod{u}{v}=v^TAu\text{.}\) Let \(U=\{e_1\}\text{.}\) Then
\begin{equation*} U^{\perp}=\{v\in \R^n: \innprod{e_1}{v}=0\}=\{v\in \R^n: v^TAe_1=0\}. \end{equation*}
Since \(e_1=(1,0,\ldots,0)^T\text{,}\) \(Ae_1\) is the first column of \(A\text{.}\) This implies that \(U^{\perp}\) is the set of vector orthogonal to the first column of \(A\) with respect to the standard inner product. Since the first column of \(A\) is nonzero, \(\dim(U^{\perp})=n-1\text{.}\) It is easy to see that \(U^{\perp}\) is the set of vector orthogonal to the first column of \(A\) with respect to the standard inner product.
🔗
🔗

Checkpoint 7.1.42.

Let \({\cal P}_2(\R)\) with discrete inner product
\begin{equation*} \innprod{p}{q}=p(0)q(0)+p(1)q(1)+p(2)q(2). \end{equation*}
Find \(\{1\}^{\perp}\) and \(\{x\}^{\perp}\text{.}\)
🔗
🔗

Proof.

(1) \(W^{\perp}\) is subspace follows from Checkpoint 7.1.40.
🔗
Let \(u\in W\cap W^{\perp}\text{.}\) Then \(\innprod{u}{u}=0\text{.}\) Hence the only vector in both \(W\) and \(W^{\perp}\) is the zero vector. Let \(x\in V\text{.}\) We need to show that there exist \(w\in W\) and \(v\in W^{\perp}\) such that \(x=w+v\text{.}\) Let \(\{u_1,\ldots, u_k\}\) be an orthonormal basis of \(W\text{.}\) Define
\begin{equation*} w = \innprod{x}{u_1}u_1+\innprod{x}{u_2}u_2+\cdots +\innprod{x}{u_k}u_k. \end{equation*}
Now define \(v=x-w\text{.}\) For each \(i=1,2,\ldots, k\text{,}\) we have
\begin{align*} \innprod{v}{u_i} \amp = \innprod{x-w}{u_i} \\ \amp = \innprod{x}{u_i}-\innprod{w}{u_i} \\ \amp = \innprod{x}{u_i}-\inner{x}{u_i}\underbrace{\innprod{u_i}{u_i}}_{1} - \sum_{j\neq i}\inner{x}{u_j}\underbrace{\innprod{u_j}{u_i}}_{0} \\ \amp = 0. \end{align*}
Hence \(v\in W^{\perp}\) and the result follows.
🔗
(2) Follows from (1).
🔗
(3) Let \(W^{\perp} =U\text{.}\) Then by (2), we have
\begin{equation*} \dim(V)=\dim(W)+\dim(U). \end{equation*}
Similarly,
\begin{equation*} \dim(V)=\dim(U^{\perp})+\dim(U). \end{equation*}
Hence \(\dim(W)=\dim(U^{\perp})\text{.}\) It remains to show that \(W\subseteq U^{\perp}\) and \(U^{\perp}\subseteq W\text{.}\) Let \(w\in W\) and \(u\in U\text{.}\) Then \(\innprod{w}{u}=0\) since \(U=W^{\perp}\text{.}\) This shows that \(W\subseteq U^{\perp}\text{.}\) Now let \(y\in U^{\perp}\text{.}\) We need to show that \(y\in W\text{.}\) By (1), there exist \(w\in W\) and \(u\in U\) such that \(y=w+u\text{.}\)
🔗
🔗
If \(W\) is a finite dimensional subspace of an inner product space \(V\text{.}\) Then the subspace \(W^{\perp}\) is called the orthogonal complement of \(W\) in \(V\text{.}\) Since \(V=W\oplus W^{\perp}\text{,}\) every vector \(x\in V\) can be uniquely written as \(x=w+v\) with \(w\in W\) and \(v\in W^{\perp}\text{.}\) The vector \(w\) is called the orthogonal projection of \(x\) onto \(W\) and is denoted by \(\proj_W(x)\text{.}\)
🔗
Futhermore, if \(\{u_1,\ldots, u_k\}\) is an orthonormal basis of \(W\text{,}\) then
\begin{equation*} \proj_W(x)=\innprod{x}{u_1}u_1+\innprod{x}{u_2}u_2+\cdots +\innprod{x}{u_k}u_k. \end{equation*}
🔗
The following theorem shows that the orthogonal projection is the best approximation of a vector in \(V\) by a vector in \(W\text{.}\)
🔗

Proof.

Let \(u\in W\text{.}\) Then \(u-w\in W\text{.}\) Since \(x-w\in W^{\perp}\text{,}\) we have \(\innprod{x-w}{u-w}=0\text{.}\) Hence by the Pythagorean theorem,
\begin{align*} \norm{x-u}^2\amp =\norm{(x-w)+(w-u)}^2\\ \amp =\norm{x-w}^2+\norm{u-w}^2\geq \norm{x-w}^2. \end{align*}
The result follows.
🔗
🔗
The Approximation Theorem shows that the orthogonal projection of a vector \(x\) onto a subspace \(W\) is the best approximation of \(x\) by a vector in \(W\text{.}\) This result has important applications in numerical analysis and scientific computing.
🔗

Example 7.1.45. Fouries Coefficients.

Let us explore one such application from Fourier analysis. Define an inner product on the space of continuous real-valued functions on \([-\pi, \pi]\) as
\begin{equation*} \innprod{f}{g}=\frac{1}{\pi}\int_{-\pi}^{\pi} f(x)g(x) dx. \end{equation*}
Consider the subspace
\begin{equation*} F_n=\text{span}\{1, \cos x, \sin x, \cos 2x, \sin 2x, \ldots, \cos nx, \sin nx\}. \end{equation*}
It is easy to check that the set \(F_n\) is orthgonal with respect to the above inner product. Also
\begin{align*} \norm{1}^2\amp =\innprod{1}{1}=2\pi,\\ \norm{\cos kx}^2\amp =\frac{1}{\pi}\int_{-\pi}^{\pi} \cos^2{kx}\,dx=\pi,\\ \norm{\sin kx}^2\amp =\frac{1}{\pi}\int_{-\pi}^{\pi} \sin^2{kx}\,dx=\pi, \end{align*}
for \(k=1,\ldots, n\text{.}\) Hence an orthonormal basis of \(F_n\) is given by
\begin{equation*} \left\{\frac{1}{\sqrt{2\pi}}, \frac{\cos x}{\sqrt{\pi}}, \frac{\sin x}{\sqrt{\pi}}, \ldots, \frac{\cos nx}{\sqrt{\pi}}, \frac{\sin nx}{\sqrt{\pi}}\right\}. \end{equation*}
Given a continuous function \(f\colon [-\pi, \pi]\to \R\text{,}\) we want to find the best approximation of \(f\text{.}\) Define
\begin{align*} a_0 \amp =\frac{1}{2\pi}\int_{-\pi}^{\pi} f(x)\,dx\\ a_k \amp =\frac{1}{2\pi}\int_{-\pi}^{\pi} f(x)\cos kx \,dx\\ b_k \amp =\frac{1}{2\pi}\int_{-\pi}^{\pi} f(x)\sin kx \,dx \end{align*}
Then by the Approximation Theorem, the orthogonal projection of \(f\) onto \(F_n\) is given by
\begin{equation*} \proj_{F_n}(f)=a_0+\sum_{k=1}^n \left(a_k \cos kx + b_k \sin kx\right). \end{equation*}
The coefficients \(a_0, a_k, b_k\) are called the Fourier coefficients of \(f\text{.}\) Thus we have shown that the Fourier coefficients give the best approximation of a function by a trigonometric polynomial of degree at most \(n\)
\begin{equation*} f(x)\approx a_0+\sum_{k=1}^n \left(a_k \cos kx + b_k \sin kx\right) \end{equation*}
🔗
🔗

Subsection 7.1.1 Linear Functionals

Let \((V,\innprod{.}{.})\) be an inner product space over \(\R\text{.}\) Fix a vector \(a\in V\) and define \(f_a\colon V\to \R\) as \(f_a(x)=\innprod{x}{a}\text{.}\) It follows from from properties of inner product that \(f_a\) is a linear map. (why?)
🔗

Definition 7.1.46.

Let \(\innprod{.}{.}\) be an inner product space over \(\R\text{.}\) Any liear map \(f\colon V \to \R\) is called a linear function.
🔗
🔗
Recall that we have characterized all linear maps from \(\R^n\to \R\text{.}\) In particular, if \(T\colon \R^n\to \R\) is a linear map, them there exist a vector \(a=(a_1,\ldots, a_n)\) such that
\begin{equation*} T(x)=\sum a_ix_i=x \cdot a. \end{equation*}
The question is can we characterize linear functionals on an inner product space.
🔗

Proof.

Let \(\{u_1,\ldots, u_n\}\) be an orthonormal basis of \(V\text{.}\) Define
\begin{equation*} y = f(u_1)u_1+f(u_2)u_2+\cdots +f(u_n)u_n. \end{equation*}
For any \(x\in V\text{,}\) we can write \(x=c_1u_1+c_2u_2+\cdots +c_nu_n\text{.}\) Hence
\begin{align*} \innprod{x}{y} \amp = \innprod{c_1u_1+c_2u_2+\cdots +c_nu_n}{f(u_1)u_1+f(u_2)u_2+\cdots +f(u_n)u_n} \\ \amp = c_1f(u_1)\underbrace{\innprod{u_1}{u_1}}_{1}+c_2f(u_2)\underbrace{\innprod{u_2}{u_2}}_{1}+\cdots +c_nf(u_n)\underbrace{\innprod{u_n}{u_n}}_{1} \\ \amp = c_1f(u_1)+c_2f(u_2)+\cdots +c_nf(u_n). \end{align*}
Since \(f\) is linear, we have
\begin{align*} f(x) \amp =f(c_1u_1+c_2u_2+\cdots +c_nu_n)\\ \amp =c_1f(u_1)+c_2f(u_2)+\cdots +c_nf(u_n). \end{align*}
Hence \(f(x)=\innprod{x}{y}\text{.}\)
🔗
To show uniqueness, suppose there exists \(z\in V\) such that \(f(x)=\innprod{x}{z}\) for all \(x\in V\text{.}\) Then for each \(i=1,2,\ldots, n\text{,}\) we have
\begin{equation*} f(u_i)=\innprod{u_i}{y}=\innprod{u_i}{z}. \end{equation*}
Hence
\begin{equation*} \innprod{u_i}{y-z}=0 \end{equation*}
for all \(i=1,2,\ldots, n\text{.}\) Since \(\{u_1,\ldots, u_n\}\) is a basis of \(V\text{,}\) we have \(y-z=0\) or \(y=z\text{.}\)
🔗
🔗
Can we characterize linear maps from a finite dimesional inner product space \(V\) to a finite dimensional inner product space \(W\text{?}\)
🔗

Proof.

Sine \(\alpha=\{u_1,\ldots,u_n\}\) is an orthonormal basis of \(V\text{,}\)
\begin{equation*} T(u_j)=\sum_i\innprod{T(u_i)}{v_i}v_i. \end{equation*}
This shows that the \(j\)-th column of \([T]_\alpha^\beta\) is
\begin{equation*} \begin{bmatrix} \innprod{T(u_j)}{v_1} \\ \innprod{T(u_j)}{v_2} \\ \vdots \\ \innprod{T(u_j)}{v_m} \end{bmatrix}. \end{equation*}
In paticular, the \((i,j)\)-th entry of \([T]_\alpha^\beta\) is
\begin{equation*} \innprod{T(u_j)}{v_i}. \end{equation*}
🔗
🔗
The above theorem shows that a linear map between two finite dimensional inner product spaces is completely determined by the images of an orthonormal basis of the domain space.
🔗
Now suppose \(W=V\) in Theorem 7.1.48 and \(T\colon V\to V\) be a linear map on \(V\text{.}\) Let \(\alpha=\{u_1,\ldots, u_n\}\) be an orthonormal basis of \(V\text{.}\) Then the matrix of \(T\) with respect to this basis is given by
\begin{equation*} [T]_\alpha=\begin{bmatrix} \innprod{T(u_1)}{u_1} \amp \innprod{T(u_2)}{u_1} \amp \cdots \amp \innprod{T(u_n)}{u_1} \\ \innprod{T(u_1)}{u_2} \amp \innprod{T(u_2)}{u_2} \amp \cdots \amp \innprod{T(u_n)}{u_2} \\ \vdots \amp \vdots \amp \ddots \amp \vdots \\ \innprod{T(u_1)}{u_n} \amp \innprod{T(u_2)}{u_n} \amp \cdots \amp \innprod{T(u_n)}{u_n} \end{bmatrix}. \end{equation*}
🔗
Using the properties of inner product, it is easy to see that \([T]_\alpha\) is a symmetric matrix if and only if
\begin{equation*} \innprod{T(u_i)}{u_j}=\innprod{T(u_j)}{u_i}=\innprod{u_i}{T(u_j)} \end{equation*}
for all \(i,j\text{.}\) This motivates the following definition.
🔗

Definition 7.1.49.

A linear map \(T\colon V\to V\) on an inner product space \((V,\innprod{.}{.})\) is called symmetric if
\begin{equation*} \innprod{T(u)}{v}=\innprod{u}{T(v)} \end{equation*}
for all \(u,v\in V\text{.}\)
🔗
🔗
Recall that we defined a linear map \(T\colon \R^n\to \R^n\) assocciated with an \(n\times n\) real matrix \(A\) as \(T_A(x)=Ax\text{.}\) Also the matrix of \(T_A\) with respect to the standard basis is \(A\text{.}\) This we can define an \(n\times n\) matrix \(A\) to be symmetric if the linear map \(T_A\colon \R^n\to \R^n\) is symmetric with respect to the standard inner product (dot product) on \(\R^n\text{.}\)
🔗
It is easy to see that an \(n\times n\) real matrix \(A\) is symmetric in the usual sense (i.e., \(A=A^T\)) if and only if the linear map \(T_A\colon \R^n\to \R^n\) is symmetric with respect to the standard inner product on \(\R^n\text{.}\)
🔗
Next we revist the orthogonal linear transformation defined defined in the last chapter and give an equivalent definition using inner products. We have the following equivalent definition.
🔗

Definition 7.1.50.

Let \((V,\innprod{.}{.})\) be a finite dimensional inner product space over \(\R\text{.}\) A linear map \(T\colon V\to V\) is an orthogonal linear transformation if and only if
\begin{equation*} \innprod{T(u)}{T(v)}=\innprod{u}{v} \end{equation*}
for all \(u,v\in V\text{.}\)
🔗
🔗
In particular orthogonal linear transformations are precisely those linear maps that preserve the inner product. It is easy to see that a linear map \(T\colon V\to V\) is an orthogonal linear transformation if and only if
\begin{equation*} \norm{T(u)}=\norm{u}. \end{equation*}
We have the following equivalent characterization of orthogonal linear transformations.
🔗

Proof.

(1) \(\Rightarrow\) (2): Follows from the definition of orthogonal linear transformation.
🔗
(2) \(\Rightarrow\) (3): Let \(\{u_1,\ldots, u_n\}\) be an orthonormal basis of \(V\text{.}\) For each \(i,j=1,2,\ldots, n\text{,}\) we have
\begin{align*} \innprod{T(u_i)}{T(u_j)} \amp = \frac{1}{2}(\norm{T(u_i)+T(u_j)}^2-\norm{T(u_i)}^2-\norm{T(u_j)}^2) \\ \amp = \frac{1}{2}(\norm{u_i+u_j}^2-\norm{u_i}^2-\norm{u_j}^2) \\ \amp = \innprod{u_i}{u_j} \end{align*}
This shows that \(\{T(u_1),\ldots, T(u_n)\}\) is an orthonormal set. Since \(T\) is a linear map, \(\{T(u_1),\ldots, T(u_n)\}\) spans \(\im(T)\text{.}\) Also \(\{T(u_1),\ldots, T(u_n)\}\) is linearly independent. Hence
\begin{equation*} n=\dim(\im(T))\leq \dim(V)=n. \end{equation*}
This implies \(\im(T)=V\text{.}\) This shows that \(\{T(u_1),\ldots, T(u_n)\}\) is an orthonormal basis of \(V\text{.}\)
🔗
(3) \(\Rightarrow\) (1): Let \(u,v\in V\text{.}\) Since \(\{u_1,\ldots, u_n\}\) is an orthonormal basis of \(V\text{,}\) we can write \(u=c_1u_1+c_2 u_2+\cdots +c_nu_n\) and \(v=d_1u_1+d_2 u_2+\cdots +d_nu_n\text{.}\) Then it is easy to see that
\begin{equation*} \innprod{u}{v}=\sum c_j d_i. \end{equation*}
Hence
\begin{align*} \innprod{T(u)}{T(v)} \amp = \innprod{\sum c_iT(u_i)}{\sum d_jT(u_j)} \\ \amp = \sum_i\sum_j c_id_j\innprod{T(u_i)}{T(u_j)} \\ \amp = \sum c_id_i=\innprod{u}{v}. \end{align*}
🔗
🔗
Now let us see what happens to the matrix of an orthogonal linear transformation with respect to an orthonormal basis. Let \(T\colon V\to V\) be an orthogonal linear transformation on a finite dimensional inner product space \((V,\innprod{.}{.})\text{.}\) Let \(\alpha=\{u_1,\ldots, u_n\}\) be an orthonormal basis of \(V\text{.}\) Then the matrix of \(T\) with respect to this basis is given by
\begin{equation*} [T]_\alpha=\begin{bmatrix} \innprod{T(u_1)}{u_1} \amp \innprod{T(u_2)}{u_1} \amp \cdots \amp \innprod{T(u_n)}{u_1} \\ \innprod{T(u_1)}{u_2} \amp \innprod{T(u_2)}{u_2} \amp \cdots \amp \innprod{T(u_n)}{u_2} \\ \vdots \amp \vdots \amp \ddots \amp \vdots \\ \innprod{T(u_1)}{u_n} \amp \innprod{T(u_2)}{u_n} \amp \cdots \amp \innprod{T(u_n)}{u_n} \end{bmatrix}=\begin{bmatrix} a_{11} \amp a_{21} \amp \cdots \amp a_{n1} \\ a_{21} \amp a_{22} \amp \cdots \amp a_{21} \\ \vdots \amp \vdots \amp \ddots \amp \vdots \\ a_{n1} \amp a_{n2}\amp \cdots \amp a_{nn} \end{bmatrix} =A. \end{equation*}
Let us write the columns of \(A\) as \(A=[c_1 \; c_2 \; \cdots \; c_n]\text{.}\) Since \(\{T(u_1),\ldots, T(u_n)\}\) is an orthonormal basis of \(V\text{,}\) it follows that
\begin{align*} \delta_{ij} \amp=\innprod{T(u_i)}{T(u_j)} \\ \amp=\innprod{\sum_k a_{ki}u_k}{\sum_s a_{sj}u_s} \\ \amp=\sum_k\sum_s a_{ki}a_{sj}=c_i\cdot c_j \end{align*}
the dot product of \(c_i\) and \(c_j\text{.}\) This shows that the columns of \(A\) form an orthonormal set in \(\R^n\text{.}\) This suggest the following definition of an othogonal matrix.
🔗

Definition 7.1.52.

A real \(n\times n\) matrix \(A\) is called an orthogonal matrix if its columns form an orthonormal set in \(\R^n\text{.}\) This is, if \(A=[c_1 \; c_2 \; \cdots \; c_n]\text{,}\) then
\begin{equation*} c_i \cdot c_j = \delta_{ij} \end{equation*}
for all \(i,j=1,2,\ldots, n\text{.}\) Equivalently, \(A\) is an orthogonal matrix if \(A^TA=I_n\text{.}\)
🔗
🔗
🔗
🔗
PrevTopNext