Inner Product

Section 7.1 Inner Product

In the last section we dealt with notion of dot product and geometry in \(\R^n\text{.}\) The dot product and related notion can be generalized to an arbitrary vector space over \(\R\) or \(\mathbb{C}\text{.}\) All the notions, we have learned in the last section can be generalized over an inner product space.

Note that the dot product of two vectors in \(\R^n\) is a scalar, in particular, dot product can be thought of as a function from \('\cdot' \colon \R^n\times \R^n \to\R\) satisfying the following properties:

\(x \cdot x\geq 0\) for all \(x\in \R^n\text{.}\)
\(x \cdot x= 0\) if and only if \(x=0\text{.}\)
\(x \cdot y=y\cdot x\) for all \(x,y \in \R^n\text{.}\)
\(x\cdot (y+z) = x\cdot y +x\cdot z\) for all \(x,y,z\in \R^n\text{.}\)
\(x\cdot (\alpha y)=\alpha x\cdot y = (\alpha x)\cdot y\text{.}\)

The notion of dot product on \(\R^n\) can ve generalized on vector space known as inner product. We have the following definition.

Definition 7.1.1. Inner Product.

Let \(V\) be a vector space over \(\R\text{.}\) An inner product on \(V\) is a function that assigns a real number \(\langle x, y\rangle\) to every pair \(x,y\) of vectors in \(V\) satisfying the following properties.

\(\langle x, x\rangle \geq 0\) for all \(x \in V\) and \(\langle x, x\rangle = 0\) if and only if \(x=0\text{.}\)
\(\langle x, y\rangle = \langle y, x\rangle\) for all \(x,y \in V\text{.}\) (Symmetry)
\(\langle x, (y+z)\rangle=\langle x, y\rangle+\langle x, z\rangle\) for all \(x,y,z\in V\text{.}\)
\(\langle x, (\alpha y)\rangle=\alpha\langle x, y\rangle\) for all \(x,y\in V\) and \(\alpha \in \R\text{.}\)

If \(V\) is real vector space with inner product \(\langle. , .\rangle\text{.}\) Then \((V, \langle . , .\rangle)\) called in inner product space over \(\R\text{.}\)

The last two properties make the inner product linear in the second variable. Using the symmetry property, it can also be shown that the inner product is linear in the first variable as well. That is,

\begin{equation*} \langle (x+y),z\rangle=\langle x, z\rangle+\langle y, z\rangle, \text{ and } \langle (\alpha x) y\rangle=\alpha\langle x, y\rangle \end{equation*}

Next we look at several examples of inner procuct on various vector spaces that we have defined in Chapter 4.

Example 7.1.2.

On \(\R^n\text{,}\) the standard dot product is an inner product. Thus define

\begin{equation*} \langle x, y\rangle:=x\cdot y \end{equation*}

This is also called the Euclidean inner product on \(\R^n\text{.}\)

Example 7.1.3.

Let \(V=M_n(\R)\text{,}\) the set of all \(n\times n\) matrices over \(\R\text{.}\) Define

\begin{equation*} \langle A, B\rangle:=tr(AB^T) \end{equation*}

It is easy to show that this is an inner product on \(M_n(\R)\text{.}\)

Note that this inner product can be thought of as the standard dot product on \(\R^{n^2}\text{.}\) The elements of the matrix \(A\) can be thought of as a vector in \(\R^{n^2}\text{.}\) Then

\begin{equation*} tr(AB^T) =\sum\sum a_{ij}b_{ij}\quad \quad \text{ why?}\text{.} \end{equation*}

Work with \(2\times 2\) matrices and then try to prove this for \(n\times n\) matrices.

Example 7.1.4.

Let \(A\) be an \(n\times n\) symmetric and positive definite matrix. On \(\R^n\text{,}\) define

\begin{equation*} \langle x, y \rangle := x\cdot Ay = x^TAy \end{equation*}

Then this is an inner product on \(\R^n\text{.}\)

Example 7.1.5.

Let \(V={\cal C}([0,1])\) the set of all continuous function from \([0,1]\) to \(\R\text{.}\) Define

\begin{equation*} \langle f, g \rangle:=\int_0^1 f(t)g(t)dt\text{.} \end{equation*}

This is an inner product on \(V\text{.}\) (You may need real analysis to conclude that \(\int_0^1 f(t)f(t)dt=\) then \(f=0\text{.}\))

Example 7.1.6.

Let \(p\) and \(q\) be two polynomials in \({\cal P}_n(\mathbb{R})\text{.}\) Then define

\begin{equation*} \langle p,q \rangle := p(0)q(0)+p(1)q(1)+\cdots +p(n)q(n)\text{.} \end{equation*}

It is easy to see that \(\langle p,q \rangle\) defined inner product on the vector space \({\cal P}_n(R)\text{.}\)

Here \(0, 1, 2, \ldots, n\) are nothing special. Instead, we can use any \(n+1\) distinct real numbers, \(c_0,c_1,\ldots, c_n\text{.}\)

Hint.

\(\langle p,p \rangle =\) mean that \(p\) has \(n+1\) roots, which is not possible unless \(p=0\) by the fundamental theorem of algebra.

Definition 7.1.7.

Let \((V, \langle .\rangle)\) be a real inner product space. Then norm of any vector \(x\in V\) corresponding to the inner product \(\langle . \rangle\) is defined as

\begin{equation*} \norm{x}=\sqrt{\langle x, x\rangle}\text{.} \end{equation*}

The distance between twp vectors \(x\) and \(y\) is defined as

\begin{equation*} d(x,y):=\norm{x-y}\text{.} \end{equation*}

Checkpoint 7.1.8.

Let \(V\) be an inner product space. Then for any two vectors \(x, y\in V\text{,}\) show that

1. \(\norm{x+y}^2=\norm{x}^2+\norm{y}^2+2\inner{x}{y}\)

2. \(\norm{x-y}^2=\norm{x}^2+\norm{y}^2-2\inner{x}{y}\)

Hint.

The proof follows by looking at \(\langle x+y,x+y \rangle \) and \(\langle x-y,x-y \rangle \) and using the properties of inner product.

Checkpoint 7.1.9.

If \(x, y\) are two vectors in an inner product space \(V\) with inner product \(\langle .\rangle\text{.}\) Then show that

\begin{equation*} \norm{x+y}^2+\norm{x-y}^2=2(\norm{x}^2+\norm{y}^2)\text{.} \end{equation*}

This is called the parallelogram identity. Geometrically, in a parallelogram, the sum of square of the diagonals is 2 times the sum of the squares of the side lengths.

Hint.

Proof follows from Checkpoint 7.1.8.

Theorem 7.1.10. Cauchy-Schwarz Inequality.

Let \(V\) be an inner product space. The for any two vectors \(x,y\in V\text{,}\) we have

\begin{equation} |\inner{x}{y}|\leq \norm{x}\norm{y}\tag{7.1.1} \end{equation}

The equality holds if and only if \(x\) and \(y\) are linearly dependent.

Theorem 7.1.11. Triangle Inequality.

Let \(x\) and \(y\) be two vectors in an inner product space \(V\text{.}\) Then

\begin{align*} \norm{x+y}^2 \amp = \norm{x+y}^2=\norm{x}^2+\norm{y}^2+2\inner{x}{y}\\ \amp \leq \norm{x}^2+\norm{y}^2+2|\inner{x}{y}| \text{ \(a\leq |a|\) }\\ \amp \leq \norm{x}^2+\norm{y}^2+2\norm{x}\norm{y} \text{ by the Cauchy-Schwarz inequality }\\ \amp = (\norm{x}+\norm{y})^2 \end{align*}

Hence for all \(x,y\text{,}\) we have

\begin{gather} \norm{x+y}\leq \norm{x}+\norm{y}\tag{7.1.2} \end{gather}

is called the triangle inequality.

Let us play with the Cauchy-Schwarz inequality (7.1.1). Suppose \(x\) and \(y\) are non zero vectors in \(V\text{,}\) Then

\begin{equation*} |\inner{x}{y}|\leq \norm{x}\norm{y}\Rightarrow \frac{|\inner{x}{y}|}{\norm{x}\norm{y}}\leq 1 \end{equation*}

Hence we have

\begin{equation*} -1\leq \frac{\inner{x}{y}}{\norm{x}\norm{y}}\leq 1\text{.} \end{equation*}

Thus for any two non zero vectors, \(x\) and \(y\text{,}\) \(\frac{\inner{x}{y}}{\norm{x}\norm{y}}\) always lies between \(-1\) and 1. This allows us to define the angle between two non zero vectors. We assign this number to \(\cos\theta\) with \(\theta\in[-\pi,\pi]\) called the angle between \(x\) and \(y\text{.}\) Thus, if \(\theta\) is the angle between \(x\) and \(y\text{,}\) then we have

\begin{equation*} \cos \theta = \frac{\inner{x}{y}}{\norm{x}\norm{y}} \end{equation*}

All the notions that we defined for dot product, namely, orthogonality, orthogonal projection, Gram-Schmidt orthogonalization process can we defined in a similar manner. All we need to do is, replace the dot product by the given inner product.

Theorem 7.1.12. Properties of Norm.

Let \((V,\langle ., .\rangle)\) be an innepr product space. The norm defined as Definition 7.1.7 has the following properties:

for all \(x\in V\text{,}\) \(\norm{x}\geq 0\) and \(\norm{x}= 0\) if and only if \(x=0\text{.}\)
for all \(\alpha \in \R\) and \(x\in V\text{,}\) \(\norm{\alpha x}=|\alpha|\norm{x}\text{.}\)
for all \(x,y \in V\text{,}\) \(\norm{x+y}\leq \norm{x}+\norm{y}\text{.}\)

Definition 7.1.13.

Any vector space \(V\) over \(\R\) with a function \(\norm{.} : V \to \R\) which satisfies all the properties mentioned in Theorem 7.1.12 is called a normed linear space.. Thus any inner product space is also a normed linear space.

Note 7.1.14.

The concepts such as orthogonality, orthogonal projection, orthogonal complement of any subset, orthogonal and orthonormal sets and Gram-Schmidt orthogonalization process etc that we defined and dealt with in the previuos chapter with respect to the dot product on \(\R^n\) can be defined on an inner product space. All we need to do is to replace the dot product by the corresponding inner product. We encourage readers to define each one of them.

Checkpoint 7.1.15.

Let \((V, \inner{.}{.})\) be a real inner product space. Define (i) orthogonality of two vectors \(x\) and \(y\) in \(V\text{,}\) (ii) orthogonal complement of a subset \(U\) of \(V\text{,}\) (iii) Orthogonal projection of a vector \(v\) onto a non-zero vector \(u\text{,}\) (iv) orthogonal set and orthonormal sets in \(V\) and (v) Gram-Schmidt orthogonalization process.

Checkpoint 7.1.16.

Let \(x, y\) be two vectors in an inner product space \(V\text{.}\) Then show that

(i) \(x\) and \(y\) are orthogonal if and only if \(\norm{x+y}=\norm{x-y}\text{.}\) (what does it mean geometrically?)

(ii) \(x+y\) and \(x-y\) are orthogonal if and only if \(\norm{x}=\norm{y}\text{.}\)

Checkpoint 7.1.17.

Let \(C([-\pi,\pi])\) be the vectors space of set of continuous functions from \([-\pi,\pi]\) to \(\R\text{.}\) Define the inner product on \(C([-\pi,\pi])\) as

\begin{equation*} \inner{f}{g}:=\int_{-\pi}^\pi f(t) g(t)`dt\text{.} \end{equation*}

Show that under this inner product \(\{1, \sin nx,\cos mx\}\) is an orthogonal set.

Checkpoint 7.1.18. Pythagoras Theorem.

Let \((V, \inner{.}{.})\) be a real inner product space. Let \(x_1,x_2\,x_n\) be \(n\) orthogonal vectors in \(V\text{.}\) Then

\begin{equation*} \norm{x_1+x_2+\cdots+x_n}^2=\norm{x_1}^2+\norm{x_2}^2+\cdots+\norm{x_n}^2\text{.} \end{equation*}

This is called the Pythagoras theorem.

Checkpoint 7.1.19.

Let \(\beta=\{u_1,\ldots, u_n\}\) be an orthogonal basis of an inner product space \(V\text{.}\) Let \(v\in V\) and \(\theta_1,\ldots, \theta_n\) be the angle between \(v\) and \(u_1,\ldots, u_n\text{,}\) respectively. Then

\begin{equation*} \cos\theta_1^2+\cdots+\cos\theta_n^2=1\text{.} \end{equation*}

Here \(\cos\theta_i\) are called the direction cosines of \(v\) corresponding to \(\beta\text{.}\)

Checkpoint 7.1.20.

Let \(\beta=\{u_1,\ldots, u_n\}\) be an orthogonal basis of an inner product space \(V\text{.}\) Let \(x\) and \(y\) be two vectors such that \(x=\sum x_i u_i\) and \(y=\sum y_i u_i\text{.}\) Then

\begin{equation*} \inner{x}{y}=x_1y_1+x_2y_2+\cdots+x_ny_n \end{equation*}

and

\begin{equation*} \norm{x}^2=x_1^2+x_2^2+\cdots +x_n^2\text{.} \end{equation*}

Example 7.1.21. Legendre Polynomials.

Consider \(V ={\cal P}_3(\R)\) with inner product \(\inner{p}{q}:=\int_{-1}^1 p(x)q(x)\,dx\text{.}\) Use the standard basis \(\beta =\{v_1,v_2,v_3,v_4\} = \{1,x,x^2,x^3\}\) to find an orthogonal basis of \({\cal P}_3(\R)\text{.}\)

First of all notice that \(\beta\) is not an orthogonal basis. For \(\inner{v_1}{v_3}=\inner{1}{x^2} = \int_{-1}^1 x^2 dx = \frac23\text{,}\) \(\inner{v_2}{v_4}=\int_{-1}^1 x^4 dx = \frac25\text{.}\) Also note that \(\inner{v_1}{v_2}=\int_{-1}^1 xdx = 0\text{.}\) \(\inner{v_2}{v_3}=\int_{-1}^1 x^3dx = 0\text{.}\) \(\inner{v_1}{v_4}=\int_{-1}^1 x^3 dx = 0\text{.}\) \(\inner{v_3}{v_4}=\int_{-1}^1 x^5dx = 0\text{.}\)

Since \(v_1\) and \(v_2\) are already orthogonal, we can choose \(u_1=v_1=1\) and \(u_2 = v_2=x\) in the Gram-Schmidt process. Next

\begin{equation*} u_3 = v_3-\inner{v_3}{u_1}/\norm{u_1}^2u_1-\inner{v_3}{u_2}/\norm{u_2}^2u_2x^2\text{.} \end{equation*}

We have

\begin{equation*} \inner{v_3}{u_1}= \int_{-1}^1 x^2 dx = \frac23, \inner{u_1}{u_1}=\int_{-1}^1 1dx = 2\text{.} \end{equation*}

Hence

\begin{equation*} u_3 = x^2 - \frac{\frac23}{2} \times 1 = x^2-\frac13\text{.} \end{equation*}

\begin{align*} u_4 \amp = u_4-\inner{v_4}{u_1}/\norm{u_1}^1u_1-\inner{v_4}{u_2}/\norm{u_2}^1u_2-\inner{v_4}{u_3}/\norm{u_3}^1u_3\\ \amp =v_4-\inner{v_4}{u_2}/\norm{u_2}^1u_2-\inner{v_4}{u_3}/\norm{u_3}^1u_3\\ \amp = x^3-\frac35 x\text{.} \end{align*}

Hence an orthogonal basis is \(\beta'=\{1,x,x^2-\frac{1}{3}, x^3-\frac35 x \}\text{.}\) These are the first four Legendre polynomials.

After normalizing the vectors, we get an orthonormal basis

\begin{equation*} \beta''=\left\{\frac{\sqrt{2}}{2}, \frac{\sqrt{6}}{2}x, \frac{3\sqrt{10}}{4}\left(x^2-\frac13\right), \frac{5\sqrt{14}}{4}\left(x^3-\frac35 x\right)\right\}\text{.} \end{equation*}

Figure 7.1.22. Graph of Legendre polynomials

Checkpoint 7.1.23.

Consider the standard basis \(\beta=\{1,x,x^2,x^3\}\) of \({\cal P}_3(\R)\) with inner product \(\inner{f}{g}:=\int_0^1 f(x)g(x)\,dx\text{.}\) Find an orthonormal basis starting with \(\beta\) using the Gram-Schmidt orthogonalization process.

Example 7.1.24.

Let \(A=\left(\begin{array}{rrr}2 \amp -1 \amp 0 \\-1 \amp 2 \amp -1 \\0 \amp -1 \amp 2 \end{array} \right)\text{.}\) It is easy to check that \(A\) is a symmetric and positive definite matrix. (why?) Define an inner product on \(\mathbb{R}^3\) as \(\inner{u}{v}:=v^TAu\text{.}\)

Use the the Gram-Schmidt orthogonalization process to find an orthonormal basis of from the standard basis vectors \(\beta=\{e_1, e_2, e_3\}\) with respect to the above inner product.

\begin{align*} u_1: =\amp (1,0,0)\\ u_2: =\amp e_2-\frac{\inner{u_2\cdot e_2}{u_1}}{\norm{u_1}^2}u_1=(0,1,0)-\frac{-1}{2}(1,0,0)=(1/2,1,0)\\ u_3: =\amp e_3 -\frac{\inner{e_3}{u_1}}{\norm{u_1}^2}u_1-\frac{\inner{e_3}{u_1}}{\norm{u_2}^2}u_2\\ = \amp (0,0,1)-\frac{0}{2}(1,0,0)-\frac{-1}{3/2}(1/2,1,0)=(1/3, 2/3, 1) \end{align*}

Example 7.1.25. Lagrange Interpolating Polynomials.

Fix any \(n+1\) distinct real numbers, \(c_0,c_1,\ldots, c_n\) and define

\begin{equation*} \langle p,q \rangle := p(c_0)q(c_0)+p(c_1)q(c_1)+\cdots +p(c_n)q(c_n) \end{equation*}

an inner product on \({\cal P}_n(\R)\text{.}\)

Recall the Lagrange Polynomial defined (Eqn. (4.5.1)).

\begin{equation*} \ell_i(x)=\prod_{j=0,j\neq i}^{n}\frac{x-c_j}{c_i-c_j}\text{.} \end{equation*}

Then

\begin{equation*} \inner{\ell_i}{\ell_j}=\sum_{j=0}^n \ell_i(c_j)\ell_j(c_j)=\delta_{ij} \end{equation*}

Hence \(\{\ell_i\}\) is an orthonormal basis of \({\cal P}_n(\R)\text{.}\)

Let \(p(x)\in {\cal P}_n(\R)\) be any polynomial, then

\begin{equation*} \inner{p}{\ell_k}=p(c_0)\ell_k(c_0)+p(c_1)\ell_k(c_1)+\cdots +p(c_n)\ell_k(c_n)=p(c_k)\text{.} \end{equation*}

Since \(\{\ell_i\}\) is an orthonormal basis of \({\cal P}_n(\R)\text{,}\) we have

\begin{gather*} p(x) = \inner{p(x)}{\ell_0(x)}\ell_0(x)+\inner{p(x)}{\ell_1(x)}\ell_1(x)+\cdots +\inner{p(x)}{\ell_n(x)}\ell_n(x)\\ = p(c_0)\ell_0(x)+p(c_1)\ell_1(x)+p(c_2)\ell_2(x)+\cdots +p(c_n)\ell_n(x) \end{gather*}

which is the Lagrange interpolation expansion of \(p(x)\text{.}\)

Definition 7.1.26. Projection onto a subspace.

Let \(V\) be an inner product space and \(W\leq V\text{,}\) a finite dimensional subspace of \(V\text{.}\) Let \(\{u_1,\ldots, u_k\}\) be an orthonormal basis of \(W\text{.}\) Suppose \(v\in V\text{.}\) Similar to definition 6.3.5, we can define the orthogonal projection of \(v\) onto \(W\) as

\begin{equation*} \proj_W(v):= \inner{v}{u_1}u_1+\inner{v}{u_2}u_2+\cdots+\inner{v}{u_k}u_k\text{.} \end{equation*}

Checkpoint 7.1.27.

Find the orthogonal projection of vector \(b=\begin{bmatrix}1\\2\\3\\4 \end{bmatrix}\) onto the subspace spanned by three vectors \(\left\{\begin{bmatrix}1\\-1\\0\\1 \end{bmatrix} , \begin{bmatrix}0\\1\\1\\-1 \end{bmatrix} , \begin{bmatrix}1\\1\\-1\\0 \end{bmatrix} \right\}\text{.}\)

Prev Top Next