Skip to main content

Linear Algebra with SageMath A Gentle Introduction

Ajit Kumar

Section 7.1 Inner Product

In the last chapter, we dealt with notion of dot product and geometry in \(\R^n\text{.}\) The dot product and related notion can be generalized to an arbitrary vector space over \(\R\) or \(\mathbb{C}\text{.}\) All the notions, we have learned in the last section can be generalized over an inner product space.
🔗
Note that the dot product of two vectors in \(\R^n\) is a scalar, in particular, dot product can be thought of as a function from \('\cdot' \colon \R^n\times \R^n \to\R\) satisfying the following properties:
🔗
  1. \(x \cdot x\geq 0\) for all \(x\in \R^n\text{.}\)
    🔗
    🔗
  2. \(x \cdot x= 0\) if and only if \(x=0\text{.}\)
    🔗
    🔗
  3. \(x \cdot y=y\cdot x\) for all \(x,y \in \R^n\text{.}\)
    🔗
    🔗
  4. \(x\cdot (y+z) = x\cdot y +x\cdot z\) for all \(x,y,z\in \R^n\text{.}\)
    🔗
    🔗
  5. \(x\cdot (\alpha y)=\alpha x\cdot y = (\alpha x)\cdot y\text{.}\)
    🔗
    🔗
🔗
The notion of dot product on \(\R^n\) can ve generalized on vector space known as inner product. We have the following definition.
🔗

Definition 7.1.1. Inner Product.

Let \(V\) be a vector space over \(\R\text{.}\) An inner product on \(V\) is a function that assigns a real number \(\langle x, y\rangle\) to every pair \(x,y\) of vectors in \(V\) (that is, a function \(\langle \cdot, \cdot\rangle\colon V \times V \to \R\)) satisfying the following properties.
🔗
  1. \(\langle x, x\rangle \geq 0\) for all \(x \in V\) and \(\langle x, x\rangle = 0\) if and only if \(x=0\text{.}\)
    🔗
    🔗
  2. \(\langle x, y\rangle = \langle y, x\rangle\) for all \(x,y \in V\text{.}\) (Symmetry)
    🔗
    🔗
  3. \(\langle x, (y+z)\rangle=\langle x, y\rangle+\langle x, z\rangle\) for all \(x,y,z\in V\text{.}\)
    🔗
    🔗
  4. \(\langle x, (\alpha y)\rangle=\alpha\langle x, y\rangle\) for all \(x,y\in V\) and \(\alpha \in \R\text{.}\)
    🔗
    🔗
🔗
If \(V\) is real vector space with inner product \(\langle. , .\rangle\text{.}\) Then \((V, \langle . , .\rangle)\) called in inner product space over \(\R\text{.}\)
🔗
🔗
The last two properties make the inner product linear in the second variable. Using the symmetry property, it can also be shown that the inner product is linear in the first variable as well. That is,
\begin{equation*} \langle (x+y),z\rangle=\langle x, z\rangle+\langle y, z\rangle, \text{ and } \langle (\alpha x) y\rangle=\alpha\langle x, y\rangle \end{equation*}
🔗
Next we look at several examples of inner procuct on various vector spaces that we have defined in Chapter 4.
🔗

Example 7.1.2.

On \(\R^n\text{,}\) the standard dot product is an inner product. Thus define
\begin{equation*} \langle x, y\rangle:=x\cdot y \end{equation*}
🔗
This is also called the Euclidean inner product on \(\R^n\text{.}\)
🔗
🔗

Example 7.1.3.

Let \(V=M_n(\R)\text{,}\) the set of all \(n\times n\) matrices over \(\R\text{.}\) Define
\begin{equation*} \langle A, B\rangle:=tr(AB^T) \end{equation*}
🔗
It is easy to show that this is an inner product on \(M_n(\R)\text{.}\)
🔗
Note that this inner product can be thought of as the standard dot product on \(\R^{n^2}\text{.}\) The elements of the matrix \(A\) can be thought of as a vector in \(\R^{n^2}\text{.}\) Then
\begin{equation*} tr(AB^T) =\sum\sum a_{ij}b_{ij}\quad \quad \text{ why?}\text{.} \end{equation*}
Work with \(2\times 2\) matrices and then try to prove this for \(n\times n\) matrices.
🔗
🔗

Example 7.1.4.

Let \(A\) be an \(n\times n\) symmetric and positive definite matrix. On \(\R^n\text{,}\) define
\begin{equation*} \langle x, y \rangle := x\cdot Ay = x^TAy \end{equation*}
🔗
Then this is an inner product on \(\R^n\text{.}\) (Where do we need \(A\) to be positive definie?)
🔗
🔗

Remark 7.1.5.

\(A\)\(B\)\(B^2=A\text{.}\)\(B\)\(A\)\(A^{1/2}\text{.}\)
\begin{equation*} \langle x,y\rangle = x^T A y = (A^{1/2}x)\cdot(A^{1/2}y), \end{equation*}
\(x\mapsto A^{1/2}x\text{.}\)
🔗

Example 7.1.6.

Let \(V={\cal C}([0,1])\) the set of all continuous function from \([0,1]\) to \(\R\text{.}\) Define
\begin{equation*} \langle f, g \rangle:=\int_0^1 f(t)g(t)dt\text{.} \end{equation*}
🔗
This is an inner product on \(V\text{.}\) (You may need real analysis to conclude that \(\int_0^1 f(t)f(t)dt=\) then \(f=0\text{.}\))
🔗
🔗

Example 7.1.7.

Let \(p\) and \(q\) be two polynomials in \({\cal P}_n(\mathbb{R})\text{.}\) Then define
\begin{equation*} \langle p,q \rangle := p(0)q(0)+p(1)q(1)+\cdots +p(n)q(n)\text{.} \end{equation*}
🔗
It is easy to see that \(\langle p,q \rangle\) defined inner product on the vector space \({\cal P}_n(R)\text{.}\)
🔗
Here \(0, 1, 2, \ldots, n\) are nothing special. Instead, we can use any \(n+1\) distinct real numbers, \(c_0,c_1,\ldots, c_n\text{.}\)
🔗
🔗

Definition 7.1.8.

Let \((V, \langle .\rangle)\) be a real inner product space. Then norm of any vector \(x\in V\) corresponding to the inner product \(\langle . \rangle\) is defined as
\begin{equation*} \norm{x}=\sqrt{\langle x, x\rangle}\text{.} \end{equation*}
The distance between twp vectors \(x\) and \(y\) is defined as
\begin{equation*} d(x,y):=\norm{x-y}\text{.} \end{equation*}
🔗
🔗

Checkpoint 7.1.9.

Let \(V\) be an inner product space. Then for any two vectors \(x, y\in V\text{,}\) show that
1. \(\norm{x+y}^2=\norm{x}^2+\norm{y}^2+2\inner{x}{y}\)
🔗
2. \(\norm{x-y}^2=\norm{x}^2+\norm{y}^2-2\inner{x}{y}\)
🔗
🔗
Hint.
The proof follows by looking at \(\langle x+y,x+y \rangle \) and \(\langle x-y,x-y \rangle \) and using the properties of inner product.
🔗
🔗
🔗

Checkpoint 7.1.10.

If \(x, y\) are two vectors in an inner product space \(V\) with inner product \(\langle .\rangle\text{.}\) Then show that
\begin{equation*} \norm{x+y}^2+\norm{x-y}^2=2(\norm{x}^2+\norm{y}^2)\text{.} \end{equation*}
🔗
This is called the parallelogram identity. Geometrically, in a parallelogram, the sum of square of the diagonals is 2 times the sum of the squares of the side lengths.
🔗
Hint.
Proof follows from Checkpoint 7.1.9.
🔗
🔗
🔗
Let us play with the Cauchy-Schwarz inequality (7.1.1). Suppose \(x\) and \(y\) are non zero vectors in \(V\text{,}\) Then
\begin{equation*} |\inner{x}{y}|\leq \norm{x}\norm{y}\Rightarrow \frac{|\inner{x}{y}|}{\norm{x}\norm{y}}\leq 1 \end{equation*}
🔗
Hence we have
\begin{equation*} -1\leq \frac{\inner{x}{y}}{\norm{x}\norm{y}}\leq 1\text{.} \end{equation*}
🔗
Thus for any two non zero vectors, \(x\) and \(y\text{,}\) \(\frac{\inner{x}{y}}{\norm{x}\norm{y}}\) always lies between \(-1\) and 1. This allows us to define the angle between two non zero vectors. We assign this number to \(\cos\theta\) with \(\theta\in[-\pi,\pi]\) called the angle between \(x\) and \(y\text{.}\) Thus, if \(\theta\) is the angle between \(x\) and \(y\text{,}\) then we have
\begin{equation*} \cos \theta = \frac{\inner{x}{y}}{\norm{x}\norm{y}} \end{equation*}
🔗
All the notions that we defined for dot product, namely, orthogonality, orthogonal projection, Gram-Schmidt orthogonalization process can we defined in a similar manner. All we need to do is, replace the dot product by the given inner product.
🔗

Definition 7.1.14.

Any vector space \(V\) over \(\R\) with a function \(\norm{.} : V \to \R\) which satisfies all the properties mentioned in Theorem 7.1.13 is called a normed linear space.. Thus any inner product space is also a normed linear space.
🔗
🔗

Note 7.1.15.

\(\R^n\)
🔗

Checkpoint 7.1.16.

Let \((V, \inner{.}{.})\) be a real inner product space. Define (i) orthogonality of two vectors \(x\) and \(y\) in \(V\text{,}\) (ii) orthogonal complement of a subset \(U\) of \(V\text{,}\) (iii) Orthogonal projection of a vector \(v\) onto a non-zero vector \(u\text{,}\) (iv) orthogonal set and orthonormal sets in \(V\) and (v) Gram-Schmidt orthogonalization process.
🔗
🔗

Checkpoint 7.1.17.

Let \(x, y\) be two vectors in an inner product space \(V\text{.}\) Then show that
🔗
(i) \(x\) and \(y\) are orthogonal if and only if \(\norm{x+y}=\norm{x-y}\text{.}\) (what does it mean geometrically?)
🔗
(ii) \(x+y\) and \(x-y\) are orthogonal if and only if \(\norm{x}=\norm{y}\text{.}\)
🔗
🔗

Checkpoint 7.1.18.

Let \(C([-\pi,\pi])\) be the vectors space of set of continuous functions from \([-\pi,\pi]\) to \(\R\text{.}\) Define the inner product on \(C([-\pi,\pi])\) as
\begin{equation*} \inner{f}{g}:=\int_{-\pi}^\pi f(t) g(t)`dt\text{.} \end{equation*}
🔗
Show that under this inner product \(\{1, \sin nx,\cos mx\}\) is an orthogonal set.
🔗
🔗

Checkpoint 7.1.19. Pythagoras Theorem.

Let \((V, \inner{.}{.})\) be a real inner product space. Let \(x_1,x_2\,x_n\) be \(n\) orthogonal vectors in \(V\text{.}\) Then
\begin{equation*} \norm{x_1+x_2+\cdots+x_n}^2=\norm{x_1}^2+\norm{x_2}^2+\cdots+\norm{x_n}^2\text{.} \end{equation*}
🔗
This is called the Pythagoras theorem.
🔗
🔗

Checkpoint 7.1.20.

Let \(\beta=\{u_1,\ldots, u_n\}\) be an orthogonal basis of an inner product space \(V\text{.}\) Let \(v\in V\) and \(\theta_1,\ldots, \theta_n\) be the angle between \(v\) and \(u_1,\ldots, u_n\text{,}\) respectively. Then
\begin{equation*} \cos\theta_1^2+\cdots+\cos\theta_n^2=1\text{.} \end{equation*}
🔗
Here \(\cos\theta_i\) are called the direction cosines of \(v\) corresponding to \(\beta\text{.}\)
🔗
🔗

Checkpoint 7.1.21.

Let \(\beta=\{u_1,\ldots, u_n\}\) be an orthogonal basis of an inner product space \(V\text{.}\) Let \(x\) and \(y\) be two vectors such that \(x=\sum x_i u_i\) and \(y=\sum y_i u_i\text{.}\) Then
\begin{equation*} \inner{x}{y}=x_1y_1+x_2y_2+\cdots+x_ny_n \end{equation*}
and
\begin{equation*} \norm{x}^2=x_1^2+x_2^2+\cdots +x_n^2\text{.} \end{equation*}
🔗
🔗

Example 7.1.22. Legendre Polynomials.

Consider \(V ={\cal P}_3(\R)\) with inner product \(\inner{p}{q}:=\int_{-1}^1 p(x)q(x)\,dx\text{.}\) Use the standard basis \(\beta =\{v_1,v_2,v_3,v_4\} = \{1,x,x^2,x^3\}\) to find an orthogonal basis of \({\cal P}_3(\R)\text{.}\)
🔗
First of all notice that \(\beta\) is not an orthogonal basis. For \(\inner{v_1}{v_3}=\inner{1}{x^2} = \int_{-1}^1 x^2 dx = \frac23\text{,}\) \(\inner{v_2}{v_4}=\int_{-1}^1 x^4 dx = \frac25\text{.}\) Also note that \(\inner{v_1}{v_2}=\int_{-1}^1 xdx = 0\text{.}\) \(\inner{v_2}{v_3}=\int_{-1}^1 x^3dx = 0\text{.}\) \(\inner{v_1}{v_4}=\int_{-1}^1 x^3 dx = 0\text{.}\) \(\inner{v_3}{v_4}=\int_{-1}^1 x^5dx = 0\text{.}\)
🔗
Since \(v_1\) and \(v_2\) are already orthogonal, we can choose \(u_1=v_1=1\) and \(u_2 = v_2=x\) in the Gram-Schmidt process. Next
\begin{equation*} u_3 = v_3-\inner{v_3}{u_1}/\norm{u_1}^2u_1-\inner{v_3}{u_2}/\norm{u_2}^2u_2x^2\text{.} \end{equation*}
🔗
We have
\begin{equation*} \inner{v_3}{u_1}= \int_{-1}^1 x^2 dx = \frac23, \inner{u_1}{u_1}=\int_{-1}^1 1dx = 2\text{.} \end{equation*}
🔗
Hence
\begin{equation*} u_3 = x^2 - \frac{\frac23}{2} \times 1 = x^2-\frac13\text{.} \end{equation*}
🔗
Next
\begin{align*} u_4 \amp = u_4-\inner{v_4}{u_1}/\norm{u_1}^1u_1-\inner{v_4}{u_2}/\norm{u_2}^1u_2-\inner{v_4}{u_3}/\norm{u_3}^1u_3\\ \amp =v_4-\inner{v_4}{u_2}/\norm{u_2}^1u_2-\inner{v_4}{u_3}/\norm{u_3}^1u_3\\ \amp = x^3-\frac35 x\text{.} \end{align*}
🔗
Hence an orthogonal basis is \(\beta'=\{1,x,x^2-\frac{1}{3}, x^3-\frac35 x \}\text{.}\) These are the first four Legendre polynomials.
🔗
After normalizing the vectors, we get an orthonormal basis
\begin{equation*} \beta''=\left\{\frac{\sqrt{2}}{2}, \frac{\sqrt{6}}{2}x, \frac{3\sqrt{10}}{4}\left(x^2-\frac13\right), \frac{5\sqrt{14}}{4}\left(x^3-\frac35 x\right)\right\}\text{.} \end{equation*}
🔗
Figure 7.1.23. Graph of Legendre polynomials
🔗
🔗

Checkpoint 7.1.24.

Consider the standard basis \(\beta=\{1,x,x^2,x^3\}\) of \({\cal P}_3(\R)\) with inner product \(\inner{f}{g}:=\int_0^1 f(x)g(x)\,dx\text{.}\) Find an orthonormal basis starting with \(\beta\) using the Gram-Schmidt orthogonalization process.
🔗
Hint.
🔗
🔗
🔗

Example 7.1.25.

Let \(A=\left(\begin{array}{rrr}2 \amp -1 \amp 0 \\-1 \amp 2 \amp -1 \\0 \amp -1 \amp 2 \end{array} \right)\text{.}\) It is easy to check that \(A\) is a symmetric and positive definite matrix. (why?) Define an inner product on \(\mathbb{R}^3\) as \(\inner{u}{v}:=v^TAu\text{.}\)
🔗
Use the the Gram-Schmidt orthogonalization process to find an orthonormal basis of from the standard basis vectors \(\beta=\{e_1, e_2, e_3\}\) with respect to the above inner product.
\begin{align*} u_1: =\amp (1,0,0)\\ u_2: =\amp e_2-\frac{\inner{u_2\cdot e_2}{u_1}}{\norm{u_1}^2}u_1=(0,1,0)-\frac{-1}{2}(1,0,0)=(1/2,1,0)\\ u_3: =\amp e_3 -\frac{\inner{e_3}{u_1}}{\norm{u_1}^2}u_1-\frac{\inner{e_3}{u_1}}{\norm{u_2}^2}u_2\\ = \amp (0,0,1)-\frac{0}{2}(1,0,0)-\frac{-1}{3/2}(1/2,1,0)=(1/3, 2/3, 1) \end{align*}
🔗
Hint.
🔗
🔗

Example 7.1.26. Lagrange Interpolating Polynomials.

Fix any \(n+1\) distinct real numbers, \(c_0,c_1,\ldots, c_n\) and define
\begin{equation*} \langle p,q \rangle := p(c_0)q(c_0)+p(c_1)q(c_1)+\cdots +p(c_n)q(c_n) \end{equation*}
an inner product on \({\cal P}_n(\R)\text{.}\)
🔗
Recall the Lagrange Polynomial defined (Eqn. (4.5.1)).
\begin{equation*} \ell_i(x)=\prod_{j=0,j\neq i}^{n}\frac{x-c_j}{c_i-c_j}\text{.} \end{equation*}
🔗
Then
\begin{equation*} \inner{\ell_i}{\ell_j}=\sum_{j=0}^n \ell_i(c_j)\ell_j(c_j)=\delta_{ij} \end{equation*}
🔗
Hence \(\{\ell_i\}\) is an orthonormal basis of \({\cal P}_n(\R)\text{.}\)
🔗
Let \(p(x)\in {\cal P}_n(\R)\) be any polynomial, then
\begin{equation*} \inner{p}{\ell_k}=p(c_0)\ell_k(c_0)+p(c_1)\ell_k(c_1)+\cdots +p(c_n)\ell_k(c_n)=p(c_k)\text{.} \end{equation*}
Since \(\{\ell_i\}\) is an orthonormal basis of \({\cal P}_n(\R)\text{,}\) we have
\begin{gather*} p(x) = \inner{p(x)}{\ell_0(x)}\ell_0(x)+\inner{p(x)}{\ell_1(x)}\ell_1(x)+\cdots +\inner{p(x)}{\ell_n(x)}\ell_n(x)\\ = p(c_0)\ell_0(x)+p(c_1)\ell_1(x)+p(c_2)\ell_2(x)+\cdots +p(c_n)\ell_n(x) \end{gather*}
which is the Lagrange interpolation expansion of \(p(x)\text{.}\)
🔗
🔗

Definition 7.1.27. Projection onto a subspace.

Let \(V\) be an inner product space and \(W\leq V\text{,}\) a finite dimensional subspace of \(V\text{.}\) Let \(\{u_1,\ldots, u_k\}\) be an orthonormal basis of \(W\text{.}\) Suppose \(v\in V\text{.}\) Similar to definition 6.3.5, we can define the orthogonal projection of \(v\) onto \(W\) as
\begin{equation*} \proj_W(v):= \inner{v}{u_1}u_1+\inner{v}{u_2}u_2+\cdots+\inner{v}{u_k}u_k\text{.} \end{equation*}
🔗
🔗

Checkpoint 7.1.28.

Find the orthogonal projection of vector \(b=\begin{bmatrix}1\\2\\3\\4 \end{bmatrix}\) onto the subspace spanned by three vectors \(\left\{\begin{bmatrix}1\\-1\\0\\1 \end{bmatrix} , \begin{bmatrix}0\\1\\1\\-1 \end{bmatrix} , \begin{bmatrix}1\\1\\-1\\0 \end{bmatrix} \right\}\text{.}\)
🔗
🔗

Activity 7.1.1.

Notion of orthogonal projection and reflection in an inner product space.
🔗
Note that the concepts of Gram–Schmidt orthogonalization, orthogonal projection, and reflection can be naturally extended to an inner product space \((V, \langle \cdot, \cdot \rangle)\text{.}\) Explore how these notions generalize in such spaces, and implement solutions to related problems using Sage.
🔗
🔗
🔗
PrevTopNext