Lecture 6: Inner Product Space

This lecture is about generalization of the dot product on $\mathbb{R}^n$ to any vector space over $\mathbb{C}^n$.

Definition

An inner product on a complex vector space $V$ is a map $ \langle \cdot,\cdot \rangle : V \times V \to \mathbb{C} $ satisfying the following axioms for all $u,v,w \in V$ and $c,d \in \mathbb{C}$:

Bilinearity: \[ \langle cu + dv,w \rangle = c\langle u,w \rangle + d\langle v,w \rangle, \] \[ \langle u,cv + dw \rangle = \overline{c}\langle u,v \rangle + \overline{d}\langle u,w \rangle. \]
Conjugate symmetry: \[ \langle v,w \rangle = \overline{\langle w,v \rangle}. \]
Positive definiteness: \[ \langle v,v \rangle \in \mathbb{R}_{\geq 0}, \qquad \langle v,v \rangle = 0 \iff v = 0. \]

Examples

On $\mathbb{R}^n$, the standard inner product is the dot product: \[\langle \mathbf{v},\mathbf{w}\rangle = \mathbf{v}\cdot\mathbf{w} = v_1w_1 + v_2w_2 + \cdots + v_nw_n.\]
On $\mathbb{C}^n$, the standard inner product is: \[\langle \mathbf{v},\mathbf{w}\rangle = v_1\overline{w_1} + v_2\overline{w_2} + \cdots + v_n\overline{w_n}.\]
On $C_{\mathrm{per}}[0,2\pi]$, the continuous periodic functions on $[0,2\pi]$ or the Reimann integrable functions with period $2\pi$, the following defines an inner product inner product: \[ \langle f,g\rangle = \frac{1}{2\pi}\int_0^{2\pi} f(x)\overline{g(x)}\,dx. \] For example: \[ \langle \sin x,\cos x\rangle = \frac{1}{2\pi}\int_0^{2\pi}\sin x\cos x\,dx = 0. \]

Consequences of the Inner Product: (1) Norm

An inner product defines notions of length in our vector space. Precisely, every inner product defines a norm. For an inner product space $V$ and $\mathbf{v}\in V$, define:

$$ \begin{align*} \|\mathbf{v}\| = \sqrt{\langle \mathbf{v},\mathbf{v}\rangle}. \end{align*} $$

This is called the norm induced by the inner product. The inner product axioms guarantee that it satisfies all the norm axioms. For example, on $\mathbb{R}^n$ with the standard inner product:

$$ \begin{align*} \|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}. \end{align*} $$

On $C[0,2\pi]$ with the normalized integral inner product:

$$ \begin{align*} \|f\|^2 = \frac{1}{2\pi}\int_0^{2\pi}|f(x)|^2\,dx. \end{align*} $$

However note that not every norm comes from an inner product. This norm can act as a metric for the inner product space $V$. To show this, we need the second consequence of the inner product: Orthogonality. This will be discussed here. Once we have Cauchy Schwarz, we can show that $(V,d)$ is a metric space using the norm we defined. In particular, let

$$ d(x,y):=\|x-y\|=\sqrt{\langle x-y,x-y\rangle}. $$

Recall that a space is a metric space if

Non-negativity: $d(x,y) \geq 0$. It's clear that we get this from the inner product axioms.
$d(x, y) = 0$ if and only if $x = y$.
Symmetry: We also get this from the inner product axioms
Triangle Inequality: $d(x, z) \leq d(x, y) + d(y, z)$

We show (4) is true as follows: we claim that

$$ \boxed{ \|u + w\| \leq \|u\| + \|w\| } $$

Observe that

$$ \begin{align*} \|u + w\|^2 &= \langle u+w, u+w \rangle \\ &= \|u\|^2 + \langle u, w\rangle + \langle w, u\rangle + \|w\|^2 \\ &= \|u\|^2 + \langle u, w\rangle + \overline{\langle u, w\rangle} + \|w\|^2 \\ &= \|u\|^2 + 2\operatorname{Re}(\langle u, w\rangle) + \|w\|^2 \\ &\leq \|u\|^2 + 2\left|\langle u, w\rangle\right| + \|w\|^2 \\ &\leq \|u\|^2 + 2\|u\|\|w\| + \|w\|^2 \quad \text{(by Cauchy Shwarz)} \\ &= (\|u\| + \|w\|)^2 \end{align*} $$

Taking the square root of both sides we obtain

$$ \begin{align*} \|u + w\| &\leq \|u\| + \|w\| \end{align*} $$

Now we can write

$$ \begin{align*} d(x,z) = \|x - z\| &= \|(x - y) + (y - z)\| \\ &\leq \|x - y\| + \|y - z\| \\ &= d(x,y) + d(y,z) \end{align*} $$

Note how we use the Pythagorean Theorem to show the Cauchy-Schwarz inequality and use Cauchy-Schwarz to prove the triangle inequality here!

Consequences of the Inner Product: (3) Orthonormal Basis and Orthogonal Projection

First let’s define an orthonormal basis

Definition (Orthonormal Basis)

Let $V$ be an inner product space and let $W$ be a finite-dimensional subspace of $V$. A set $\{e_1,\ldots,e_d\}\subseteq W$ is an orthonormal basis of $W$ if:

The vectors are mutually orthogonal and have norm one so $ \langle e_i,e_j\rangle=0$ for $i\neq j$ and $\|e_i\|=1$ for all $i$.
$d=\dim(W)$.

An orthonormal basis of $W$ always exists by the Gram–Schmidt process. This implies that every $w\in W$ can be written as

$$ w=\sum_{i=1}^{d}\langle w,e_i\rangle e_i. \tag{2} $$

Let’s return now specifically to the space of continuous $2\pi$-periodic functions, with inner product

$$ \langle f,g\rangle=\frac{1}{2\pi}\int_0^{2\pi}f(x)\overline{g(x)}\,dx. $$

We already have an orthonormal set in this space:

$$ \{e^{inx}:n\in\mathbb Z\}. $$

Distinct functions in this set are orthogonal, and each has norm (1). Hence we already satisfy property (1) in the conditions above. But what about property (2)? How do we find a spanning set for an infinite dimensional space? Consider instead a finite dimensional subspace $W$ and define

Definition (Orthogonal Complement)

Given a subspace $W$, its orthogonal complement is

$$ W^\perp = \{v\in V:\langle v,w\rangle=0 \text{ for every }w\in W\}. $$

So $W^\perp$ is the set of all vectors in $V$ that are orthogonal to every vector in $W$. So if $W$ was the horizontal axis in $\mathbb{R}^2$, then $W^\perp$ is the vertical axis. Now, assuming that $W$ is finite

Definition (Orthogonal Projection)

The orthogonal projection of any vector $x$ onto $W$ is the unique vector, denoted by $\operatorname{proj}_W \mathbf{x} \in W$, such that $\mathbf{x} - \operatorname{proj}_W \mathbf{x} \in W^\perp$, i.e.

$$ \left\langle \mathbf{x} - \operatorname{proj}_W \mathbf{x}, \mathbf{w} \right\rangle = 0 $$

for all $\mathbf{w} \in W$.

So we can define the projection of $x$ onto $W$ this way but we need to justify

Existence: some vector in $W$ satisfies them.
Uniqueness: at most one vector in $W$ satisfies them.

Theorem

Let $W$ be a finite-dimensional subspace of an inner product space $V$. Then $\operatorname{proj}_W$ is a well-defined operator from $V$ onto $W$. Furthermore, if $\{\mathbf{e}_1,\ldots,\mathbf{e}_n\}$ is an orthonormal basis for $W$, then $$ \begin{align*} \operatorname{proj}_W(\mathbf{x}) &= \sum_{k=1}^{n}\langle \mathbf{x},\mathbf{e}_k\rangle\mathbf{e}_k. \end{align*} $$

Proof. First note that given a vector $\mathbf{x}$. Then by definition, an orthogonal projection $\mathbf{u}$ of $\mathbf{x}$ onto $W$ must satisfy both:

$$ \begin{align*} \mathbf{u}\in W \qquad\text{and}\qquad \langle \mathbf{x}-\mathbf{u},\mathbf{w}\rangle=0 \quad\text{for every }\mathbf{w}\in W. \end{align*} $$

for every $\mathbf{w}\in W$. Let’s prove uniqueness first. Suppose that $\mathbf{u}_1,\mathbf{u}_2\in W$ satisfy

$$ \begin{align*} \langle \mathbf{x}-\mathbf{u}_1,\mathbf{w}\rangle &= \langle \mathbf{x}-\mathbf{u}_2,\mathbf{w}\rangle = 0 \end{align*} $$

for all $\mathbf{w}\in W$. Since $W$ is a subspace, then $\mathbf{u_1}-\mathbf{u_2}$ is also in $W$. Hence if we first subtract the two equalities to get

$$ \begin{align*} \langle \mathbf{u}_1-\mathbf{u}_2,\mathbf{w}\rangle = \langle \mathbf{u}_1,\mathbf{w}\rangle - \langle \mathbf{u}_2,\mathbf{w}\rangle = 0 \quad\text{for every }\mathbf{w}\in W. \end{align*} $$

If we particularly choose $\mathbf{w}=\mathbf{u}_1-\mathbf{u}_2\in W$, then

$$ \begin{align*} \|\mathbf{u}_1-\mathbf{u}_2\|^2 &= 0. \end{align*} $$

Hence, $\mathbf{u}_1=\mathbf{u}_2$.

To prove existence, let $\{\mathbf{e}_1,\ldots,\mathbf{e}_n\}$ be an orthonormal basis for $W$, where $n=\dim W$. Such a basis exists by the Gram–Schmidt process. Define

$$ \begin{align*} \boxed{ \mathbf{u} = \sum_{k=1}^{n} \langle \mathbf{x},\mathbf{e}_k\rangle\mathbf{e}_k. } \end{align*} $$

We claim that $u$ is the orthogonal projection of $\mathbf{x}$ onto $W$. To show this, we need to prove

$\mathbf{u}\in W$.
$\langle \mathbf{x}-\mathbf{u},\mathbf{w}\rangle=0$ for every $\mathbf{w}\in W$.

Note that $\mathbf{u} \in W$ since it’s a linear combination of the basis vectors. To show property $2$. Let $\mathbf{w} \in W$. Then we can write $\mathbf{w}=\sum_{k=1}^{n}a_k\mathbf{e}_k\in W$ (any $w$ is a linear combination of the basis vectors). Then

$$ \begin{align*} \langle \mathbf{x}-\mathbf{u},\mathbf{w}\rangle &= \langle \mathbf{x},\mathbf{w}\rangle - \langle \mathbf{u},\mathbf{w}\rangle. \end{align*} $$

Note that

$$ \begin{align*} \langle \mathbf{x},\mathbf{w}\rangle &=\left\langle \mathbf{x},\sum_{k=1}^{n}a_k\mathbf{e}_k\right\rangle =\sum_{k=1}^{n}\langle \mathbf{x},a_k\mathbf{e}_k\rangle =\sum_{k=1}^{n}\overline{a_k}\langle \mathbf{x},\mathbf{e}_k\rangle = \sum_{k=1}^{n} \langle \mathbf{x},\mathbf{e}_k\rangle\overline{a_k}, \end{align*} $$

Moreover,

$$ \begin{align*} \langle \mathbf{u},\mathbf{w}\rangle &= \left\langle \sum_{j=1}^{n}\langle \mathbf{x},\mathbf{e}_j\rangle\mathbf{e}_j, \sum_{k=1}^{n}a_k\mathbf{e}_k \right\rangle \\ &= \sum_{j=1}^{n} \langle \mathbf{x},\mathbf{e}_j\rangle \left\langle \mathbf{e}_j,\sum_{k=1}^{n}a_k\mathbf{e}_k\right\rangle \quad \text{(linearity in the first slot)} \\ &= \sum_{j=1}^{n} \langle \mathbf{x},\mathbf{e}_j\rangle \sum_{k=1}^{n}\left\langle \mathbf{e}_j,a_k\mathbf{e}_k\right\rangle \\ &= \sum_{j=1}^{n} \langle \mathbf{x},\mathbf{e}_j\rangle \sum_{k=1}^{n}\overline{a_k} \langle \mathbf{e}_j,\mathbf{e}_k\rangle \quad \text{(conjugate linearity in the second slot)} \\ &= \sum_{j=1}^{n} \langle \mathbf{x},\mathbf{e}_j\rangle \left( \overline{a_j}\underbrace{\langle \mathbf{e}_j,\mathbf{e}_j\rangle}_{1} + \sum_{\substack{k=1\\k\ne j}}^{n} \overline{a_k}\underbrace{\langle \mathbf{e}_j,\mathbf{e}_k\rangle}_{0} \right) \quad \text{(orthonormality)} \\ &= \sum_{j=1}^{n} \langle \mathbf{x},\mathbf{e}_j\rangle\overline{a_j} \\ &= \sum_{k=1}^{n} \langle \mathbf{x},\mathbf{e}_k\rangle\overline{a_k} \quad \text{(renaming the summation index).} \end{align*} $$

Therefore,

$$ \begin{align*} \langle \mathbf{x}-\mathbf{u},\mathbf{w}\rangle &= 0 \end{align*} $$

for every $\mathbf{w}\in W$. Thus, $\mathbf{u}$ is the unique orthogonal projection of $\mathbf{x}$ onto $W$. Finally, for every $\mathbf{w}\in W$, the vector $\mathbf{w}$ itself satisfies the defining conditions, so $\operatorname{proj}_W(\mathbf{w})=\mathbf{w}$. Hence, $\operatorname{proj}_W$ maps $V$ onto $W$.

The following identities may be useful later for an orthonormal set: \begin{align} \left\langle \sum_{i=1}^{n}a_i\mathbf{e}_i, \sum_{i=1}^{n}b_i\mathbf{e}_i \right\rangle &= \sum_{i=1}^{n}a_i\overline{b_i},
\left|a_1\mathbf{e}_1+\cdots+a_n\mathbf{e}_n\right|^2 &= \sum_{i=1}^{n}|a_i|^2. \end{align}

Finally, we can apply these results to Fourier series. Let $V=L^2[0,2\pi]$ and define \begin{align} W_N &= \operatorname{span}\left{ e^{inx}: n=-N,\ldots,-1,0,1,\ldots,N \right}. \end{align} With the normalized inner product \begin{align} \langle f,g\rangle &= \frac{1}{2\pi}\int_0^{2\pi}f(x)\overline{g(x)}\,dx, \end{align} the functions $\{e^{inx}\}_{n=-N}^{N}$ form an orthonormal basis for $W_N$. Therefore, for every $f\in V$, \begin{align} \operatorname{proj}_{W_N}f &= \sum_{n=-N}^{N}\langle f,e^{inx}\rangle e^{inx}
&= \sum_{n=-N}^{N}\widehat{f}(n)e^{inx}
&= S_Nf(x). \end{align}

Now, if $W$ is finite dimensional, then for any vector $v \in V$, we can write $v$ uniquely as

$$ v=p+r,\qquad p\in W,\quad r\in W^\perp. $$

In fact, $p=\operatorname{proj}_W v$. So $r = v-\operatorname{proj}_W v\in W^\perp$ and equivalently,

$$ \langle v-\operatorname{proj}_W v,w\rangle=0 \qquad \text{for every }w\in W. $$

Consider

$$ W_N=\operatorname{span}\{e^{-iNx},\ldots,1,\ldots,e^{iNx}\} $$

So $W_N$ consists of all functions of the form

$$ \sum_{n=-N}^{N}c_n e^{inx}. $$

It’s a finite-dimensional subspace of our continuous periodic function space, with dimension $2N+1$. Next, we’ll define

This leads to the following theorem:

Theorem (Cauchy–Schwarz inequality)

Let $V$ be an inner product space, and let $W$ be a subspace with an orthonormal basis $\{e_1,\ldots,e_d\}$. Then the orthogonal projection of any vector $v\in V$ onto $W$ is $$ \operatorname{proj}_W v =\sum_{i=1}^{d}\langle v,e_i\rangle e_i. $$

Proof. First let

$v$ is the original vector in $V$, which might not belong to $W$.
$e_1,\ldots,e_d$ are the orthonormal basis vectors for $W$.
$\langle v,e_i\rangle$ is a scalar measuring the component of $v$ along $e_i$.
$p=\sum_i\langle v,e_i\rangle e_i$ adds those components together. This is the vector we’re trying to prove is the projection.
$w=\sum_i b_i e_i$ is an arbitrary vector in $W$. The $b_i$ are its coordinates.

$$ \begin{align*} \big\langle v-\sum_{i=1}^{d}\langle v,e_i\rangle e_i, \sum_{i=1}^{d}b_i e_i \big\rangle &=\bigl\langle v,\sum_{j=1}^{d}b_j e_j\bigr\rangle -\bigl\langle \sum_{i=1}^{d}\langle v,e_i\rangle e_i, \sum_{j=1}^{d}b_j e_j \bigr\rangle\\[1em] &=\sum_{j=1}^{d}\overline{b_j}\langle v,e_j\rangle -\sum_{i=1}^{d}\sum_{j=1}^{d} \langle v,e_i\rangle\overline{b_j}\langle e_i,e_j\rangle\\[1em] &=\sum_{j=1}^{d}\overline{b_j}\langle v,e_j\rangle -\sum_{i=1}^{d}\langle v,e_i\rangle\overline{b_i}\\[1em] &=\sum_{i=1}^{d} \bigl( \overline{b_i}\langle v,e_i\rangle -\langle v,e_i\rangle\overline{b_i} \bigr)\\[1em] &=0. \end{align*} $$

Define

$$ \begin{align*} p=\sum_{i=1}^{d}\langle v,e_i\rangle e_i. \end{align*} $$

Since each $e_i\in W$, we have $p\in W$. Every vector $w\in W$ can be written as $w=\sum_{i=1}^{d}b_i e_i.$ Therefore,

$$ \begin{align*} \langle v-p,w\rangle &= \left\langle v-\sum_{i=1}^{d}\langle v,e_i\rangle e_i, \sum_{i=1}^{d}b_i e_i \right\rangle\\ &=\sum_{i=1}^{d}\overline{b_i}\langle v,e_i\rangle -\sum_{i=1}^{d}\langle v,e_i\rangle\overline{b_i}\\ &=0. \end{align*} $$

Thus $v-p$ is orthogonal to every vector in $W$, so $v-p\in W^\perp$. Since $p\in W$, it is the orthogonal projection of $v$ onto $W$:

$$ \begin{align*} \operatorname{proj}_W v =\sum_{i=1}^{d}\langle v,e_i\rangle e_i. \qquad \square \end{align*} $$

References

SFSU Fourier Analysis by Professor Chun-Kit Lai
Fourier Analysis: An Introduction by Elias M.Stein and Rami Shakarchi