Lecture 8: Convergence in Inner Product Spaces

In the previous lecture, we studied inner product spaces so we could use vector geometry to understand Fourier series. This is because we saw that each Fourier partial sum is actually an orthogonal projection onto a finite-dimensional space of trigonometric polynomials. From the perspective of inner product, we are able to treat functions as vectors and say that

The functions $e^{inx}$ are orthogonal "directions".
Fourier coefficients tell us the component of $f$ in each direction.
The partial sum $S_Nf$ is the orthogonal projection of $f$ onto the span of finitely many of those directions.

More precisely, we studied orthogonal projection. We defined it to be the the unique vector denoted by $\operatorname{proj}_W \mathbf{x}$, such that

$\operatorname{proj}_W \mathbf{x} \in W$
$\mathbf{x} - \operatorname{proj}_W \mathbf{x} \in W^\perp$. i.e $\left\langle \mathbf{x} - \operatorname{proj}_W \mathbf{x}, \mathbf{w} \right\rangle = 0$ for all $\mathbf{w} \in W$.

We then showed that $\operatorname{proj}_W$ is a well-defined operator from $V$ to $W$. Furthermore, if $\{\mathbf{e}_1,\ldots,\mathbf{e}_n\}$ is an orthonormal basis for $W$, then

$$ \begin{align*} \boxed{ \operatorname{proj}_W(\mathbf{x}) = \sum_{k=1}^{n}\langle \mathbf{x},\mathbf{e}_k\rangle\mathbf{e}_k. } \end{align*} $$

Finally if we specifically let $V=L^2[0,2\pi]$ and define

$$ \begin{align*} W_N &= \operatorname{span}\left\{ e^{inx}: n=-N,\ldots,-1,0,1,\ldots,N \right\}. \end{align*} $$

With the normalized inner product

$$ \begin{align*} & \\[0.001em] \boxed{\langle f,g \rangle= \frac{1}{2\pi}\int_0^{2\pi}f(x)\overline{g(x)}\,dx,} \end{align*} $$

the functions $\{e^{inx}\}_{n=-N}^{N}$ form an orthonormal basis for $W_N$. Therefore, for every $f\in V$,

$$ \begin{align*} \boxed{ \operatorname{proj}_{W_N}f = \sum_{n=-N}^{N}\langle f,e^{inx}\rangle e^{inx} = \sum_{n=-N}^{N}\widehat{f}(n)e^{inx} = S_Nf(x).} \end{align*} $$

So now the next question to ask is whether these projections converge to $f$ as we include more frequencies.

Convergence in Inner Product Spaces

A natural question is to ask now for convergence. However, the notion of convergence in an inner product space needs to be defined.

Definition (1.0)

Let $(V,\langle\cdot,\cdot\rangle)$ be an inner product space. We say that a sequence $\mathbf{v}_n\in V$ converges to $\mathbf{v}\in V$ if for every $\epsilon>0$, we can find $N\in\mathbb{N}$ such that for all $n\ge N$,

$$ \|\mathbf{v}_n-\mathbf{v}\| < \epsilon. $$

Convergence here is more delicate than it might seem. For example, consider $V=\mathbb{Q}^n$ with the usual inner product. A Cauchy sequence need not converge to an element of $V$. This leads to the following definition.

Definition (2.0)

Let $(V,\langle\cdot,\cdot\rangle)$ be an inner product space. We say that $V$ is complete if every Cauchy sequence in $V$ converges to an element $\mathbf{v}\in V$. A complete inner product space is called a Hilbert space.

First why do we care? more on this after OH [TODO]

Examples

$\mathbb{R}^n$ and $\mathbb{C}^n$ are complete. For an infinite-dimensional inner product space, define

$$ \begin{align*} \ell^2(\mathbb{Z}) &= \left\{ (a_n)_{n\in\mathbb{Z}} : \sum_{n=-\infty}^{\infty}|a_n|^2<\infty \right\}. \end{align*} $$

We can define its inner product to be

$$ \begin{align*} \langle (a_n),(b_n)\rangle &= \sum_{n=-\infty}^{+\infty}a_n\overline{b_n}. \end{align*} $$

We can actually show that $\ell^2(\mathbb{Z})$ is complete. This is explored in here

The Problem with $C([0,2\pi])$

The next question to ask is whether $C([0,2\pi])$ with the supremum norm is complete. Unfortunately, the supremum norm doesn’t come from an inner product, so it doesn’t provide the orthogonality and orthogonal projections we need here. The next idea is to consider using the integral inner product. Unfortunately, $C([0,2\pi])$ is NOT complete under this norm. To see why check this note Issues with $C[0,2\pi]$

How to obtain a Complete Space?

To obtain a complete space, we take the completion of $C([0,2\pi])$ under the integral norm by introducing a space containing all possible limits of Cauchy sequences. Abstractly, this is called the completion but it’s not helpful. Fortunately, the Lebesgue integral gives a concrete description of this space.

Definition

The space $L^2([0,2\pi])$ consists of measurable functions whose squared absolute value has a finite Lebesgue integral: \[ L^2([0,2\pi]) = \left\{ f:[0,2\pi]\to\mathbb C: f\text{ is measurable and } \int_0^{2\pi}|f(x)|^2\,dx<\infty \right\}. \]

The $L^2$ spaces uses the norm

$$ \begin{align*} \|f\|_2 = \left(\frac{1}{2\pi}\int_0^{2\pi}|f(x)|^2\,dx\right)^{1/2}. \end{align*} $$

Also recall that $L^2([0,2\pi])$ is a Hilbert space because:

It is an inner product space.
It is complete under the norm induced by that inner product—every Cauchy sequence in that norm converges to an element of $L^2$.

Orthonormal Bases for Hilbert Space

So now what makes an orthonormal set an orthonormal basis? We have two requirements

Orthonormal bases for Hilbert space

We say that $\{\mathbf{e}_n\}$ is an orthonormal basis for the Hilbert space $H$ if:

Orthonormality. $ \langle \mathbf{e}_n,\mathbf{e}_m\rangle = \begin{cases} 0, & n\ne m,\\ 1, & n=m. \end{cases}$
Totality. For every $\mathbf{u}\in H$, $u = \sum_{n=1}^{\infty} \langle u,e_n \rangle e_n $. i.e. if we let $S_N = p_N(u) = \sum_{n=1}^{\infty} \langle u,e_n \rangle e_n $ where $S_N$ is the projection of $u$ onto $W_N$, then $$ \begin{align*} \left\|u- S_N\right\| \longrightarrow0. \end{align*} $$

The totality condition says that $u \in H$ can be approximated arbitrarily well by its projections onto the first $N$ vectors. It’s the infinite-dimensional version of the spanning condition. More precisely, let

$$ \begin{align*} \operatorname{span}\{e_n:n\ge1\} = \bigcup_{N=1}^{\infty}\operatorname{span}\{e_1,\ldots,e_N\}. \end{align*} $$

Then totality is equivalent to

$$ \begin{align*} \overline{\operatorname{span}\{e_n:n\ge1\}}=H. \end{align*} $$

Here:

Span contains all finite linear combinations of the $e_n$.
Closure also includes all limits of those combinations in the norm of $H$.

So every vector in $H$ can be approximated arbitrarily closely by finite linear combinations of the $e_n$. More on totality after OH [TODO]

An Orthonormal Basis for $L^2$

So far

We showed that the Fourier partial sum is an orthogonal projection: $S_Nf=\operatorname{proj}_{W_N}f$, where $W_N=\operatorname{span}\{e^{inx}:-N\le n\le N\}$.
We want to study convergence of Fourier series from this projection perspective: do these projections approach $f$ as $N\to\infty$?
We defined convergence in an inner product space using its induced norm: $v_n\to v$ if, for every $\epsilon>0$, there exists $N$ such that $\|v_n-v\|<\epsilon$ whenever $n\ge N$.
We saw that $C([0,2\pi])$ is not complete under the integral norm. Completeness guarantees that every Cauchy sequence has a limit inside the space.
We therefore pass to its completion, $L^2([0,2\pi])$. With the integral inner product, this is a Hilbert space.
We defined an orthonormal basis: an orthonormal set whose finite linear combinations are dense in the space. Equivalently, the projections onto its first $N$ vectors converge to every vector. (More on this)

Our next goal is to show that the Fourier exponentials form an orthonormal basis of $L^2([0,2\pi])$. We already know orthonormality under the normalized integral inner product. We still need totality. That is given a function $f \in L^2$, where its Fourier partial sum is

$$ \begin{align*} S_Nf(x)=\sum_{n=-N}^{N}\widehat f(n)e^{inx}. \end{align*} $$

We want to show that as $N \to \infty$, the $L^2$ error becomes arbitrarily small:

$$ \begin{align*} \|f-S_Nf\|_2\to0. \end{align*} $$

for every $f\in L^2([0,2\pi])$. Before we can show this, we need a few tools. Starting with the following lemma: <!–Lemma 0.3 provides a tool toward this goal. It relates the projection error to the coefficients through

$$ \begin{align*} \boxed{ \|f-S_Nf\|_2^2=\|f\|_2^2-\sum_{n=-N}^{N}|\widehat f(n)|^2 } \end{align*} $$

So let’s see Lemma 0.3 first: –>

Lemma 0.3

Let $\{\mathbf{e}_n\}_{n=1}^{\infty}$ be an orthonormal set in the Hilbert space $H$. Let \[ p_N(\mathbf{u})= \operatorname{proj}_{W_N}\mathbf{u} = \sum_{n=1}^{N}\langle \mathbf{u},\mathbf{e_n}\rangle \mathbf{e_n} \] where $W_N=\operatorname{span}\{\mathbf{e_1},\ldots, \mathbf{e_N}\}.$ Then, for every $\mathbf{u}\in H$ and every $N\in\mathbb{N}$, \begin{align*} \|\mathbf{u}\|^2 &= \left\| \mathbf{u} - p_N(\mathbf{u}) \right\|^2 + \left\| p_N(\mathbf{u}) \right\|^2 \\ &= \left\| \mathbf{u} - p_N(\mathbf{u}) \right\|^2 + \sum_{n=1}^{N} |\langle\mathbf{u},\mathbf{e}_n\rangle|^2. \end{align*} Furthermore, Bessel's inequality holds: \begin{align*} \sum_{n=1}^{\infty} |\langle\mathbf{u},\mathbf{e}_n\rangle|^2 &\le \|\mathbf{u}\|^2. \end{align*}

Proof. Note that the second line in the first property can be derived as follows

$$ \begin{align*} \|p_N(\mathbf{u})\|^2 &= \left\langle \sum_{j=1}^{N}\langle\mathbf{u},\mathbf{e}_j\rangle\mathbf{e}_j, \sum_{k=1}^{N}\langle\mathbf{u},\mathbf{e}_k\rangle\mathbf{e}_k \right\rangle \\ &= \sum_{j=1}^{N}\langle\mathbf{u},\mathbf{e}_j\rangle \left\langle \mathbf{e}_j, \sum_{k=1}^{N}\langle\mathbf{u},\mathbf{e}_k\rangle\mathbf{e}_k \right\rangle \quad \text{(Linearity in the first slot)} \\ &= \sum_{j=1}^{N}\sum_{k=1}^{N} \langle\mathbf{u},\mathbf{e}_j\rangle \overline{\langle\mathbf{u},\mathbf{e}_k\rangle} \langle\mathbf{e}_j,\mathbf{e}_k\rangle \\ &= \sum_{n=1}^{N} \langle\mathbf{u},\mathbf{e}_n\rangle \overline{\langle\mathbf{u},\mathbf{e}_n\rangle} \quad \text{($\langle\mathbf{e}_j,\mathbf{e}_k\rangle=0$ when $j \neq k$)} \\ &= \sum_{n=1}^{N}|\langle\mathbf{u},\mathbf{e}_n\rangle|^2. \tag{1} \end{align*} $$

Now let

$$ \begin{align*} \mathbf{u}=(\mathbf{u}-p_N(\mathbf{u}))+p_N(\mathbf{u}). \end{align*} $$

Recall that $p_N(u)$ is the projection of $u$ onto $W_N$ and by definition of the projection, we must have $\left\langle \mathbf{u} - p_N(\mathbf{u}), \mathbf{w} \right\rangle = 0$ for all $\mathbf{w} \in W$. But $p_N(u) \in W$. Hence

$$ \begin{align*} \left\langle \mathbf{u} - p_N(\mathbf{u}), p_N(\mathbf{u}) \right\rangle = 0 \end{align*} $$

Therefore, we can now apply the Pythagorean theorem to get

$$ \begin{align*} \|\mathbf{u}\|^2 &= \left\| \mathbf{u} - p_N(\mathbf{u}) \right\|^2 + \left\| p_N(\mathbf{u}) \right\|^2 \\ &= \left\| \mathbf{u} - p_N(\mathbf{u}) \right\|^2 + \sum_{n=1}^{N}|\langle\mathbf{u},\mathbf{e}_n\rangle|^2 \quad \text{(by (1))} \end{align*} $$

To show Bessel’s inequality, note that since $\|u-p_N(u)\|^2\ge0$, then by (2) we obtain

$$ \begin{align*} \sum_{n=1}^{N}|\langle u,e_n\rangle|^2 \le \|u\|^2 \qquad\text{for every }N. \end{align*} $$

These partial sums are increasing or staying the same because each term $|\langle u,e_n\rangle|^2$ is nonnegative. They are also bounded above by $\|u\|^2$. $u$ is fixed so $\|u\|^2$ is a finite constant that doesn’t depend on $N$. Hence by the Monotone Convergence Theorem for sequences, the partial sums are increasing and bounded above so they must converge so

$$ \begin{align*} \sum_{n=1}^{\infty}|\langle u,e_n\rangle|^2 = \lim_{N \to \infty} \sum_{n=1}^{N}|\langle u,e_n\rangle|^2 \le \|u\|^2 \end{align*} $$

Thus,

$$ \begin{align*} \boxed{\sum_{n=1}^{\infty}|\langle u,e_n\rangle|^2\le\|u\|^2.} \end{align*} $$

Note: so once we have an orthonormal set for inner product space, we can apply Bessel. (TODO: Why a Hilbert Space is needed? question for OH)

So now using this lemma we can show that

Fundamental Theorem of Hilbert Space

Suppose that $\{\mathbf{e}_n\}$ is an orthonormal set for the Hilbert space $H$. Then the following are equivalent.

(Completeness) $\langle \mathbf{u},\mathbf{e}_n\rangle=0$ for all $n \Longrightarrow \mathbf{u}=0$.
(Totality) $\displaystyle \mathbf{u}=\sum_{n=1}^{\infty} \langle\mathbf{u},\mathbf{e}_n\rangle\mathbf{e}_n$ for all $\mathbf{u}\in H$.
Equivalently, if we let $S_n = p_N(u) = \sum_{n=1}^{N} \langle\mathbf{u},\mathbf{e}_n\rangle\mathbf{e}_n $, then as $N \to \infty$ \[ \|\mathbf{u} - S_N\| \to 0 \] for all $u \in H$
(Parseval identity) $\displaystyle \sum_{n=1}^{\infty}\bigl|\langle\mathbf{u},\mathbf{e}_n\rangle\bigr|^2 =\|\mathbf{u}\|^2$

The proof of this theorem is in here

Parseval for Continuous Functions

So now suppose we’re given a function in $L^2([-\pi,\pi])$. We know we have an orthonormal set $\{e^{inx}\}_{n\in\mathbb{Z}}$. We also know it’s a Hilbert space so it’s complete. But to use Parseval, we still need to show (1) completeness or (2) totality in order to apply Parversal. It isn’t enough to just have a Hilbert space and an orthonormal set.

But this is another way, we state it below and then prove it!

Theorem 0.5 (Parseval for Continuous Functions)

Let $f$ be a continuous $2\pi$-periodic function. Then \[ \lim_{N\to\infty} \int_0^{2\pi} |f(x)-S_Nf(x)|^2\,dx = 0, \] and \[ \sum_{n=-\infty}^{\infty} |\widehat{f}(n)|^2 = \|f\|^2. \]

We explore the proof in here.

References

SFSU Fourier Analysis by Professor Chun-Kit Lai
Fourier Analysis: An Introduction by Elias M.Stein and Rami Shakarchi