In this specific note, we will discuss the second consequences the Inner Product: Orthogonality.
First define
Definition (Orthogonal)
We say that two vectors \(\mathbf{u},\mathbf{v}\in V\) are orthogonal if
$$
\begin{align*}
\langle \mathbf{u},\mathbf{v}\rangle = 0.
\end{align*}
$$
This gives rise to the following theorem:
Theorem (Pythagorean theorem)
Suppose that \(u\) and \(v\) are orthogonal. Then
\[
\|u+v\|^2 = \|u\|^2 + \|v\|^2.
\]
Proof. By the definition of the induced norm,
$$
\begin{align*}
\|u+v\|^2
&= \langle u+v,u+v\rangle \\
&= \langle u,u\rangle + \langle u,v\rangle
+ \langle v,u\rangle + \langle v,v\rangle \\
&= \|u\|^2 + \|v\|^2.
\end{align*}
$$
The two cross terms vanish because \(\langle u,v\rangle=0\) and \(\langle v,u\rangle=\overline{\langle u,v\rangle}=0\).
Now suppose we’re in \(\mathbb{R}^2\) and we want to find the length of the vector \(\alpha v\). In other words, the projection of \(w\) onto \(v\) in
Then we can note that \(w - \alpha v\) is actually orthogonal to \(\alpha v\). Hence we obtain
$$
\begin{align*}
\langle w - \alpha v, v\rangle &= 0 \\
\langle w,v \rangle - \alpha \langle v, v\rangle &= 0 \\
\alpha &= \frac{\langle w,v \rangle}{\langle v, v\rangle}
\end{align*}
$$
This leads to the following definition
Definition (Projection)
Let \(w, v \in V\). The orthogonal projection of \(w\) onto \(v\) is the vector
\[
\operatorname{proj}_v w = \Pi_{v}(w) = \frac{\langle w,v \rangle}{\|v\|^2}v
\]
which then leads to an important theorem for any inner product space:
Theorem (Cauchy–Schwarz inequality)
Let \(V\) be an inner product space. Then for any \(v,w\in V\),
\[
|\langle v,w \rangle|\leq \|v\|\|w\|.
\]
Equality holds if and only if \(v\) and \(w\) are linearly dependent (parallel to each other).
Proof. The inequality holds immediately if \(v=0\). Assume that \(v\neq 0\). Define
$$
\begin{align*}
u &= w - \operatorname{proj}_v w \\
&= w-\frac{\langle w,v\rangle}{\|v\|^2}v \tag{1}
\end{align*}
$$
Recall in the figure above which we’ll draw again here
So \(\frac{\langle w,v\rangle}{\|v\|^2}v\) is the projection of \(w\) onto \(v\). We’re removing this component of \(w\) in \(v\). Hence we get a vector \(u\) that is perpendicular to \(v\). This happens precisely since
$$
\begin{align*}
\langle u,v\rangle
&= \left\langle w-\frac{\langle w,v\rangle}{\|v\|^2}v, v\right\rangle \\
&= \langle w,v\rangle
- \left\langle \frac{\langle w,v\rangle}{\|v\|^2}v,v\right\rangle \\
&= \langle w,v\rangle
- \frac{\langle w,v\rangle}{\|v\|^2}\langle v,v\rangle \\
&= \langle w,v\rangle
- \frac{\langle w,v\rangle}{\|v\|^2}\|v\|^2 \\
&= \langle w,v\rangle-\langle w,v\rangle \\
&= 0.
\end{align*}
$$
Since orthogonality is defined by having inner product zero, this establishes \(u\perp v\) in any inner product space even when we can’t draw the vectors. Use (1) to re-write
$$
\begin{align*}
w= u + \frac{\langle w,v\rangle}{\|v\|^2}v
\end{align*}
$$
and by the Pythagorean theorem,
$$
\begin{align*}
\|w\|^2
&=\left\|u+\frac{\langle w,v\rangle}{\|v\|^2}v\right\|^2\\
&=\|u\|^2+\frac{|\langle w,v\rangle|^2}{\|v\|^4}\|v\|^2 \quad \text{($u$ and $\operatorname{proj}_v w$ are orthogonal)} \\
&=\|u\|^2+\frac{|\langle w,v\rangle|^2}{\|v\|^2}\\
&\geq \frac{|\langle w,v\rangle|^2}{\|v\|^2}.
\end{align*}
$$
Multiplying by \(\|v\|^2\) and taking square roots gives
$$
\begin{align*}
|\langle w,v\rangle|^2 &\leq \|w\|^2\|v\|^2\\
|\langle w,v\rangle| &\leq \|w\|\|v\|.
\end{align*}
$$
For \(v\neq 0\), equality holds exactly when \(\|u\|^2=0\), or equivalently, \(u=0\). By (1), this means
$$
w=\frac{\langle w,v\rangle}{\|v\|^2}v.
$$
Thus equality holds precisely when \(w\) and \(v\) are linearly dependent. When \(v=0\), equality also holds and the vectors are linearly dependent. \(\square\)
With the Cauchy–Schwarz inequality, we can define the angle between two nonzero vectors \(u,v\) in a real inner product space \(V\) by
$$
\cos\theta:=\frac{\langle u,v\rangle}{\|u\|\|v\|},
\qquad \theta\in[0,\pi].
$$
Cauchy–Schwarz ensures that this ratio lies in \([-1,1]\). Moreover, on \(\mathbb{R}^n\), the Cauchy–Schwarz inequality becomes
$$
\left|\sum_{i=1}^{n}u_i v_i\right| \leq \left(\sum_{i=1}^{n}u_i^2\right)^{1/2} \left(\sum_{i=1}^{n}v_i^2\right)^{1/2}.
$$
and on \(C[a,b]\), it becomes
$$
\left|\int_a^b f(x)\overline{g(x)}\,dx\right| \leq
\left(\int_a^b |f(x)|^2\,dx\right)^{1/2}
\left(\int_a^b |g(x)|^2\,dx\right)^{1/2}.
$$
For real-valued functions, \(\overline{g(x)}=g(x)\).
Example
Example. On \(C_{\mathrm{per}}[0,2\pi]\), with inner product
$$
\langle f,g\rangle=\int_0^{2\pi}f(x)\overline{g(x)}\,dx,
$$
If define the family \(f_n(x)=e^{inx}\), indexed by integers. Then \(f_n\) and \(f_m\) are two members of that family, and they’re orthogonal when \(n \neq m\). Precisely,
$$
\langle f_n,f_m\rangle
=\int_0^{2\pi}e^{i(n-m)x}\,dx
=0
\qquad (n\neq m).
$$
References
- SFSU Fourier Analysis by Professor Chun-Kit Lai
- Fourier Analysis: An Introduction by Elias M.Stein and Rami Shakarchi