In this specific note, we will discuss the second consequences the Inner Product: Orthogonality.
First define
Definition (Orthogonal)
We say that two vectors \(\mathbf{u},\mathbf{v}\in V\) are orthogonal if
$$
\begin{align*}
\langle \mathbf{u},\mathbf{v}\rangle = 0.
\end{align*}
$$
This gives rise to the following theorem:
Theorem (Pythagorean theorem)
Suppose that \(u\) and \(v\) are orthogonal. Then
\[
\|u+v\|^2 = \|u\|^2 + \|v\|^2.
\]
Proof. By the definition of the induced norm,
$$
\begin{align*}
\|u+v\|^2
&= \langle u+v,u+v\rangle \\
&= \langle u,u\rangle + \langle u,v\rangle
+ \langle v,u\rangle + \langle v,v\rangle \\
&= \|u\|^2 + \|v\|^2.
\end{align*}
$$
The two cross terms vanish because \(\langle u,v\rangle=0\) and \(\langle v,u\rangle=\overline{\langle u,v\rangle}=0\).
Now suppose we’re in \(\mathbb{R}^2\) and we want to find the length of the vector \(\alpha v\). In other words, the projection of \(w\) onto \(v\) in
Then we can note that \(w - \alpha v\) is actually orthogonal to \(\alpha v\). Hence we obtain
$$
\begin{align*}
\langle w - \alpha v, v\rangle &= 0 \\
\langle w,v \rangle - \alpha \langle v, v\rangle &= 0 \\
\alpha &= \frac{\langle w,v \rangle}{\langle v, v\rangle}
\end{align*}
$$
This leads to the following definition
Definition (Projection)
Let \(w, v \in V\). The orthogonal projection of \(w\) onto \(v\) is the vector
\[
\operatorname{proj}_v w = \Pi_{v}(w) = \frac{\langle w,v \rangle}{\|v\|^2}v
\]
which then leads to an important theorem for any inner product space:
Theorem (Cauchy–Schwarz inequality)
Let \(V\) be an inner product space. Then for any \(v,w\in V\),
\[
|\langle v,w \rangle|\leq \|v\|\|w\|.
\]
Equality holds if and only if \(v\) and \(w\) are linearly dependent (parallel to each other).
Proof. The inequality holds immediately if \(v=0\). Assume that \(v\neq 0\). Define
$$
\begin{align*}
\alpha = \frac{\langle w,v\rangle}{\|v\|^2}
\end{align*}
$$
Recall in the figure above which we’ll draw again here
So \(\alpha v = \frac{\langle w,v\rangle}{\|v\|^2}v\) is the projection of \(w\) onto \(v\). Now observe what happens if we remove that projection component from \(w\)
$$
\begin{align*}
\left\langle w-\alpha v, v\right\rangle &= \langle w,v\rangle - \left\langle \alpha v,v\right\rangle \\
&= \langle w,v\rangle - \alpha \langle v,v\rangle \\
&= \langle w,v\rangle
- \frac{\langle w,v\rangle}{\|v\|^2}\|v\|^2 \\
&= \langle w,v\rangle-\langle w,v\rangle \\
&= 0.
\end{align*}
$$
Note that \(w - \alpha v\) is also orthogonal to \(\alpha v\) since
$$
\begin{align*}
\langle w-\alpha v,\alpha v\rangle
&=\overline{\alpha}\,\langle w-\alpha v,v\rangle\\
&=\overline{\alpha}\bigl(\langle w,v\rangle-\alpha\langle v,v\rangle\bigr)\\
&=\overline{\alpha}\left(\langle w,v\rangle-
\frac{\langle w,v\rangle}{\|v\|^2}\|v\|^2\right)\\
&=0.
\end{align*}
$$
Since orthogonality is defined by having an inner product equal to zero, this establishes that \(w - \alpha v\) and \(\alpha v\) are orthogonal in any inner product space even when we can’t draw the vectors Re-write the vector \(w\) to be
$$
\begin{align*}
w= w - \alpha v + \alpha v
\end{align*}
$$
Since \(w - \alpha v\) and \(\alpha v\) orthogonal, then by the Pythagorean theorem,
$$
\begin{align*}
\|w\|^2
&=\|w-\alpha v\|^2+\|\alpha v\|^2\\
&=\|w-\alpha v\|^2+|\alpha|^2\|v\|^2\\
&=\|w-\alpha v\|^2+
\frac{|\langle w,v\rangle|^2}{\|v\|^2}\\
&\ge \frac{|\langle w,v\rangle|^2}{\|v\|^2}.
\end{align*}
$$
Multiplying by \(\|v\|^2\) and taking square roots gives
$$
\begin{align*}
|\langle w,v\rangle|^2 &\leq \|w\|^2\|v\|^2\\
|\langle w,v\rangle| &\leq \|w\|\|v\|.
\end{align*}
$$
For \(v\neq 0\), equality holds exactly when
$$
\begin{align*}
\|w-\alpha v\|^2=0,
\end{align*}
$$
or equivalently, \(w=\alpha v\). Thus
$$
\begin{align*}
w=\frac{\langle w,v\rangle}{\|v\|^2}v.
\end{align*}
$$
Hence equality holds precisely when \(w\) and \(v\) are linearly dependent. When \(v=0\), equality also holds and the vectors are linearly dependent. \(\square\)
With the Cauchy–Schwarz inequality, we can define the angle between two nonzero vectors \(u,v\) in a real inner product space \(V\) by
$$
\cos\theta:=\frac{\langle u,v\rangle}{\|u\|\|v\|},
\qquad \theta\in[0,\pi].
$$
Cauchy–Schwarz ensures that this ratio lies in \([-1,1]\). Moreover, on \(\mathbb{R}^n\), the Cauchy–Schwarz inequality becomes
$$
\left|\sum_{i=1}^{n}u_i v_i\right| \leq \left(\sum_{i=1}^{n}u_i^2\right)^{1/2} \left(\sum_{i=1}^{n}v_i^2\right)^{1/2}.
$$
and on \(C[a,b]\), it becomes
$$
\left|\int_a^b f(x)\overline{g(x)}\,dx\right| \leq
\left(\int_a^b |f(x)|^2\,dx\right)^{1/2}
\left(\int_a^b |g(x)|^2\,dx\right)^{1/2}.
$$
For real-valued functions, \(\overline{g(x)}=g(x)\).
Example
Example. On \(C_{\mathrm{per}}[0,2\pi]\), with inner product
$$
\langle f,g\rangle=\int_0^{2\pi}f(x)\overline{g(x)}\,dx,
$$
If define the family \(f_n(x)=e^{inx}\), indexed by integers. Then \(f_n\) and \(f_m\) are two members of that family, and they’re orthogonal when \(n \neq m\). Precisely,
$$
\langle f_n,f_m\rangle
=\int_0^{2\pi}e^{i(n-m)x}\,dx
=0
\qquad (n\neq m).
$$
References
- SFSU Fourier Analysis by Professor Chun-Kit Lai
- Fourier Analysis: An Introduction by Elias M.Stein and Rami Shakarchi