Orthonormal Bases

Bessel's inequality showed that the sum of squared coordinates of a vector against an orthonormal family never exceeds its squared length. This post asks when they spend all of it. An orthonormal family large enough that no length is wasted is a basis, and against it every vector becomes the sum of its coordinates, a single structure that contains Fourier series, eigenfunction expansions, and the Karhunen-Loeve expansion as special cases. The setting is a Hilbert space $H$ , where completeness is what lets an infinite orthogonal series converge to an actual vector [1], [2]. We assume $H$ is separable, so that a countable orthonormal family suffices, and we take the scalars to be real, so that every coordinate appears squared rather than modulus-squared; over $\mathbb{C}$ each $c_n^2$ becomes $\abs{c_n}^2$ and the proofs go through verbatim.

#Convergence of orthogonal series

The first question is which infinite combinations $\sum_n c_n e_n$ of an orthonormal family make sense. Pythagoras answers it exactly.

Proposition1

Let $(e_n)$ be orthonormal in $H$ and $(c_n)$ real scalars. The series $\sum_n c_n e_n$ converges in $H$ if and only if $\sum_n c_n^2<\infty$ , and then $\norm{\sum_n c_n e_n}^2=\sum_n c_n^2$ .

Proof

Let $s_N=\sum_{n=1}^N c_n e_n$ . For $M<N$ the increment $s_N-s_M=\sum_{n=M+1}^N c_n e_n$ has, by Pythagoras, squared norm $\sum_{n=M+1}^N c_n^2$ . Hence $(s_N)$ is Cauchy in $H$ if and only if the partial sums of $\sum_n c_n^2$ are Cauchy in $\R$ , that is if and only if $\sum_n c_n^2<\infty$ . Completeness of $H$ turns the Cauchy condition into convergence, and the norm identity follows from $\norm{s_N}^2=\sum_{n=1}^N c_n^2$ by letting $N\to\infty$ , the norm being continuous by Cauchy-Schwarz.

For any $x$ , Bessel's inequality gives $\sum_n\ip{x}{e_n}^2\le\norm{x}^2<\infty$ , so by Proposition 1 the series $\sum_n\ip{x}{e_n}e_n$ always converges, whether or not $(e_n)$ is a basis. Write $c_n(x)=\ip{x}{e_n}$ and $Px=\sum_n c_n(x)e_n$ for its limit. The residual $x-Px$ is orthogonal to every $e_n$ .

Lemma2

For every $x$ , the residual $x-Px$ satisfies $\ip{x-Px}{e_m}=0$ for all $m$ .

Proof

The map $y\mapsto\ip{y}{e_m}$ is continuous by Cauchy-Schwarz (the inner-product-spaces post), so it passes through the series, giving $\ip{Px}{e_m}=\sum_n c_n(x)\ip{e_n}{e_m}=c_m(x)$ by orthonormality. Then $\ip{x-Px}{e_m}=c_m(x)-c_m(x)=0$ .

#The basis conditions

An orthonormal family is a basis when it leaves no length unaccounted for. Several precise statements of that idea coincide.

Theorem3

For a countable orthonormal family $(e_n)$ in a Hilbert space $H$ , the following are equivalent. First, the closed linear span of $(e_n)$ is all of $H$ . Second, $x=\sum_n c_n(x)e_n$ for every $x$ . Third, the Parseval identity $\norm{x}^2=\sum_n c_n(x)^2$ holds for every $x$ . Fourth, the only vector orthogonal to every $e_n$ is $0$ . A family with these properties is an orthonormal basis.

Proof

For a finite family this is the elementary finite-dimensional statement, with $Px=\sum_{k=1}^N c_k(x)e_k$ the orthogonal projection onto the closed subspace $M=\mathrm{span}(e_1,\dots,e_N)$ and Parseval the finite Pythagoras, so the same $1\Rightarrow2\Rightarrow3\Rightarrow4\Rightarrow1$ cycle holds with finite sums; assume henceforth the family is infinite. Write $M$ for the closed span and recall $Px=\sum_n c_n(x)e_n\in M$ , the limit of partial sums lying in $M$ .

First implies second. If $M=H$ , then $x-Px\in H=M$ . By Lemma 2 the residual is orthogonal to every $e_n$ , hence to their span, hence to its closure $M$ by continuity. A vector of $M$ orthogonal to $M$ is orthogonal to itself, so $x-Px=0$ and $x=Px$ .

Second implies third. If $x=Px=\sum_n c_n(x)e_n$ , then Proposition 1 gives $\norm{x}^2=\sum_n c_n(x)^2$ .

Third implies fourth. If $\ip{x}{e_n}=0$ for all $n$ , the Parseval identity gives $\norm{x}^2=0$ , so $x=0$ .

Fourth implies first. For any $x$ the residual $x-Px$ is orthogonal to every $e_n$ by Lemma 2, so by the fourth condition $x-Px=0$ , putting $x=Px\in M$ . Thus $H=M$ .

Since $x-Px$ is orthogonal to every $e_n$ , it is orthogonal to their span and, by continuity of the inner product, to its closure $M$ ; as $Px\in M$ , the vector $Px$ is the orthogonal projection of $x$ onto $M$ .

The fourth condition is maximality. An orthonormal family is a basis exactly when it cannot be extended, since any unit vector orthogonal to all of it could be appended. The Parseval identity is the infinite-dimensional Pythagoras, splitting a vector's squared length into the squares of its coordinates with nothing left over, and the second condition is the expansion that makes the coordinates a faithful description of the vector.

#Existence by Gram-Schmidt

A basis is only useful if one exists. In a separable space it always does, manufactured from a dense sequence by orthonormalising it.

Theorem4

Every separable Hilbert space has an orthonormal basis, finite or countably infinite.

Proof

Separability gives a countable dense sequence $(x_n)$ . Discard each $x_n$ that lies in the span of its predecessors, leaving a subsequence $(v_k)$ that is linearly independent with the same span. Apply the Gram-Schmidt process, setting $e_1=v_1/\norm{v_1}$ and inductively

w_k=v_k-\sum_{j=1}^{k-1}\ip{v_k}{e_j}e_j,\qquad e_k=\frac{w_k}{\norm{w_k}}. \tag{1}

The vector $w_k$ is nonzero because $v_k$ is independent of $e_1,\dots,e_{k-1}$ , whose span equals that of $v_1,\dots,v_{k-1}$ , and $w_k$ is orthogonal to each $e_j$ with $j<k$ by construction, so $(e_k)$ is orthonormal. At every stage the span of $e_1,\dots,e_k$ equals that of $v_1,\dots,v_k$ , so the closed span of $(e_k)$ contains the closure of the span of $(x_n)$ , which is $H$ by density. By Theorem 3 the family $(e_k)$ is a basis.

#The isometry with l-squared

A basis identifies the abstract space with the concrete sequence space. Sending a vector to its coordinates is an isometry, and completeness makes it onto.

Theorem5

Let $(e_n)$ be an orthonormal basis of an infinite-dimensional separable Hilbert space $H$ . The coordinate map $U x=(c_n(x))_n$ is an isometric isomorphism of $H$ onto $\ell^2$ .

Proof

Linearity of $Ux$ is linearity of the inner product in $x$ . The Parseval identity of Theorem 3 reads $\norm{Ux}_{\ell^2}^2=\sum_n c_n(x)^2=\norm{x}^2$ , so $U$ preserves the norm, which makes it injective. For surjectivity take any $(a_n)\in\ell^2$ . Since $\sum_n a_n^2<\infty$ , Proposition 1 makes $x=\sum_n a_n e_n$ converge in $H$ , and $c_m(x)=\ip{x}{e_m}=a_m$ by the computation in Lemma 2, so $Ux=(a_n)$ . Thus $U$ is a norm-preserving linear bijection, an isometric isomorphism.

Every separable infinite-dimensional Hilbert space is therefore a copy of $\ell^2$ . A finite-dimensional space is the elementary case, a copy of Euclidean $\R^d$ under the coordinate map on a finite basis, where surjectivity is immediate and no convergence argument is needed. The choice of basis is the only freedom, and the right basis makes a problem transparent. The Fourier basis diagonalises translation and differentiation, and the eigenvectors of a compact self-adjoint operator, constructed in the spectral theorem to come, form the basis in which that operator becomes a diagonal multiplication. The Karhunen-Loeve expansion is exactly this last construction applied to the covariance operator of a process, producing the orthonormal basis in which the process has uncorrelated coordinates. A basis is the coordinate system in which the structure of the problem is laid bare.

[1]

W. Rudin, Functional Analysis, 2nd ed. McGraw-Hill, 1991.

[2]

J. B. Conway, A Course in Functional Analysis, 2nd ed. Springer, 1990.

Explore connections

see in the atlas

referenced by (3)

cite

@misc{orthonormal-bases,
  author = {Zac Kienzle},
  title  = {Orthonormal Bases},
  year   = {2026},
  month  = {05},
  url    = {https://zackienzle.com/blog/orthonormal-bases}
}

#Convergence of orthogonal series

The first question is which infinite combinations $\sum_n c_n e_n$ of an orthonormal family make sense. Pythagoras answers it exactly.

Proposition1

Let $(e_n)$ be orthonormal in $H$ and $(c_n)$ real scalars. The series $\sum_n c_n e_n$ converges in $H$ if and only if $\sum_n c_n^2<\infty$ , and then $\norm{\sum_n c_n e_n}^2=\sum_n c_n^2$ .

Proof

Lemma2

For every $x$ , the residual $x-Px$ satisfies $\ip{x-Px}{e_m}=0$ for all $m$ .

Proof

#The basis conditions

An orthonormal family is a basis when it leaves no length unaccounted for. Several precise statements of that idea coincide.

Theorem3

Proof

Second implies third. If $x=Px=\sum_n c_n(x)e_n$ , then Proposition 1 gives $\norm{x}^2=\sum_n c_n(x)^2$ .

Third implies fourth. If $\ip{x}{e_n}=0$ for all $n$ , the Parseval identity gives $\norm{x}^2=0$ , so $x=0$ .

Fourth implies first. For any $x$ the residual $x-Px$ is orthogonal to every $e_n$ by Lemma 2, so by the fourth condition $x-Px=0$ , putting $x=Px\in M$ . Thus $H=M$ .

#Existence by Gram-Schmidt

A basis is only useful if one exists. In a separable space it always does, manufactured from a dense sequence by orthonormalising it.

Theorem4

Every separable Hilbert space has an orthonormal basis, finite or countably infinite.

Proof

w_k=v_k-\sum_{j=1}^{k-1}\ip{v_k}{e_j}e_j,\qquad e_k=\frac{w_k}{\norm{w_k}}. \tag{1}

#The isometry with l-squared

A basis identifies the abstract space with the concrete sequence space. Sending a vector to its coordinates is an isometry, and completeness makes it onto.

Theorem5

Let $(e_n)$ be an orthonormal basis of an infinite-dimensional separable Hilbert space $H$ . The coordinate map $U x=(c_n(x))_n$ is an isometric isomorphism of $H$ onto $\ell^2$ .

Proof

[1]

W. Rudin, Functional Analysis, 2nd ed. McGraw-Hill, 1991.

[2]

J. B. Conway, A Course in Functional Analysis, 2nd ed. Springer, 1990.

Explore connections

see in the atlas

referenced by (3)

cite

@misc{orthonormal-bases,
  author = {Zac Kienzle},
  title  = {Orthonormal Bases},
  year   = {2026},
  month  = {05},
  url    = {https://zackienzle.com/blog/orthonormal-bases}
}