Orthogonal projector onto a plane

The vectors $\ket a$ and $\ket b$ are not orthogonal, since $\braket ab=1$. We apply the Gram—Schmidt procedure. First, $\ket{e_1}=\frac{1}{\sqrt2}(1,-1,0)$. Next, $\braket{e_1}{b}=\frac{1}{\sqrt2}$, and

$$\ket b-\braket{e_1}{b}\ket{e_1}=(1,0,-1)-\tfrac12(1,-1,0)=\bigl(\tfrac12,\tfrac12,-1\bigr),$$

whose norm is $\sqrt{3/2}$, hence $\ket{e_2}=\frac{1}{\sqrt6}(1,1,-2)$. The matrices of the two rank-one projectors are

$$\begin{aligned} \ket{e_1}\bra{e_1}=\frac12\begin{pmatrix}1&-1&0\\-1&1&0\\0&0&0\end{pmatrix}, \qquad \ket{e_2}\bra{e_2}=\frac16\begin{pmatrix}1&1&-2\\1&1&-2\\-2&-2&4\end{pmatrix}, \end{aligned}$$

and their sum is

$$\begin{aligned} P=\frac13\begin{pmatrix}2&-1&-1\\-1&2&-1\\-1&-1&2\end{pmatrix}. \end{aligned}$$

The matrix $P$ is real and symmetric, so $P^\dagger=P$. Its trace is $\frac13(2+2+2)=2$, the dimension of $F$. For the square, for example, we calculate the first entry: $\frac19\bigl(2\cdot2+(-1)(-1)+(-1)(-1)\bigr)=\frac69=\frac23$, and the second: $\frac19\bigl(2\cdot(-1)+(-1)\cdot2+(-1)(-1)\bigr)=-\frac39=-\frac13$; the other entries follow by symmetry, and $P^2=P$. This may also be obtained without calculation, from $\braket{e_i}{e_j}=\delta_{ij}$.

The vectors $\ket a$ and $\ket b$ are orthogonal to $\ket n$, since the sum of their components is zero: $F$ is the plane orthogonal to $\ket n$. We verify that $P\ket n=\frac{1}{3\sqrt3}(2-1-1,...)=0$. Now $\ket n\bra n=\frac13J$, where $J$ is the matrix all of whose entries are equal to $1$, and $\mathbf{1}-\frac13J$ is exactly the matrix $P$. The projector onto $F$ is the complementary projector to the projector onto the orthogonal line: $(\ket{e_1},\ket{e_2},\ket n)$ is an orthonormal basis of $\R^3$, and the closure relation reads $\ket{e_1}\bra{e_1}+\ket{e_2}\bra{e_2}+\ket n\bra n=\mathbf{1}$.

The simplest method is to use $P=\mathbf{1}-\ket n\bra n$. We have $\braket nx=\frac{1}{\sqrt3}(1+2+3)=\frac{6}{\sqrt3}$, hence

$$P\ket x=\ket x-\braket nx\ket n=(1,2,3)-\frac{6}{3}(1,1,1)=(-1,0,1).$$

We verify that this vector does belong to $F$ (the sum of its components is zero). The orthogonal component is $\ket x-P\ket x=(2,2,2)$, and the distance from $\ket x$ to the plane is $\norm{(2,2,2)}=2\sqrt3$.

Write $\ket x-\ket y=\bigl(\ket x-P\ket x\bigr)+\bigl(P\ket x-\ket y\bigr)$. The first vector belongs to $F^\perp$, since $(\mathbf{1}-P)$ is the projector onto $F^\perp$, and the second belongs to $F$, since $P\ket x$ and $\ket y$ are in $F$. These two vectors are orthogonal, and the Pythagorean theorem gives

$$\norm{x-y}^2=\norm{x-Px}^2+\norm{Px-y}^2\geq\norm{x-Px}^2,$$

with equality if and only if $\ket y=P\ket x$. Orthogonal projection yields the best approximation of a vector by a vector in the subspace: this is what justifies, in infinite dimension, approximating a function by the first terms of its expansion in a Hilbert basis.