Rotations in three-dimensional space: axis and angle

The matrix is block diagonal. The $2\times2$ block is the rotation in the $(x,y)$ plane, with eigenvalues $e^{\pm i\theta}$, and the final entry gives the eigenvalue $1$, with eigenvector $\mathbf e_z$. The trace is $1+2\cos\theta$. The eigenvalue $1$ corresponds to the rotation axis: the vectors along this axis are left invariant.

Using $RR^{\mathsf T}=\mathbf{1}$, we write $R-\mathbf{1}=R-RR^{\mathsf T}=R(\mathbf{1}-R^{\mathsf T})$. Thus

$$\det(R-\mathbf{1})=\det R\,\det(\mathbf{1}-R^{\mathsf T})=\det(\mathbf{1}-R)^{\mathsf T}=\det(\mathbf{1}-R)=(-1)^3\det(R-\mathbf{1}),$$

since $\det R=1$, the determinant of a transpose equals that of the original matrix, and multiplying a $3\times3$ matrix by $-1$ multiplies its determinant by $(-1)^3$. We obtain $\det(R-\mathbf{1})=-\det(R-\mathbf{1})$, hence $\det(R-\mathbf{1})=0$: $1$ is an eigenvalue. In odd dimension, a rotation always leaves a direction invariant; this is false in even dimension, as the plane rotation shows.

The axis is the eigenspace associated with the eigenvalue $1$, obtained by solving $R\mathbf v=\mathbf v$. The trace is independent of the basis and is therefore equal to $1+2\cos\theta$ in every basis. It follows that $\cos\theta=\frac{\tr R-1}{2}$, which determines the angle up to its sign. The sign depends on the orientation chosen for the axis; it can be fixed by calculating the image of a vector perpendicular to the axis.

The columns of $M$ are the vectors $\mathbf e_2$, $\mathbf e_3$, $\mathbf e_1$, which form an orthonormal basis: $M^{\mathsf T}M=\mathbf{1}$. Expanding along the first row gives the determinant $1\cdot(1\cdot1-0\cdot0)=1$. It is therefore a rotation. It maps $\mathbf e_1$ to $\mathbf e_2$, $\mathbf e_2$ to $\mathbf e_3$, and $\mathbf e_3$ to $\mathbf e_1$: it cyclically permutes the axes.

The equation $M\mathbf v=\mathbf v$ is $(v_3,v_1,v_2)=(v_1,v_2,v_3)$, hence $v_1=v_2=v_3$: the axis lies along $\mathbf n=\frac{1}{\sqrt3}(1,1,1)$. The trace is zero, so $\cos\theta=-\frac12$ and $\theta=\pm\frac{2\pi}{3}$. By symmetry, applying the rotation three times returns each axis to itself: $M^3=\mathbf{1}$, which is consistent with an angle of $2\pi/3$. Viewed from the tip of $\mathbf n$, the rotation takes $\mathbf e_1$ to $\mathbf e_2$ and then to $\mathbf e_3$ in the anticlockwise direction: it is the rotation through an angle $+2\pi/3$ about $\mathbf n$.

From the preceding results, the other two eigenvalues are $e^{\pm2i\pi/3}$, that is, $\omega$ and $\omega^2=\omega^*$. Seek an eigenvector of the form $(1,a,b)$ for the eigenvalue $\lambda$: the equation $M\mathbf v=\lambda\mathbf v$ is $(b,1,a)=\lambda(1,a,b)$, hence $b=\lambda$, $a=1/\lambda$, and the condition $a=\lambda b$ gives $\lambda^3=1$. For $\lambda=\omega^2$, we obtain $b=\omega^2$ and $a=\omega^{-2}=\omega$: the vector $\frac{1}{\sqrt3}(1,\omega,\omega^2)$. For $\lambda=\omega$, we obtain $\frac{1}{\sqrt3}(1,\omega^2,\omega)$. Together with the vector $\frac{1}{\sqrt3}(1,1,1)$, these vectors form an orthonormal basis of $\C^3$: this is the three-point discrete Fourier transform, which also arises in the study of cyclic molecules. The matrix $M$, which is real and orthogonal, is diagonalisable over $\C$ but not over $\R$.