When $A$ has complex eigenvalues $\alpha \pm \beta i$, the eigenvector method still works, but it produces complex solutions. As with complex roots of one equation, their real and imaginary parts are the real solutions, and they spiral.
Key idea
If $A\mathbf{v} = \lambda\mathbf{v}$ with $\lambda = \alpha + \beta i$ and $\mathbf{v} = \mathbf{a} + i\mathbf{b}$, then $$\mathbf{u}_1 = e^{\alpha t}(\mathbf{a}\cos\beta t - \mathbf{b}\sin\beta t),$$ $$\mathbf{u}_2 = e^{\alpha t}(\mathbf{a}\sin\beta t + \mathbf{b}\cos\beta t)$$ are real solutions, and $\mathbf{x} = c_1\mathbf{u}_1 + c_2\mathbf{u}_2$. Trajectories spiral in if $\alpha < 0$, out if $\alpha > 0$, and close up if $\alpha = 0$.
Why it works
$e^{\lambda t}\mathbf{v}$ $= e^{\alpha t}(\cos\beta t + i\sin\beta t)(\mathbf{a}$ $+ i\mathbf{b})$. Multiplying out, the real part is $\mathbf{u}_1$ and the imaginary part is $\mathbf{u}_2$. Because $A$ is real, the equation $(\mathbf{u}_1 + i\mathbf{u}_2)' = A(\mathbf{u}_1 + i\mathbf{u}_2)$ splits into a real part and an imaginary part that must each balance, so $\mathbf{u}_1$ and $\mathbf{u}_2$ are both solutions. The direction of rotation is easy to read without any of this: at the point $(1, 0)$ the velocity is the first column of $A$, so if $a_{21} > 0$ the motion turns upward there, counterclockwise.
Worked example: Eigenvalues and rotation
Describe the trajectories of $\mathbf{x}' = \begin{bmatrix} -1 & -2 \\ 2 & -1 \end{bmatrix}\mathbf{x}$.
- $\tau = -2$, $\Delta = 5$: $\lambda^{2} + 2\lambda + 5 = 0$, so $\lambda = -1 \pm 2i$.
- $\alpha = -1 < 0$: they spiral in.
- At $(1, 0)$ the velocity is $(-1, 2)$: moving up, so counterclockwise.
Answer: spirals in, counterclockwise
Worked example: Solve with an initial condition
Solve the same system with $\mathbf{x}(0) = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$.
- For $\lambda = -1 + 2i$: $(A - \lambda I)\mathbf{v} = 0$ has first row $-2i v_1 - 2v_2 = 0$, so $\mathbf{v} = \begin{bmatrix} 1 \\ -i \end{bmatrix}$: $\mathbf{a} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} 0 \\ -1 \end{bmatrix}$.
- $\mathbf{u}_1 = e^{-t}\begin{bmatrix} \cos 2t \\ \sin 2t \end{bmatrix}$ and $\mathbf{u}_2 = e^{-t}\begin{bmatrix} \sin 2t \\ -\cos 2t \end{bmatrix}$.
- At $t = 0$: $c_1\begin{bmatrix} 1 \\ 0 \end{bmatrix} + c_2\begin{bmatrix} 0 \\ -1 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$, so $c_1 = 1$, $c_2 = 0$.
Answer: $x_1 = e^{-t}\cos 2t$, $x_2 = e^{-t}\sin 2t$: a point circling counterclockwise as it shrinks toward the origin
Worked example: A center
Describe $\mathbf{x}' = \begin{bmatrix} 0 & 4 \\ -1 & 0 \end{bmatrix}\mathbf{x}$.
- $\tau = 0$, $\Delta = 4$: $\lambda = \pm 2i$, so $\alpha = 0$.
- At $(1, 0)$ the velocity is $(0, -1)$: clockwise.
Answer: closed loops (ellipses), traversed clockwise with period $\pi$
Common mistake
Judging stability from $\beta$ or from the eigenvector. Only the real part $\alpha$ (half the trace) decides spiral in or out; $\beta$ sets how fast it turns. And the direction of rotation is not in the eigenvalue's sign of $\beta$: check the velocity at one point.