IMC 2005 · Day 2 · P9

If $A$ is a nonzero symmetric matrix, then $\operatorname{trace}(A^2) = \operatorname{trace}(A^t A)$ is the sum of the squared entries of $A$ which is positive. So $V$ cannot contain any symmetric matrix but 0.

Denote by $S$ the linear space of all real $n \times n$ symmetric matrices; $\dim V = \frac{n(n+1)}{2}$.

Since $V \cap S = \{0\}$, we have $\dim V + \dim S \le n^2$ and thus $\dim V \le n^2 - \frac{n(n+1)}{2} = \frac{n(n-1)}{2}$.

The space of strictly upper triangular matrices has dimension $\frac{n(n-1)}{2}$ and satisfies the condition of the problem.

Therefore the maximum dimension of $V$ is $\frac{n(n-1)}{2}$.