Let us see that $AB-BA=O_n$. Not sure, but for 11th grade, are we allowed to use Jordan normal form and assume without loss of generality that $C$ is diagonalized with $k$ zeros and $n-k$ ones on the main diagonal? If $k=0$, that is $C=I_n$, then $C^*=I_n$ and if $AB-BA=I_n$ taking trace from both sides we obtain $0=n$, which is a contradiction. Let $k=1$. Then adjoint matrix is $E_{11}$, that has one at the intersection of the first row and first column and zero everywhere else. However, it is never equal to $AB-BA$. Indeed, taking trace from $E_{11}=AB-BA$ we have $1=0$, which is again a contradiction. The adjoint matrix for the case $k\geq 2$ is just a zero matrix, since there will always be a row (or column) made of zeros in any $(n-1)\times (n-1)$ submatrix. So $O_n=AB-BA$ and we are done.

rustam wrote: Not sure, but for 11th grade, are we allowed to use Jordan normal form and assume without loss of generality that $C$ is diagonalized with $k$ zeros and $n-k$ ones on the main diagonal? Normally, students shouldn't know about these properties until they go to university in our country. But most of those who take part in the olympiad learn on their own or are being taught about them from as early as high school. Anyway, the problem has an elementary solution too.

Catalin wrote: Anyway, the problem has an elementary solution too. I was afraid of that Well, would love to see the elementary solution, since I cannot unthink of the diagonalization argument. Is $CC^*=C^*C=\det C\cdot I_n$ would be of any use?

Here is the official solution: First we show that $C$ cannot be invertible. Indeed, from $C^2=C$ we would get $C=I_n, $ hence $AB-BA=C^*=I_n,$ which gives a contradiction after considering traces. Thus $\operatorname{rank}C \leq n-1.$ $\: \bullet \: \operatorname{rank}C \leq n-2.$ Then $AB-BA = C^* = 0,$ so $(AB-BA)^2 = 0.$ $\: \bullet \: \operatorname{rank}C = n-1.$ Then $C^* \neq 0$ and since $C \cdot C^* = 0,$ from Sylvester's inequality we will have $$1 \leq \operatorname{rank}C^* = \operatorname{rank}C + \operatorname{rank}C^* - n +1 \leq \operatorname{rank}(C\cdot C^*)+1 = 1$$hence $\operatorname{rank}C^* = 1.$ Then $(AB-BA)^2 = \operatorname{tr}(AB-BA) \cdot (AB-BA)=0.$

Catalin wrote: hence $\operatorname{rank}C^* = 1.$ Then $(AB-BA)^2 = \operatorname{tr}(AB-BA) \cdot (AB-BA)=0.$ Why is that? Isn't it $(AB-BA)^n=Tr(AB-BA)(AB-BA)^{n-1}$ using Cayley-Hamilton and Cauchy-Binet formula?

rustam wrote: These high-school students are supposed to be that tricky, huh? Yes, unfortunately they have to know way more complicated properties than this particular one...