cf. the "Proposition" in see here We obtain that $A,B$ are simultaneously triangularizable.

Probably there is another solution without simultaneously triangularizable. Because it is a problem grade 11. others results RMO 2014 https://artofproblemsolving.com/community/c7h615732p3665759 again RMO2014 https://artofproblemsolving.com/community/c7h1811287p12067332

loup blanc wrote: cf. the "Proposition" in see here We obtain that $A,B$ are simultaneously triangularizable. Using this result, we can get $det(A) = det(B)$. To conclude the solution, note that $\left(A-xI_n\right)^2 + \left(B - xI_n\right)^2 = 2\left(A-xI_n\right)\left(B-xI_n\right)$ and we can argue similarly $det(A-xI_n) = det(B-xI_n)$.

@Removable , if we know that $A,B$ are S.T, then we immediately obtain that $A,B$ have same eigenvalues (cf. reference above). What is more interesting is that you saw that, to get the considered result, it suffices to show that $\det(A)=\det(B)$.

Yes, that is an elementary manipulation that I use. But just like @Moubinool said that this is Grade 11 level olympiad, so is there any elementary manipulation to arrive to the result $det(A) = det(B)$ without deploy a heavy machinery?

Let's prove that $\det(A)=\det(B)$ by induction on $n$ ($n=1$ is trivial). The hypothesis implies $A^2-AB=AB-B^2$, or $A(A-B)=(A-B)B$, so we are done if $A-B$ is invertible. If $A-B$ is singular, then consider $W=\ker(A-B)$. $W$ is both $A$-invariant and $B$-invariant, since if $Av=Bv=w$, then $Aw+Bw=A^2v+B^2v=2ABv=2Aw$, i.e. $Aw=Bw$. Then, take a basis of $\mathbb{C}^n$ obtained extending a basis of $W$. In such a basis $A$ and $B$ are represented by matrices of the form $\begin{pmatrix}C&A_1\\0&A_2\end{pmatrix}$, $\begin{pmatrix}C&B_1\\0&B_2\end{pmatrix}$ respectively. Iposing the hypothesis gives $A_2^2+B_2^2=2A_2B_2$, so by induction $\det (A_2)=\det (B_2)$ and thus $\det(A)=\det(C)\det(A_2)=\det(C)\det(B_2)=\det(B)$.

very nice solution Acridian9

Here is a solution that is basically Acridian9's, but more fleshed out. Claim: [Shifting by $I$] $A+kI, B+kI$ satisfies the condition in the problem. This claim allows us to select $A,B$ that are invertible. This is a translation of the characteristic polynomial $\chi_A(x) = \det(A-xI)$ Claim: [Reduction to Linear Maps] For any invertible matrix $P$, $(PAP^{-1}, PBP^{-1})$ also satisfies the condition in the problem. Proof: Clear. The purpose of this move deserves more attention. Note that $PAP^{-1}$ can be seen as $A$ in a different basis of $\mathbb{C}^n$. Same with $PBP^{-1}$ with respect to $B$, but with respect to the same basis. So basically, we should think of $A,B$ as linear transformations instead of just matrices. Induct on $n$. Rearrange the condition to $A(A-B)=(A-B)B$. Let $V=Ker(A-B) \subset \mathbb{C}^n$. If $\dim V = 0$ then $A-B$ is invertible, so $A,B$ are similar to each other and this is clear. If $\dim V = n$ then $A-B$ is the zero matrix, so we are also done.

Suppose $0 < \dim V < n$. Let $k=\dim V$ and $\mathcal{B}=\{v_1,\cdots,v_k\}$ be a basis of $V$. Extend this to $v_1,\cdots,v_n$, a basis of $\mathbb{C}^n$. Note $V$ is $B$-invariant because if $(A-B)v = 0$ then $0=A(A-B)v = (A-B)(Bv) $. Since $Bv \in V, Av=Bv\in V$, so $V$ is also $A$-invariant. In fact, if I treat $A',B'$ as linear maps with respect to $A,B$ by $A'(e_j) = \sum_{k=1}^n a_{k,j}e_k$ and $(e_i)_{i=1}^n$ is the canonical basis. then it is intuitive to consider $_{\mathcal{B}} [A']_{\mathcal{B}}$ definition

and divide it into block matrices. This implies I can divide $A$ into blocks $$A=\begin{bmatrix} C & D \\ 0 & E \end{bmatrix}$$Where $C$ is a $k\times k$ matrix, $0$ is the zero $(n-k)\times k$ matrix, $D$ is a $k\times (n-k)$ matrix and $E$ is a $(n-k)\times (n-k)$ matrix. In fact, if we restrict $A$'s domain to $V$, then $A|_V=B|_V$. Thus, $$B=\begin{bmatrix} C & F \\ 0 & G \end{bmatrix}$$Where $C$ is a $k\times k$ matrix, $0$ is the zero $(n-k)\times k$ matrix, $F$ is a $k\times (n-k)$ matrix and $G$ is a $(n-k)\times (n-k)$ matrix. By block matrix multiplication, we can verify that $E^2+G^2=2EG$. This implies that $\chi_E(x) = \chi_G(x)$. Thus, $$\chi_A(x) = \chi_C(x) \chi_E(x) = \chi_C(x)\chi_G(x) = \chi_B(x)$$

Using the Acridian9's method in post#9, we can directly deduce that $A,B$ are simult. triangularizable (ST). Indeed, we keep the first 10 lines ... by induction $A_2,B_2$ are ST under a change of basis $Q$. Let $P$ be a change of basis that triangularizes $C$; then $A,B$ are ST under the change of basis $diag(P,Q)$. $\square$ We may assume that $A,B$ are triangular with diagonals $(a_i),(b_i)$. Then $a_i=b_i$, $A,B$ have same eigenvalues and, moreover, $A-B$ is nilpotent. When $n=2$, the solutions satisfy $AB=BA$. Yet, when $n=3$, there are solutions s.t. $AB\not= BA$. For example $A=\begin{pmatrix}0&1&u\\0&0&3\\0&0&0\end{pmatrix},B=\begin{pmatrix}0&-1&v\\0&0&1\\0&0&0\end{pmatrix}$. EDIT. The above example was not chosen at random $\textbf{Exercise.}$ If $A,B\in M_3(\mathbb{C})$, $A^2+B^2=2AB$ and $AB\not= BA$, then show that $A=uI+N,B=uI+M$, where $N,M$ are nilpotent and $N^2+M^2=2NM,NM\not= MN$.