Solution: From the hypotesis, AB(BA−In)=On. Based on Sylvester's inequality for ranks, it follows that
rank(AB)+rank(BA−In)⩽n+rank(AB(BA−In))=n.(1)
Also, it is true in general that
rank(AB−In)=rank(BA−In),(2)
so
rank(AB−In)+rank(AB)⩽n.(3)
But
Ker(AB−In)⊆ImAB, so
rank(AB−In)+rank(AB)=n+rank((AB)2−AB)(4)
(this is the equality case in Sylvester's inequality for the matrices
AB−In and
AB). Combining (3) and (4), it follows that
(AB)2=AB.
Using now the identity from the hypotesis and (AB)2=AB, we obtain
(AB−BA)2(AB−BA)3(AB−BA)4=(AB)2+(BA)2−AB2A−BA2B=(BA)2−BA2B=−BA(AB−BA)=−BA(AB−BA)2=(BA)2(AB−BA)=(BA)2(AB−BA)2=−(BA)3(AB−BA)=−B(AB)2A(AB−BA)=−B(AB)A(AB−BA)=−(BA)2(AB−BA)=−(AB−BA)3,
hence
(AB−BA)4=−(AB−BA)3.
Let λ be any eigenvalue of AB−BA. Then the previous identity implies λ4=−λ3, so λ∈{0,−1}. Since Tr(AB−BA)=0, it follows that all eigenvalues of AB−BA must be 0. Then (AB−BA)n=On, and hence, (AB−BA)3=On.
Transcribed from the official solutions, pages 1–2. This transcription has not been checked a second time; if in doubt, look at the original.