(proposed by Mike Daas, Universiteit Leiden)
Hint: Write a=eix and b=eiy, and transform the RHS to a product.
Solution 1. Write a=eix and b=eiy for some x,y∈[0,2π). Using Euler's formula, and the well-known identities
sinx+siny=2sin2x+ycos2x−yandsinx=2sin2xcos2x,
we get a product form of the left-hand side as
Im(a+b+ab)=(sinx+siny)+sin(x−y)=2sin2x+ycos2x−y+2sin2x−ycos2x−y=2(sin2x+y+sin2x−y)cos2x−y=4sin2x⋅cos2y⋅cos2x−y.
Hence, a+b+ab is real if and only if either sin2x=0, cos2y=0 or cos2x−y=0, which respectivelly correspond to x=2kπ, y=(2k+1)π and x=y+(2k+1)π.
Therefore, the solutions are
(1,b),(a,−1)and(a,−a)with ∣a∣=1, ∣b∣=1.
Solution 2. Notice that
a+b+ab∈R⟺1+a+b+ab∈R.
Let c∈C be such that a=c2. Now observe that
c(1+a+b+ab)=c+cc2+cb+cc2b=c+c+cb+cb∈R,
where we used that
cc=1 and
z+z∈R for any
z∈C. We conclude that either
c∈R, or
1+a+b+ab=0. In the first case,
c=±1 and so
a=1. In the second case, we factor the equation as
(a+b)(1+b)=1+a+1b+ab=0,and as such,a=−borb=−1.
We find precisely three families of pairs (a,b): the pairs (1,b) for b on the unit circle; the pairs (a,−1) for a on the unit circle; and the pairs (a,−a) for a on the unit circle.
Transcribed from the official solutions, pages 1–2. This transcription has not been checked a second time; if in doubt, look at the original.