It's just sin(150)*sin(150) + cos(120)*cos(120). No need for the extra formula there.
Just know that sin(150) = sin(5pi/6) and that cos(120) = cos(2pi/3). Then you know that sin(5pi/6) = 1/2 and cos(2pi/3) = -1/2.
So (1/2)^2 + (1/2)^2 = 1/4 + 1/4 = 1/2.
1/2 is the answer, not 3/2. I'm not sure what those equations are.