For momentum problems like this, you have to pay attention to direction (momentum is a vector with direction because velocity is a vector with direction)
So let's say going to the right is positive, going to the left is negative.
It's an inelastic collision (only momentum is conserved), and you are only concerned with the horizontal velocity due to the question.
For the 5 kg ball, you need to break the motion into components, as it looks like you did. The horizontal velocity then comes out to 5... I did sin 30= o/h 1/2= x/10, so velocity is the x-direction is 5 m/s.
This is going to the right, so its positive 5. The 20 kg ball is going to the right, so it's negative velocity (-10 m/s)
Thus:
5 m/s (5 kg) + 20 kg (-10 m/s) = 25 x
25 -200= 25x
-175=25x
-7= x
Final velocity is 7 m/s to the left (since this is our assigned negative direction)
Hope this helps! I think you just missed assigning the directions-- remember, momentum is a vector!
