Plenty of the rads residents I've worked with groan when you mention physics to them. It's an important part of understanding what goes into getting the images/artifacts we do. Granted I have yet to sit through nearly as many lecture as I will in the coming years but I tend to find it more it more interesting in the context of medicine than in the vacuum of something like a pure physics course.
That is to say: no, enjoyment of physics is not requisite for going into radiology. You don't 'use' physics on a daily basis explicitly but a thorough background can help in interpreting the images you see.