Well, there's one person's opinion.
You definitely don't have to like all of the sciences, let alone love them. You'll forget alot of what you learn during the 1st 2 years of medical school.
There are some physics related things in physiology, and you may like physiology, but it's ok if you hate physics. I don't think anyone is going to care if you can predict at what yard line the Punter's kick is going to land, or his hang time.
There is definitely some chemistry involved in Biochemistry, but you don't have to be a chemistry master for it. They aren't going to expect you to be able to roll of Organic Reactions like you just took the Organic Final yesterday.
So, no, if you don't love, or don't like something, that's not a reason to quit. I HATED Gen Chem 1 and 2. Organic was a little better, but still not something I'd take as an elective. I hated Physics.
There may even be some classes you dislike in medical school. But, the truth is, that's not reason to stop. If you shadow, and you think, man I'd really hate to be that guy, stop and re-evaluate.