Do you know anything about military life?
Long hours, odd hours, duty, moving consistently, being stationed in the middle of nowhere, deploying and being away from friends and family for long periods of time, dangerous situations if deployed, strained relationships, if you have kids you'll be missing a lot of time with them.
It's difficult to learn to love. Even people who go in wanting to serve for the right reasons and having been from a military family where they understand as much as one can without doing it themselves yet sometimes end up going in and not liking it.
While life as a military physician is certainly different than that of someone enlisted, or even other officers, the above will still apply.
If you pick up rank, you may also end up in more administrative positions where you will practice less and less medicine - this is the case where it could affect you going back to civilian practice, but this is usually if you stay in for a little more than the minimum because it's often rank based.
The fact that you started out saying you dont love it or hate it and are inquiring about HPSP to cover your costs tells me that you're probably looking into it for the wrong reasons, and it will make you bitter. Not to mention, when I was in, you could tell who really didnt want to be there, and no one wanted to be around them.
It's a great gig for those who REALLY WANT to be in and who would join even without the school coverage, but if you wouldn't join without the scholarship, then its probably not for you.