Says who? Why not just do what makes you happy, while being comfortable? You'll probably do a better job when you are on your shift then the guy next to you who is miserable and hates having to work so much.
I'd rather go to the doctor who wants to be there.
1. Medicine is exceedingly demanding. It takes dedication. To put yourself through med school and residency for a "part time job" borders on insanity. If your goal is lifestyle, I don't think you could make yourself put up with the stuff you have to for near a decade to get to the "lifestyle" point.
2. There is a shortage of physicians is nearly every specialty. If you want a part time job, do something else and let the guy behind you in line for acceptance to a med school get the nod.
3. You'll almost definitely make more per hour practicing medicine than anything else you would do. Could you practice real estate? Sure, but you'd be losing money. Academics (usually) make less per hour doing non-clinical work.
4. Medicine inherently isn't a part time job. It's very dynamic and will take a certain amount of time just to stay up to date. I've talked to one of my mentors about this. He spends about half the time in clinic now that he's in academics and says he has trouble remembering all of the medication dosage.
Don't get me wrong, I'm not advocating for physicians to do 80 hour work weeks and not have a work/life balance. There are lifestyle specialties, and I'm even going into one that's a little more family friendly. I'm just saying that if lifestyle is your goal, going to med school doesn't seem practical.