If you like your job long hours are less of a big deal - time flies by remarkably fast when you are engaged and interested. If you don't like your job, even working 40 hours/wk a week can be oppressive. So a focus on free time rather than doing something you enjoy is misplaced.
The average physician works in the 60+ hour a week range, with periodic weekend work, so going in, that's kind of what you should expect. It's a hard long houred job but there are many professionals out there in other professions who work similarly. While it's comforting to tell yourself that after training you can tailor your schedule to meet your needs and to some extent that's true, I wouldn't buy into that hook line and sinker --it's only half the story. We all "know a guy" who earns a nice income while hardly working, but the truth is a bit more nebulous. There are always trade offs and to get that kind of schedule that guy likely gave up a lot in terms of upward progression and job security.
Early on in your career you may be expected to pay your dues, take the lions share of call, work all the major holidays etc for your group. Many new attendings and private practitioners actually see hours go up, not down, immediately after residency.
And you'll find that a desired career trajectory may dictate working longer hours and/or serving on various committees, going to meetings or other administrative functions to help you move up the ranks. The guy who works every other week may seem like he has a good gig until you realize that he's expendable -- the first guy out the door when budgets tighten, and the last guy to get a raise or a promotion, if ever. Face time and the quasi work related things you do on top of your job can make the difference on certain career paths. If you work for an employer they can always replace you with someone who will go that extra mile if you won't. So I caution you that whenever you see someone working a schedule that seems too good to be true, it might be a bit of an illusion -- often they've traded away too much upside. A lot of those dream schedules are better left on the table.