I don't think med school is the hardest thing in the world, and free time definitely exists throughout all years. I am not debating that point with you.
Having a job, taking classes, and studying for the MCAT concurrently is probably harder than medical school. However, not all of that is pure studying. Imagine spending 2/3 of the time as above but using it only to study. There is a large difference between studying for 4 hours a day and studying 8 hours a day. Motivation is harder to come by, things start to meld together, etc. etc.