I have both. I'm even one of those rare idiots who did them separately.
It really is an apples to oranges comparison. Medical school was like getting repeatedly kicked in the balls, but all you had to do was hold on and let the wave pass over you. Everything is scheduled, finite, and predictable. Yes, it blows, but you can literally count the days until it's over. Also, while studying gets monotonous, the clerkships provided ample variety. Even the worst ones came to an end like clockwork. After it's over, your life is more or less on a set path: residency +/- fellowship then job. Simple.
A PhD based on scientific experimentation is a whole different ballgame. You are essentially set adrift with little more than enough rope to hang yourself with. Your PI can be great or a total dick, your lab can be supportive or backstabbing, and your experiments are almost guaranteed to fail, fail, and fail again before they work. There is no set finish date, just a seemingly endless horizon that will always need one more gel, one more Western, one more PCR. To top it off, there is the constant threat that your work will ultimately just fail, or you'll get scooped, and despite years of toil, in a heartbeat you can literally be left with nothing. All this and then you get to postdoc for five more years, and then you get to fight for one of the scant jobs, and then you're either fighting for funding and tenure or you're in industry and acting as someone's bitch. It's really quite soul crushing.