I got into a high ranked undergrad and didn't give much thought to the whole ranking thing once I was in - though I have to say that I looked closely at the average stats when I applied, because I wanted to be surrounded by people who challenged me. Then I took one class at another school, ranked much lower, and realized that yes, those differences do exist. I went from being very average at my home school to some sort of class god in a week, it was actually quite a disturbing experience. I honestly never thought there'd be such a disparity. So, based on my personal experience with this, I'm gonna go out on a limb and say that yes, I think the caliber of people in Harvard med can be quite different. And I can say it considering I'll never get into Harvard med, and that's fine with me - I've met many people smarter than me in college, and it doesn't particularly bother me that I'm not the best-read, most eloquent, and smartest of them all. Being at the top is lonely, as they say. I like being able to learn not just from my prof, but also from my peers.
Also, highly selective institutions look not just for academic powerhouses when it comes to applicants. My school had a very diverse undergrad population, from the kids of investment bankers who went to private high school, to kids who grew up in the trailer park, and we all learned from each other in the classroom.