There was a first year student in UVA that was offered a full scholarship to both Harvard and John Hopkins, she turned down both schools to come to UVA because she had a better "feel" for our school. Essentially, our admissions committee had to scamble up 4 years of full tutition from the alumni fund to match the previous 2 offers to get her to come to UVA. Harvard has a great name and when you apply to residency, the doors will be wide open for you. However, if you are a superb applicant, you will get multiple accepetances, in the end, you should choose a school that will be the best "fit" for you, in terms of the type of people, location, how the classes are structured, etc, etc. The worst thing is going to a place for the name and end up hating your life for the next 4 years. This has definitely happened to people before. If you do well in school, you will match to a great residency whether you come from a top 20 medical school or Harvard. One of my best friends who was applying to Orthopedics last year, could have gone to Wash Univ for Orthopedics residency (which has a huge name in the Ortho world), but decided to rank another program of lesser name higher (and matched there) because he did not feel a "fit" with the people at Wash U. His reasoning was that, 5 yrs of working at a place that I will hate or not enjoy is NOT worth the name.
At the same time, it is hard to get a feel for the medical program just by a single day of interview. I guess you will have to go with a "gut" feeling. I decided to go to UVA over U of Michigan and another top 10 medical school, but looking back, I am 100% happy with my decision. In the end, you will get a great education no matter where you go to medical school. A lot of the learning in medical school and after, is self directed.