Look at the 2007-2008 interview trail thread.
As a blanket statement, big programs (5+ peryear) and small programs (<5) feel different. Big programs bring a little more clout, more variety, etc. Small programs bring more "personal training", more electives, more "we can put you on a rotation wherever you would like". the residents also seemed closer.
So interview at some of the top big programs, and some of the small programs. See where you fit in with personality, and where your interests are.
Then look at location. Have you always wanted to give living in the Pacific northwest a shot? Do you get frostbite in 90 degree weather? Do you like to fish and golf year round? If the city is not for you, you will not be interested in the program, and will probably be wasting an interview day/slot.
I found most applicants were very geographically centered. Especially the East Coasters, and the Midwesterners. South people were everywhere.
Look at the individual details about programs. How many electives, faculty, clinical sites, vacation, sponsoring trips to conferences, inpatient vs outpatient rotations, emg exposure, etc. These are about the only "tangibles" there are.
As far as interview experiences go...it was only one day and usually less than 6 hours. A lot was the feel of the faculty and residents that you met on that day. Was the day organized? Even if you liked the other people you interviewed with. All programs put on their A game while you are there. When asked about perceived weakness in the program, they will mostly tell you there are none. Then they will say something like "not a lot of clinical researchers in the dept". Most interviewers seemed to genuinely believe their residencies were meeting all the demands of the field.
As far as asking residents millions of questions about why the WOULDN:T pick their program, don't do it. They feel put on the spot.
In summary pick programs that are a little different from each other, geographically sound, and meet a few concrete criteria. Then go give them a shot. One applicants opinion is usually different than the next. But you will see some repeated comments as you cruise through interview/rotation reviews