In my medium-sized and relatively (but not highest level) competitive specialty, when I was a PD, I got to know many of the other PD's at other programs bc we met twice a year at our specialty conferences and we just got to know the letter writers over time.
The most important determination in residency selection was audition sub internship performance at our institution (essentially a month-long interview), or a like institution where I knew the PD (as they have little incentive to write great things about a student who is not from their school). Then sub internship performance at home institution. Then Clerkship grades, but especially paying attention to the qualitative comments. There are some top schools where most everyone gets the highest "grade" but as one reads the MSPE or MSPE-equivalent, themes do emerge. In a residency class, it is extremely important to select for character, dependability, etc even more so than medical school, bc any professional issues are likely to be magnified in residency and affect the others in the residency and residency leadership (and of course, patients). We almost never pay attention to shelf exams, unless scores are consistently so low as to give concerns about passing specialty board certification exams. We do not want to accept medical students who spent time that they should have spent with patients instead putting their head in a book in the library.
