Well of course they are self-selected but that tells me a ton. If an applicant has a LOR that is vague or impersonal - that tells me a lot about the applicant's lack of perspective. If you can't figure out who can write you a good letter or you don't have enough relationships with people to have 3 people who can write good letters, that tells me something. If you have no idea how your superiors perceive you, that tells me something. And by "good" I'm not saying who can say the most good things, I'm saying LORs that say "you know, this person was unsure when they started XYZ rotation but by the end, they were as good as my residents" or "they took criticism well" or "they really worked hard to develop an efficient process for working up patients", etc. Willingness and ability to grow is huge and is not a skill you can teach. I can teach clinical skills, I can't teach someone how not to be a lazy ass or a self-important d-bag.
Certainly you can go wrong sometimes, just like you can go wrong with any part of the application. I wish I had paid more attention one year to one LOR that would have saved me a lot of strife. Overall, it's still somewhat of a crapshoot but writing LORs is a PITA so if someone writes a good, meaningful LOR that carries a lot of weight to me.
