Agree with the previous, and would add:
1. A letter needs to be really good. All of the letters we see are generally positive, so if your letter is one of those "generally positive" letters then it will get overlooked. A lukewarm letter, or a letter with red flags, is extremely prejudicial because either (a) it demonstrates poor judgment, because you really should know better than to obtain a letter from someone who is not going to write you a great letter; or (b) it suggests that you are actually a lukewarm applicant or applicant with red flags. It is also possible that (c) you are actually a shining star and that your attendings can't see through the superficial games and politics or that the poor marks you received were the result of some misunderstanding or unfair circumstance. The false positive sucks from the perspective of the applicant, but from our perspective we just have too many other competitive applicants to consider for it to be worth our time to investigate as a serious possibility.
2. We like writers to comment on intellectual curiosity, maturity, teachableness, psychological mindedness, etc. Ability to work well with others in a team setting is important, but that's pretty much a given because everyone comments on it.
3. Your letter will be memorable if the writer provides a specific example consistent with the praise. e.g., "We had a patient in the ICU with a mysterious constellation of symptoms, and her failure to improve stymied even our infectious disease consultation service. On Melissa's first day of the rotation, while she was getting to know this patient and the patient's family, she made some very astute observations and asked critical questions that enabled us to diagnose a comorbidity of critical importance much earlier than would have been tested for otherwise. Consistent with her first day's performance, time and time again during the rotation she demonstrated a unique ability to establish rapport with even the most difficult patients...", etc.
4. Your letter will be viewed as more credible if the writer (if s/he is unknown to the reader/admissions committee) provides some sort of metric by which we can judge the generally positive tone. e.g., "I have worked with many medical students who have gone on to become chief residents of surgery and medicine at Brigham and Women's Hospital, UCSF, and Duke, and Marian stands out even among this elite group", "Jack compares very favorably to Jenny Smith, who graduated from your program in 2006 and who I mentored as a medical student", etc.
2-4 above will also show that the writer knows you well.