Here's how it works at Duke.
In February of third year, the top 25% of the class is invited to submit a CV or letter to the AOA committee (which includes both faculty and students). Class rank is determined by first and second year grades only (since our third year is for research). The grades are converted to a single numerical score.
Committee members are given the CVs or letters and numerical scores for each of the 25. They discuss each person, and committee members speak on behalf of applicants they know. At the end of the meeting, committee members vote for the students they want. The top 8 are selected. The same process is repeated the fall of our 4th year and an additional 8 are selected.
We are told that selection is based on academic record and a whole host of other less quantifiable attributes (leadership, community service, extracurriculars, compassion, etc.) Committee members report that there's no real consensus about what's most important. Some reportedly vote for the 8 with the top grades; others vote for those with the most community service, and so forth.
Of course knowing people on the committee helps, but it's hard to predict who will be at the meeting or what they will say. For example, I was told afterwards that when I was being considered, I received a glowing testimonial from the director of my psychiatry rotation. This really suprised me: I really had a rough time with that rotation, and thought the director didn't like me at all. Good thing I didn't know he was on the committee, or I might have subconsciously tried kissing ass, which would probably have backfired since it would have come across as phony.
The internal inconsistency of the process is reflected by the 8 who were selected this year: all are bright, but some are very, er, grade-oriented, while others have more extra-curricular/community service stuff. And while some folks are genuinely commited to service, in a few cases it struck me as being more of a "resume-builder" than anything else. In that respect the AOA selection process is much like the medical school application process.