I can't speak for anyone else, since it's a very personal thing, but my criteria, in no particular order, were:
1. Large size of program (>4 residents per first year class)
2. Major academic tertiary referral center
3. High surgical volume, quality of surgicals
4. Research opportunities & funding, paid post-doc available
5. Strong CP/CP-only track available (theoretically, a place with CP-only might have stronger/better CP - still my untested hypothesis)
6. Track record of graduates getting top fellowships
7. Track record of graduates getting jobs in academia AND private practice in the geographic region I ultimately want to live in
8. Vibe & personality match with the residents and faculty, friendships between residents, amount of socializing outside work (not that I'm necessarily a social butterfly, but it's nice to be friends as well as coworkers)
9. Quality of life outside work - the cities I was strongly considering were Seattle, San Francisco, and LA - looking for lots of cultural activities (theater, art museums, restaurants, sports) and easily accessible outdoor activities (backpacking, hiking, skiing, sailing). Salary vs. cost of living was a factor.
#9 actually ended up being the tie-breaker, as I was looking at excellent programs that I loved in all 3 cities, so ultimately it really came down to which city we wanted to live in.