Even if you try to not have it effect you, it does. Last year we had a person from 50 miles away do a phone interview and someone for the same spot fly in from overseas. Who do you think got the offer? If you are getting an interview offer it means you are qualified, but so are the other 2-3 people who got an interview offer for the same spot. Even if I tried to go solely based on the qualifications, I would find it hard not to be influenced by who came to the interview.
Full disclosure, I am not a professor, but am a 4th year grad student who has voted on applicants a few years. This is just what is true for me.