Everyone will tell you it doesn't matter, it's just for your benefit to get to know the residents, etc. etc. Bullsh*t. The residents more than likely do share information about the applicants that is factored in to the decision making process, it's ony natural when that is one of the few things they have to go on about you as a person. That happens at my prelims program this year, and all is fair in this game. People talk.
I missed 2 pre-interview dinners because of travel conflicts last year, and both places it was mentioned during the interview the next day. One I was asked about it by one of the chiefs I interviewed with, one with a program director. It sucks, but it does say in a way that that program is not top priority for you. Just a heads up...if you are REALLY serious about the program, and know you will probably rank it high, take every opportunity you have to get quality time with the players.