Agree completely that nobody will consciously hold it against you for not being able to attend. Residents in particular still recall the stresses of interview travel and the difficulties of timing flights to be able to attend these functions.
Of course, I did use he word "consciously" above. While attendings may be most interested in board scores or publications, residents are most looking for folks with personalities will fit in well with the program. Fellow residents who will be easy to work with and most likely to cover one another. Subconsciously, residents may lean towards candidates who can attend these functions because they have more time to get to know them. Of course, the flip side is also true in that it's a big negative if you come out that night and are a complete douche (there are a few every year).
My recommendation? Don't kill yourself trying to make the event as we understand not everyone can. If you're not able to, make sure to send a quick e-mail to the chief resident thanking them for the invite but explaining why you can't make it. Make an extra effort on interview day to meet the residents and give them a flavor of your personality (without being overbearing, of course).