I think it varies depending on the place. At my current residency, I don't think it would be viewed as that big of a deal.
However, back when I was interviewing, I do think there were certain places that were unhappy that I couldn't make it to their dinners. At one place, it was clear that they considered their pre-interview social "the first day of the interview" and the Chair of the dept even gave a talk that night. The chair seemed unhappy that I had missed it when I met with him during the interview day, even though I had explained that I was interviewing in the midst of an internship and I couldn't get the extra day off.
If you just can't make it, all you can do is express your reasons clearly and express your interest in other ways. Hopefully they'll understand. If they can't be understanding/flexible about it, maybe that's just a sign that they're not the right place for you.