A little side-note: While you need to be polite and ask permission to miss a day to go on your interviews, your attendings have been through this before and shouldn't really have a problem letting you go. When I was going through this, missing about three days per rotation was the max before you started running into problems. I know it can be uncomfortable asking for days off to go on interviews, but you have to do it, they know you have to do it, so just be polite and ask.
You do need to show that you are doing everything you can to miss as few days as possible on your rotation. I made up for some missed days by working weekends, and once I went from the hospital, to the airport, spent the night in a hotel, went to the interview the next day, then flew back and went straight to the hospital to take call.
Back to your question:
As far as the pre-interview dinners go, you really should go, but it's not required. There may be a few exceptions to that in the really competitive fields/programs.
I went to every one that I could, but missed a few due to scheduling issues. People understand that. If it's a program that you are particularly interested in, I would really make a strong effort to go. But if you can't make it, you can't make it. The dinners are good for meeting the residents, getting a different perspective on the program, and meeting your co-applicants. Programs kind of expect you to go, but I really think they'll understand if you just can't make it. If they don't, it's probably a reflection on them.
During the interview day people may ask you if you were at the dinner the night before, and I think it's perfectly acceptable to say "I had to work yesterday and wasn't able to leave Washington until late last night," or whatever. That is something that most physicians will understand.
Missing a dinner because you want to a movie or something instead probably won't go over so well (for good reason).
Hope that helps. Congrats on the interview invite. Trying to work in the pre-interview dinner is a nice problem to have.