Agreed. Business casual is the way to go for the pre-interview dinners (ie. shirt +/- tie).
Of interest, most program directors do get some degree input from their residents. This 'input' usually comes in the form of major negatives. I only know of one program that - when I was interviewing - actually had the residents rate the interviewees; this is not a common practice.
IMO, there's no need to be on interview-mode, but it would make sense not to say anything too negative or opinionated at these dinners. As the saying goes, "the nail that stands up gets hammered." Just be your pleasant self and enjoy the meal.