Just to allay any apprehensions:
If there is a student dinner on thursday night (there isn't always, i'm not sure what is going on this interview date), wear whatever you'd like, casual or dressy is up to you.
For the friday lunch/dinner - many people will have just come from the Friday Lecture or interviewing, so there will be many in suits/interview attire, which is completely fine. If you would like to change at some point before the dinner, feel free, but note that the Griffith Faculty Club requires business casual.
You will have an opportunity to change after dinner before going out with the students. Good luck; it's a long but fun day, from what I remember.